# Fine-tuning

Train a Jepela model of your own on your labelled cases. It is used only if it answers cases it never saw better than the standard engine and still listens to memory as well, and only you can use it.

## Memory or fine-tuning?

| | Memory | Fine-tuning |
|---|---|---|
| Holds | facts about one subject: "Anna is our biggest customer", "opted out of marketing" | how you decide, across all cases: "how we sort our tickets" |
| Changes | at once: remember, forget | by training again |
| Forgetting a person | one `forget` | not possible: what a model learned stays in it |

Put facts about people and accounts in memory. Let training cases teach the pattern. A fine-tuned model reads memory like the standard one, so the two work together.

## 1. Add labelled cases

Your golden set is the training data (see [Quality](quality.md)): each case is a situation, your questions, and the right answers.

```bash
curl -s $JEPELA_BASE_URL/v1/golden -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" -d '{
  "cases": [
    {"id": "t-001", "state": "Where is my order 88213?",
     "questions": {"intent": {"type": "choice", "instructions": "What does the customer want?",
       "criteria": {"order_status": "where an order is", "billing": "charges and refunds", "other": "none of these"}}},
     "expected": {"intent": "order_status"}}
  ]}'
```

At least 20 cases are needed. More is better: a few hundred real cases from your own work say far more than 20.

A case can name a `subject` (and `values`, `derive`, `memory` options, as in a request). Jepela then trains on the case exactly as it is answered: with the lines that subject's memory recalls, and the computed facts, placed before the state. Cases whose right answer depends on memory teach the model to use it.

## 2. Train

```bash
curl -s $JEPELA_BASE_URL/v1/finetune -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" \
  -d '{"base": "english"}'
```

`base` is `english` (default), `multilingual` or `typed-decisions`. The answer is the job: `job`, `model` (the name your model will have), `cases`, `held_out`, `state: running`. One job runs at a time per account.

- One case in five (chosen by its id, so the same cases every time) is held out. Jepela trains on the rest.
- Then it answers the held-out cases twice: with the standard engine and with your new model.
- **Two tests, both must pass:**
  1. **Accuracy:** on the held-out cases, your model is right more often than the standard engine.
  2. **Memory:** memory still moves its answers at least as much as the standard engine's. Jepela asks a fixed set of paired cases (the same situation with and without one memory line), plus your held-out cases that have memory, both ways. Your model's average effect may be at most 0.05 lower, with at most one fewer strong effect (0.2 or more).
- If either fails, the model is discarded and nothing changes. A model trained only on cases without memory can learn to ignore it; the second test catches that.

## 3. Follow the job

`GET /v1/finetune/<job>` (or `GET /v1/finetune` for all of them): `state` is `running`, `saving`, `done` or `failed` (with `error`). When done, `result` has `won`, `accuracy_won`, `memory_kept`, `base_accuracy`, `tuned_accuracy`, `base_memory_effect`, `tuned_memory_effect`, `held_out_questions`, the log loss of both, and `training_seconds`; `usable` is true when your model is in service. `cases_with_memory` counts the cases trained with memory lines, and `warnings` lists anything to look at, such as cases that seem to hold personal data.

The console at [app.jepela.com](https://app.jepela.com) has the same under **Fine-tuning**: your case count, a Train button, your jobs and your models.

## 4. Use your model

Name it in any request:

```json
{"model": "jepela-ft-acme-1a2b3c4d", "state": "I was charged twice.", "questions": {...}}
```

`GET /v1/models` lists it for your account only. Memories, rules, windows and every other option work as with the standard models. Another account naming it gets `unknown model`.

## Delete a model

```bash
curl -s $JEPELA_BASE_URL/v1/finetune/delete -H "Authorization: Bearer $JEPELA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model": "jepela-ft-acme-1a2b3c4d"}'
```

Its files are deleted and its name stops working. Deleting your account deletes all your models.

## Deleting training cases

When you delete golden cases (`POST /v1/golden/delete`), every model in use that trained on them is marked: the answer lists them in `models_trained_on_deleted_cases`, and the job gets a warning. What a model learned from a case cannot be taken out of it. If a case was deleted because a person asked to be forgotten, train again without it and delete the old model.

## Good to know

- The engine has to be running to train. Training takes about a minute per hundred cases.
- Training cases with an email address or phone number get a warning: keep such facts in memory.
- Thirteen held-out cases are a small test. Judge a model on your own golden set too: `POST /v1/golden/run` with `{"model": "jepela-ft-..."}`.
- Fine-tuning is free during the preview; you pay for the requests your model answers, like any other.