The task.
A message arrives and has to be given one label from a list you control: a queue, a priority, a product area, a reason code. Today a large general-purpose model reads the ticket with a long prompt that lists the categories and a few examples, and returns one of them.
Why it is a candidate for a smaller model.
- The output is one of a closed set. There is nothing to compose, so a small model has less to get wrong.
- You already have labelled history. Every ticket an agent resolved carries the queue it ended up in, which is a label a person stood behind.
- It runs on every ticket. A cost or a delay that is small per call is paid thousands of times.
What would be measured.
These are the acceptance criteria this task usually needs. You would set the figures; the model you run today would be scored against them first, on held-out examples, and the fine-tuned candidate would have to meet them on the same examples.
- Overall accuracy on held-out tickets, and accuracy for each category on its own: an average can hide a category that is always wrong.
- Which pairs of categories are confused with each other, since some mistakes cost more than others.
- What happens on the rare categories, where there are few examples to learn from.
- Latency for one ticket, and the estimated cost of a ticket, against the model you run today.
What you would bring.
Past tickets with the label a person gave or confirmed. Tickets that were re-routed are worth marking: the first label was wrong and the last one was right. Labels that only ever came from the current model are a different kind of evidence and will be kept apart.
Examples will be checked for quality, duplicates and sensitive data and split into training, validation and held-out sets before anything is trained. The first step explains why.
When a fine-tune is the wrong tool for it.
- The categories change often. A fine-tuned model knows the list it was trained on; a list that changes monthly is better served by a prompt that carries it.
- Sorting depends on facts outside the ticket, such as the customer's plan or an open incident. That is a lookup, or retrieval, before any model.
- A handful of keywords already decide most tickets. Then the answer is a rule, with a model only for what the rule cannot place.
The decision guide goes through the alternatives. Daidon is being built to point to them when your examples do.
Where this stands.
Daidon is in development. There is nothing to install or run yet. Join the waiting list and we will email you once, when early access opens.