The task.
A document, or its text, has to be turned into the same record every time: supplier, date, total, line items; or parties, term and notice period. Today a large model is given the schema in the prompt and asked for JSON.
Why it is a candidate for a smaller model.
- The schema is fixed. The model is asked the same question about every document, which is the repetition fine-tuning is for.
- Correctness can be checked field by field against records you already hold in a system of record.
- Structured output is the weak point of a general prompt and the thing a narrow model can be trained and tested on directly.
What would be measured.
These are the acceptance criteria this task usually needs. You would set the figures; the model you run today would be scored against them first, on held-out examples, and the fine-tuned candidate would have to meet them on the same examples.
- Accuracy for each field separately. A model that reads dates well and totals badly is not ready, whatever the average says.
- The share of outputs that are valid against the schema with no repair step.
- How it behaves when a field is absent from the document: leaving it empty is right, inventing a value is the failure that matters most.
- Latency for one document, and the estimated cost of a document, against the model you run today.
What you would bring.
Documents paired with the record that was accepted for them: what was posted to the ledger, or what a person corrected and saved. The corrected version is the label. The uncorrected output of the current model is not.
Examples will be checked for quality, duplicates and sensitive data and split into training, validation and held-out sets before anything is trained. The first step explains why.
When a fine-tune is the wrong tool for it.
- The layouts are few and fixed. A template or a parser is cheaper and exact; keep the model for the documents that do not fit.
- The schema is still changing. Settle what you extract before training a model to extract it.
- The difficulty is reading the page, not understanding it. Poor scans are a text-recognition problem, and no fine-tune of a language model fixes the input.
The decision guide goes through the alternatives. Daidon is being built to point to them when your examples do.
Where this stands.
Daidon is in development. There is nothing to install or run yet. Join the waiting list and we will email you once, when early access opens.