01 / Prepare
Start from examples you can stand behind.
Bring your examples as CSV or JSONL. They will be checked for quality, duplicates and sensitive data, then split into training, validation and held-out sets.
An example is an input the model sees and the output that is correct for it: a ticket and its queue, a document and its fields, a message and its intent. You will bring them as CSV or JSONL. Before anything is trained, they will be checked.
- Quality. Empty outputs, inputs cut short, labels outside the set you said was allowed, the same input labelled two different ways. Problems like these teach a model the wrong thing and are cheaper to fix in the data than in the model.
- Duplicates. The same example appearing in training and in the test makes the test meaningless: the model is being asked something it has memorised. Duplicates will be found before the split, so they cannot land on both sides.
- Sensitive data. Names, addresses, account numbers and the like will be flagged, so that you decide what goes into a training run with your eyes open.
- Provenance. Every record will keep where its label came from: a person, a system of record, or a model. A label that is a model's unchecked output will be marked as that and never treated as a trusted label automatically. Training a small model on a larger one's responses is an established technique, usually called distillation,[1] and it is useful; it also copies the larger model's mistakes unless someone has checked the responses.
The three-way split.
The examples will then be divided into three sets that do different jobs. The training set is what the model learns from. The validation set is used during training to see whether it is still improving or has started memorising. The held-out set is locked away until the end and used once, to judge.
The held-out set is the one that matters. A model has not seen it, so its score there is a fair estimate of how it will behave on tomorrow's inputs. Everything this product reports as a result will be measured on held-out examples.
02 / Benchmark
Measure the model you run today, first.
The model you run today will be measured on the held-out examples, against acceptance criteria you set. That is the bar.
Acceptance criteria.
Before any training, you will say what “good enough” means for this task. Acceptance criteria are the conditions a replacement has to meet: accuracy no lower than a figure you name, a share of outputs that parse against your schema, a latency ceiling. They are written down first so that they cannot drift towards whatever the new model happens to score.
Why the baseline comes before the training run.
Your current model, with your current prompt, will be run on the held-out examples and scored against those criteria. This does three things.
- It sets the bar. The fine-tuned model will be compared with a measured number, on the same examples, and not with an impression of how well the current one works.
- It tests the test. If the model you trust in production scores badly, the fault is often in the examples or the criteria. Better to find that before paying for training.
- It can end the project early. If the current model misses your criteria too, the problem is not model size. If a shorter prompt or a cheaper hosted model already clears them, you may not need to fine-tune at all.
The page on when fine-tuning is the right answer covers the alternatives this step can point to.
03 / Train
A managed fine-tune of a smaller open-weight model.
A managed supervised fine-tune of a smaller open-weight model, with LoRA where the model supports it. Selected Qwen models first.
Supervised fine-tuning means training on labelled pairs: this input, that output. The model's parameters are adjusted so that it produces the kind of output your examples show for the kind of input they show,[2] and the validation set is watched to stop before it starts memorising them.
Where the model supports it, training will use LoRA. The base model's weights stay frozen and a small set of added matrices is trained in their place, which takes far fewer trainable parameters than changing the whole model.[3] The adapter can be merged into the base weights afterwards, so it need not add latency when the model is served.[4]
The models are open-weight, starting with selected Qwen models. (Qwen models are published with their weights; the Qwen2.5-7B-Instruct card, for one, lists an Apache-2.0 licence.[5]) Open weights are what make the last step possible: a model you can export is one you can run where you choose.
“Managed” means the run will be set up, monitored and recorded for you, and can be repeated: the same data, the same settings, the same result to within the noise of training. A saving that cannot be reproduced is not one you can plan around.
04 / Evaluate
The same test, the same criteria, side by side.
The candidate will be compared with the baseline on the same held-out examples: accuracy, valid structured output, latency and estimated cost.
- Accuracy. How often the output is correct on held-out examples, overall and broken down, because an average hides the category or the field that is always wrong.
- Structured-output validity. The share of outputs that parse and match your schema as they are, with no repair step.
- Latency. How long one request takes, measured under stated conditions.
- Estimated cost. What a request would cost on the candidate against what it costs today. It will be labelled an estimate until an experiment you can re-run has measured it.
Each line will be shown against the baseline and against your acceptance criteria. A candidate that is cheaper and faster but misses the accuracy you asked for has failed, and the result will say so in those words.
05 / Decide
Deploy, export or reject.
Deploy the model, export it, or reject the experiment. Rejecting is a result, and you will know why.
- Deploy. The candidate met your criteria and you want it served for you.
- Export. The candidate met your criteria and you want to run it yourself: you will be able to download the supported artefacts.
- Reject. The candidate missed. You keep the measurements, the reason, and the model you already had.
Reject is on the list on purpose. The cheapest outcome of an experiment like this is finding out, before any migration, that the smaller model is not ready, or that the task wanted a different tool.
Questions
What is a held-out test set?
A share of your examples that is set aside before any training and never shown to the model while it learns. Because the model has not seen them, how it does on them is a fair estimate of how it will do on new inputs. A model scored on its own training examples is being marked on an exam it has already read.
What are acceptance criteria?
The conditions the smaller model has to meet for you to use it, written down before the experiment: for example a minimum accuracy on held-out examples, a minimum share of outputs that are valid against your schema, and a latency you can live with. They are yours to set, and they do not move once the results are in.
Why measure the current model first?
Because the question is whether the smaller model can replace it. Without the current model's numbers on the same examples there is nothing to compare against, and the criteria tend to be chosen after the fact to fit whatever the new model scored.
What is LoRA?
Low-rank adaptation: a way of fine-tuning that leaves the original model's weights frozen and trains a small set of additional matrices instead. It needs far fewer trainable parameters than changing the whole model, and the result can be merged back into the base weights.
What happens if the smaller model is not good enough?
The experiment is rejected, and the result says which criteria it missed and by how much. That is a finished piece of work: you know the task is not ready for a smaller model on these examples, and you did not migrate to find out.
Can I run these steps today?
Daidon is in development. There is nothing to install or run yet. Join the waiting list and we will email you once, when early access opens.
Read next
Sources
Each was read on 12 November 2026.
- Amazon Bedrock User Guide, “Customize a model with distillation in Amazon Bedrock”: https://docs.aws.amazon.com/bedrock/latest/userguide/model-distillation.html
- Amazon Bedrock User Guide, “Customize your model to improve its performance for your use case”: https://docs.aws.amazon.com/bedrock/latest/userguide/custom-models.html
- “LoRA: Low-Rank Adaptation of Large Language Models”, arXiv:2106.09685 (v2, 16 October 2021): https://arxiv.org/abs/2106.09685
- Hugging Face PEFT documentation, “LoRA” conceptual guide: https://huggingface.co/docs/peft/main/en/conceptual_guides/lora
- Qwen2.5-7B-Instruct model card, Hugging Face: https://huggingface.co/Qwen/Qwen2.5-7B-Instruct