Fine-tuning improved the structure. Answer quality remained uncertain.
The research question
How does LoRA-based supervised fine-tuning compare with a strong five-shot prompt in producing schema-valid, useful, metaphorically coherent, and stylistically consistent ChatG&T responses on unseen prompts—and what trade-offs does it introduce in prompt-token usage and latency?
TLDR
Fine-tuning made ChatG&T more structurally reliable and almost eliminated the need for examples in the system prompt.
On 60 unseen prompts, the fine-tuned model produced 52 schema-valid responses. The five-shot baseline produced 43. At the same time, the average input fell from approximately 2,576 tokens to 32.
The model learned the form of ChatG&T more readily than the substance of its answers.
Schema-valid
52 / 60
Fine-tuned, compared with 43 / 60 prompted.
Recurring input
32
Average prompt tokens, down from approximately 2,576.
Full response
26 / 60
Fine-tuned, compared with 16 / 60 prompted.
Generation time
5.70s
Fine-tuned, compared with 3.92 seconds prompted.
01 / 05
What we compared
The primary comparison used the same 1.5-billion-parameter base model in two configurations. The prompted system received a detailed system prompt with five worked examples. The fine-tuned system used the selected LoRA adapter and a minimal system prompt.
Both answered the same 60 prompts, which had been withheld from training. Each prompt was attempted once. Responses were not repaired, regenerated, or manually corrected.
A successful response had to do more than contain valid JSON. It needed to follow the required schema, answer the user's underlying question, express that answer through a coherent cocktail metaphor, and read like a plausible recipe.
02 / 05
The structure became reliable
Five-shot prompt
43 / 60
Schema-valid responses
Fine-tuned
52 / 60
Schema-valid responses
Fine-tuning made the model more reliable at returning responses the product could render. It followed the required schema on 52 of 60 prompts, compared with 43 for the five-shot baseline.
That is an improvement of 15 percentage points. Allowing for variation across the 60 prompts, the estimated improvement ranged from 1.7 to 28.3 percentage points and remained above zero. This gives us reasonable confidence that fine-tuning improved structural reliability in this test.
This result is about format, not answer quality. A response can follow the schema and still contain poor advice.
Fine-tuning taught the model the response contract: which fields to produce, how to organise them, and how the parts of a ChatG&T answer fit together.
03 / 05
Answer quality is harder to call
The automated evaluator rated the fine-tuned responses more highly for usefulness, metaphor, and recipe quality. However, these assessments were subjective, and the human review did not consistently agree. The apparent improvement in answer quality is therefore less certain than the improvement in structure.
Criterion
Prompted
Fine-tuned
Underlying answer
19 / 60
29 / 60
Cocktail metaphor
33 / 60
42 / 60
Recipe execution
39 / 60
45 / 60
Usefulness
The automated evaluator judged 29 of 60 fine-tuned responses useful, compared with 19 of 60 prompted responses. However, the human reviewer agreed with the evaluator in only 9 of 18 calibration cases. The result suggests an improvement, but does not establish that people would find the fine-tuned answers more useful.
Recipe style
The fine-tuned model produced more acceptable recipes overall: 45 of 60, compared with 39 of 60. This advantage came from producing more correctly structured responses. Among schema-valid responses, the prompted model passed the recipe-style check slightly more often—91% compared with 87%. Fine-tuning increased the number of usable recipes, but did not improve recipe style itself.
04 / 05
The long prompt disappeared
Average input tokens
2,576
→ 32
Approximately 98.8% fewer recurring input tokens.
Timed generation
3.92s
→ 5.70s
The shorter prompt did not make generation faster.
Fine-tuning allowed us to remove the five worked examples from every request because the adapter had learned the behaviour they demonstrated. This reduced the average input from approximately 2,576 tokens to 32, leaving more context available for the user's request.
The shorter prompt did not make responses faster. The fine-tuned model took an average of 5.70 seconds to generate a response, compared with 3.92 seconds for the prompted model. Fine-tuning also required the upfront work of preparing data, training the adapter, and evaluating it.
This latency result applies only to model generation in our experimental setup. It does not show that LoRA is generally slower or predict how either system would perform in production.
05 / 05
What fine-tuning changed
Fine-tuning moved much of the ChatG&T behaviour out of a long system prompt and into the model.
Fine-tuning clearly improved the structure of the responses, but we found weaker evidence that it improved their content.
One possible explanation is the size of the base model. A 1.5-billion-parameter model may be capable of learning a consistent response format without being capable of producing consistently useful answers.
We tested only one model size, so we cannot confirm that model capacity was the problem. The next step would be to repeat the experiment with a stronger base model. This would help us distinguish between the limits of the model and the limits of our training approach.
The experiment establishes
- Fine-tuning improved first-attempt structural reliability.
- It removed almost all recurring prompt context.
- It was slower in the matched generation measurement.
The experiment does not establish
- That fine-tuning generally produces better answers than prompting.
- That people would consistently prefer the fine-tuned responses.
- That these results generalise beyond this model and evaluation.