Experiment.

There are two main ways to get a language model to perform a specialised task:

• Give it detailed instructions and examples in the prompt.
• Fine-tune it on examples of the behaviour we want.

We wanted to compare these approaches using the same small language model.

Put simply, we asked whether fine-tuning worked better than a well-developed prompt—and what each approach required in input tokens and generation time.

ChatG&T was the test case. Its responses needed to follow a fixed structure, answer the user well and express that answer as a convincing cocktail recipe. This allowed us to measure structural reliability separately from response quality.

To answer the question, we needed to compare prompting and fine-tuning in a way that showed what each approach changed.

We designed four systems using the same base model:

Before training anything, we needed a reliable model, a repeatable way to run it, and a fair prompt-engineered baseline.

We selected Qwen2.5-1.5B-Instruct. It was small enough to train and deploy within the project’s budget, supported LoRA, could already produce structured output, and had a permissive licence.

We then tested the complete LoRA process on rented GPU hardware. We confirmed that we could attach an adapter, train it without changing the base model, save it, reload it and reproduce the same output.

Next, we built one inference system for the whole experiment. It recorded the prompt, raw response, model and adapter, token counts, seed and generation time. We also built a strict validator for the required JSON structure. Invalid responses were recorded as failures rather than repaired.

Finally, we developed the five-example prompt. We tested four versions on a separate set of 20 development prompts. Version 3 won the selection rule we had defined in advance. Version 4 performed worse, so we stopped rather than continuing to revise the prompt.

Version 3 did not perform best on every measure when we confirmed it in the final runtime. We kept it because it won the agreed selection process, not because it made the later fine-tuning comparison easier.

At the end of this stage, the base model, runtime, measurement tools and prompt baseline were fixed. Fine-tuning had not begun, and the final test prompts had not been created.

Before creating the training data, we decided exactly how the systems would be tested.

We planned a final test containing 60 prompts across five types of task: advice, explanation, low-stakes emotional support, creative writing and short-form rewriting. The test would include ordinary requests, unfamiliar subject areas and prompts designed to challenge the required format. At this point, we defined the shape of the test but did not write the exact prompts.

We also fixed how each response would be evaluated. First, it would be checked automatically against the JSON schema. If it passed, it would then be judged on the quality of the answer, the cocktail metaphor and the recipe style. A response received a full pass only if every part succeeded.

The main comparison would be between the five-example prompt and the fine-tuned model. Their responses would be compared without revealing which system had produced them. We also decided how to record cases where one or both systems failed the structural check.

For every response, we would record the number of input tokens, the number of generated tokens and the time taken by the model to generate it.

We knew that 60 prompts could not perfectly represent every possible user request. We therefore decided in advance how to report the uncertainty around differences between the systems. We also selected a fixed sample for the project author to review, so we could see how closely the automated evaluator agreed with a human judgement.

Finally, we defined how the future test prompts would be checked against development and training material. We also planned to compare the generated responses with examples each system had seen, looking for suspicious reuse rather than assuming that every similarity proved memorisation.

At the end of this stage, the complete evaluation method was fixed. The training data had not been created, the exact test prompts had not been written, and fine-tuning had not begun.

Once the evaluation method was fixed, we created the examples that would be used to fine-tune the model.

The dataset contained 200 user prompts paired with complete ChatG&T responses. It covered the same five types of task defined for the experiment, with 40 examples of each.

Every response had to follow the JSON schema and pass a separate review of its answer, metaphor and recipe style. Examples that did not pass were revised or replaced rather than included as training targets.

We split the dataset into 160 training examples and 40 validation examples. Related versions of the same situation were kept in the same split. This prevented the model from training on one version of a scenario and then being validated on a close variation.

We also selected a 40-example pilot set from within the training data. This gave us a smaller dataset for checking the training process before running the full adapter search.

We did not use Qwen’s responses or training performance to decide which examples to keep or change. The exact final test prompts had not yet been written, so the dataset could not be designed around the questions the model would later face.

At the end of this stage, the training data, validation data and pilot set were reviewed and frozen. Only then did we begin creating the held-out test.

After the training data and five-example prompt were frozen, we wrote the 60 prompts for the final test.

The set contained 12 prompts from each of the five task types. It included:

• 30 ordinary requests similar to the intended use of ChatG&T.
• 15 requests about photography, tabletop games and pottery—subjects deliberately excluded from the training data.
• 15 prompts designed to challenge the required behaviour or JSON format.

We also varied how the prompts were written. Some were questions, some were direct requests, and others were statements or fragments. Some combined several requirements, while others included specific constraints the response needed to follow.

We compared the 60 prompts with 231 prompts already used elsewhere in the project. The check found no exact matches and no pairs with wording similar enough to trigger review.

We then reviewed the complete set. Every prompt needed to be clear, low-stakes, answerable without external tools and suitable for evaluation.

No model responses were generated while the test was being created. The prompts were frozen before fine-tuning began and could not be changed after we saw how the model performed.

This gave us a controlled final test that had remained separate from prompt development, dataset creation and adapter training.

We began with a small pilot using 40 training examples. Its purpose was to check that the complete training process worked before moving to the full dataset.

The pilot completed successfully. The training and validation loss improved, the base model remained unchanged, and the adapter could be saved and reloaded correctly.

We then tested the pilot adapter on ten development prompts. It changed the model’s responses, but none followed the required schema. This showed that a technically successful training run did not necessarily produce the behaviour we wanted.

We moved on to a limited search of five full candidates. Each candidate tested a specific idea:

1. Train on the complete dataset.
2. Train for longer.
3. Apply LoRA to more of the model’s attention layers.
4. Extend LoRA into both the attention and feed-forward layers.
5. Give more training weight to the ingredient names and method steps, where much of the answer was expressed.

Within each training run, validation loss was used to choose the checkpoint. It was not used to decide which candidate was best. That decision depended on the responses each candidate generated.

Every candidate answered the same ten development prompts. Their identities were hidden while the responses were evaluated.

A candidate was considered viable only if:

• At least 8 of 10 responses followed the schema.
• At least 7 of 10 passed the complete quality check.
• Every task type contained at least one complete pass.

None of the five candidates met all three requirements.

None of the five adapters passed the product-quality gate. However, we still needed a fine-tuned model to answer the original research question.

We chose Candidate 3 before generating any responses from the held-out test. It was the most useful candidate for understanding what fine-tuning had changed:

• All ten development responses followed the schema.
• Six of ten passed the complete quality check.
• It was the only candidate with complete passes across all five task types.
• It used a simpler adapter than Candidate 4, which achieved the same number of complete passes.

Candidate 3 was not accepted as a product-quality model. It was included as a diagnostic model: something we could evaluate to understand the effect of the fine-tuning work.

This gave us the four systems for the final experiment:

We ran the final experiment once.

Each of the four systems answered the same 60 held-out prompts, producing 240 responses in total.

The complete run used one RTX 4090, the same base model and tokenizer, and the generation settings and seed policy fixed earlier. Each system received one attempt at every prompt.

Once generation began, every response counted. Invalid JSON, incomplete answers and factual mistakes were results—not reasons to try again. We did not repair, regenerate or replace any output.

We then applied the evaluation process defined before training:

1. All 240 responses were checked against the JSON schema.
2. Every schema-valid response was evaluated for answer quality, metaphor and recipe style.
3. The prompt-engineered and fine-tuned responses were compared without revealing which system had produced them.
4. The project author reviewed the fixed human sample so we could measure agreement with the automated evaluator.
5. Input tokens, generated tokens and generation time were recorded for every response.
6. The responses were compared with examples the systems had seen, looking for suspicious reuse.
7. The final differences and uncertainty ranges were calculated.

All 240 scheduled responses completed. The results could describe how the systems performed, but they could not be used to retrain the model, change the prompt or revise the evaluation method.

The experiment was complete. The systems had been fixed, the test had remained separate, and every first attempt had been retained.

This was a controlled comparison, not a universal test of prompting and fine-tuning.

The experiment can tell us:

• How the four systems performed on the same 60 held-out prompts.
• How reliably each system followed the required JSON structure.
• How the evaluated responses compared in answer quality, metaphor and recipe style.
• How many input tokens each approach required.
• How long generation took in the measured setup.
• What Candidate 3 changed in this particular small model.

The experiment cannot tell us:

• Whether the same result would appear with a larger or different model.
• Whether the 60 authored prompts represent every possible user request.
• Whether one automated evaluator represents general human preference.
• Whether one response per prompt captures every way the systems could behave.
• Whether the measured generation time represents complete production latency or serving cost.
• Whether prompting or fine-tuning is always the better approach.

Candidate 3 remained below the product-quality gate. The final evaluation measured what the adapter had changed, but it did not make the candidate suitable for deployment.

These limits do not make the result less useful. They define what the evidence supports:

The Results page shows what we found within those limits.