What the work taught us.

We learned that asking whether fine-tuning makes a model ‘better’ is too vague.

We learned this by considering responses that succeeded in one way but failed in another. A response could follow the correct format while giving poor advice, or give a useful answer that the website could not display. A system could also use fewer tokens while taking longer to respond.

We therefore measured structure, usefulness, metaphor, recipe quality, prompt length, and speed separately.

We learned that comparing a prompted base model directly with a minimally prompted fine-tuned model does not explain why their responses differ.

We learned this by noticing that the comparison changed both the model and the prompt. We added two more systems so we could test base and fine-tuned models with both short and detailed prompts.

We also tested four versions of the detailed prompt. This gave fine-tuning a strong baseline rather than an easy one.

We learned that a model name does not fully describe what produces a response.

We learned this when one runtime silently added its own instructions, and when the model placed otherwise valid JSON inside Markdown code blocks that the website could not use. We also found that shortening the detailed prompt reduced its quality.

The prompt, chat template, runtime settings, tokenizer, and output handling all affected the result.

We learned that reaching 200 examples did not make the dataset complete.

We learned this by checking its coverage, quality, repetition, token lengths, and recipe variety. An audit found seven examples that were too close to earlier work or to one another. Replacing them then accidentally removed the only recipes with seven and eight ingredients.

A small change to a few examples could alter the balance of the whole dataset.

We learned that a model can improve during training without showing the behaviour we want.

We learned this when the first adapter trained correctly, reloaded correctly, and achieved lower validation loss, but produced no correctly structured responses on ten development prompts.

Later candidates produced 7, then 9, then 10 correctly structured responses out of 10. Complete successes improved more slowly: from 3, to 4, to 6. The model learned the ChatG&T format before it reliably learned to give good answers inside it.

We learned that response structure can be measured more confidently than answer quality.

We learned this during the final evaluation of 240 first responses. The prompted system produced 43 correctly structured responses and 16 complete successes. The fine-tuned system produced 52 correctly structured responses and 26 complete successes.

Structure could be checked directly. Usefulness required judgment. A human reviewer and the automated evaluator usually agreed about recipe style, but agreed on only half of the underlying-answer decisions.

We learned that fine-tuning cannot be described with one score.

We learned this when it improved structural reliability and reduced the average input from about 2,576 tokens to 32, but increased generation time from 3.92 to 5.70 seconds.

We also found that adding the detailed prompt to the fine-tuned model reduced complete successes from 26 to 21. Prompted and learned instructions did not simply add together.

We learned that the training loop is only a small part of a real machine-learning experiment.

We learned this when GPUs with enough memory still failed because of incompatible drivers, and when machines created from the correct container image failed while the provider’s official template worked.

The first three candidate runners used 113 seconds and about one cent of compute. The complete session cost about twelve cents because provisioning, installation, downloads, checks, transfers, and cleanup took longer than training.

We learned that putting a trained model online is not simply a matter of copying the training files.

We learned this when the adapter contained a path that existed only on the training machine. We replaced that path for publication while keeping the learned weights unchanged.

ZeroGPU also used a different PyTorch version and did not provide a GPU until a request arrived. The private service credential had to remain on the server rather than being sent to the browser. Deployment introduced its own compatibility, security, and failure boundaries.

We learned that more process does not always produce better evidence.

We learned this when an early audit produced 41,957 possible matches for review despite finding no automatic failures. The process had become larger than the risk it was meant to control, so we preserved the failed attempt and replaced it with a smaller audit.

We applied the same discipline to training. Our strongest candidate achieved six complete successes when the required minimum was seven. Two more candidates also failed, so we stopped after five instead of lowering the target or continuing until something passed.

We learned that Codex can complete a complex workflow autonomously when it has tools to act, skills that explain how to use them, and clear limits on what it may change.

We learned this by giving Codex access to Runpod through its CLI, along with the project’s Runpod skills and a fixed experimental plan. Codex provisioned the GPUs, recovered from failed machines and incompatible drivers, ran and monitored all five training candidates, verified the results, retrieved the evidence, and stopped the billing.

It did not simply follow a prepared list of commands. It handled unexpected problems while keeping the model, data, budget, and experimental rules unchanged.