What twelve dollars and one GPU buys: training a dialect model

The Saudi chat model we published cost about twelve dollars fifty to train. That figure invites the wrong conclusion, which is that dialect models are cheap. The accurate conclusion is that compute stopped being the bottleneck a while ago, and everything that is still hard — what data, which baseline, which metric, and whether the metric is lying to you — costs time rather than money.

This is the method, written out, because it is reusable and because the interesting part is the run that failed.

Rule one: baseline everything, including the metric

A fine-tuning result without an untrained baseline on the same test set is not a result. It is a number. Every card we publish carries the base model's score in the same table, because the only honest way to read an improvement is against what the thing scored before you touched it.

Less obvious, and more important: baseline the metric itself before you trust it. We score Saudi dialect by running replies through MARBERTv2, a published classifier with no part in training, and counting what share come back Gulf. Before spending anything we ran it over 400 genuine held-out Saudi replies and 400 Modern Standard Arabic ones.

That second finding is the reason our chat model card carries a reference row alongside the model's score. Without it, 89.8 per cent looks like a model with ten points of obvious headroom. Against a 94.5 ceiling it is a different and more accurate story.

Rule two: the metric will still lie to you, so read the samples

We trained three data mixes and shipped one. Mix B used the enterprise dialogues alone, and by the headline metric it was a success: Gulf replies rose seventeen points over the base model. It was also unusable, and nothing in the aggregate numbers said so.

Asked its opinion on a television for the final, Mix B answered that the system supports the TV and to buy it from the store. It had learned that Arabic sentences are support tickets. Mix C, with everyday conversation added at 34 per cent of the supervised text, answers the same question with something a person would say: والله فكرة، بس أنا ما أحب أشتري شي جديد.

The transferable lesson is not about Arabic. It is that a single aggregate metric on a narrow training set measures whether you moved the thing you optimised, and cannot tell you what you broke to do it. Read fifty samples. It takes twenty minutes and it is the only step in this whole method that has never once been wasted.

Rule three: evaluate on something you did not write

Our chat model's test split comes entirely from a published third-party corpus and never contains our own generated dialogues, in any mix. This is the rule we would least like to relax and the one most often quietly broken elsewhere, because breaking it produces better numbers.

Scoring a generated-data model on more of its own generator's output measures how well the model matched the prompt that produced both. It will give you a high number that means nothing, and you will not be able to tell from the number itself.

Rule four: gate the data, do not assert it

Two gates, both published, both measured on a known-good and a known-bad set first. A Najdi lexicon check for the function words Saudis actually use, and the same dialect classifier the evaluation uses, requiring 60 per cent of a dialogue's replies to read as Gulf. Using the evaluation's own classifier as a training-data filter means the data cannot be worse than the bar the model is later held to.

Two smaller filters did more work than they look like they would. Ballast data from a Modern Standard Arabic corpus is capped by assistant words rather than by rows, because uncapped it is 69 per cent of all supervised text and its answers run three times longer than a spoken reply, which would train the model toward the exact MSA the exercise exists to remove. And anything containing markdown, tables, code fences or HTML is dropped: 15 per cent of that corpus has some, and a voice agent that emits a table makes the speech model read pipes and asterisks out loud.

Rule five: diversity is the constraint, not dialect

Generating data at scale fails through repetition long before it fails through quality. Our grid is eight verticals by four scenarios by four roles, 128 combinations, and at one dialogue per call it started repeating openings almost immediately — duplicates ran 16, then 23, then 24 across three batches and became the largest reject class.

• Generate eight dialogues per call. Make each one different is an instruction a model can follow within one response and cannot follow across independent calls. It is also about five times cheaper per dialogue.
• Randomise the caller, not the scenario. An impatient shop owner, an older man who cannot use the app, a technician who does not know the procedure. Callers vary more than scenarios do.
• Shuffle the grid with a fixed seed. Built vertical-major, a run stopped early has nothing at all for the verticals at the end of the dictionary, silently.
• Stage to disk after every batch. An hours-long run must survive a stop, a crash or a network blip. Three runs were lost before this existed.
• If duplicates dominate again, add scenarios rather than raising the temperature. More ground beats more randomness over the same ground.

After those four changes: 487 staged dialogues with 487 unique openings and 1,953 unique replies out of 1,959.

Rule six: small, and cheap enough to rerun

0.6B to 4B parameters, one L4, a few dollars a run. The value of being this cheap is not the saving; it is that a failed run costs eight dollars and half a day, so you can afford to have one. Mix B was a failure we would not have run if the run had cost thousands, and Mix C exists because of what Mix B taught us.

Two pieces of operational grit, since they cost us real money. Idle time between runs was the single largest waste — delete the VM the moment the model is off it, because the disk bills until the VM is deleted rather than stopped. And GGUF quants must be uploaded one file per commit: a single commit for all six failed on the F16 and took the four good quants with it.

The method, in one list

• Baseline the untrained model on the same test set, and publish it next to the result.
• Baseline the metric on known-good and known-bad data before you trust it, and publish the ceiling it implies.
• Build your own data where the open data does not cover the job, and gate it with the same classifier the evaluation uses.
• Keep the test split entirely third-party, always.
• Read the samples even when the metric says you won, especially then.
• Stay small enough that a failed run is affordable, because you will have one and it is where the learning is.

This is documented on every model card we publish rather than living in a playbook nobody outside sees, and the next language runs the same way. If a step in it looks wrong to you, the cards carry the numbers to argue with.