Synthetic vs real training data: where synthetic actually works

What synthetic data is good at

Augmentation. Adding room impulse responses, noise, speed and pitch variation to real audio genuinely improves robustness and costs almost nothing.

Rare class balancing. If one label is drastically under-represented, generating additional examples can stop the model ignoring it entirely.

Cold start. Bootstrapping an annotation schema, generating seed prompts for human editing, or producing a first pass that humans correct — all reasonable and efficient.

Privacy-sensitive substitution. Where real data cannot legally be used at all, imperfect synthetic data may beat no data.

What it is bad at

Synthetic data can only contain what its generator already knew. That is the whole limitation, and everything else follows from it.

A TTS system generating training audio produces speech with the prosody, phoneme realisation and channel characteristics of that TTS system — not of a farmer on a 2G connection in a market. An LLM generating instruction data produces the distribution of the LLM, including its blind spots, its register and its cultural assumptions.

So synthetic data is systematically weakest at exactly the tail that real collection exists to capture.

The model collapse problem

Training generation after generation of models on the previous generation's output narrows the distribution progressively. Rare events disappear first, then variance shrinks, then the output converges toward a bland mean. The literature calls this model collapse.

It does not require anyone to make an obvious mistake. It happens gradually, through reasonable individual decisions to use available synthetic data, and it is difficult to detect from aggregate metrics because average quality can look stable while diversity quietly disappears.

Why low-resource languages are the worst case

The temptation is strongest exactly where the risk is highest. Real Saraiki data is expensive and slow to collect; generating it is fast and cheap.

But the generator's Saraiki is weak — that is precisely why you need data. Generating training data from a model that handles the language poorly, then training on it, amplifies the existing errors and teaches them as ground truth. You end up with a confident model that is confidently wrong in the same ways as its parent.

A defensible combination

Use real data for the distribution — the speakers, dialects, devices, environments and registers your model must handle. Use synthetic data for augmentation on top of it, and for balancing classes that real collection under-samples.

Keep them labelled separately in your pipeline so you can measure each one's contribution and detect drift. And evaluate on real held-out data always, because evaluating synthetic-trained models on synthetic test sets is how collapse stays invisible until users find it.