Word Error Rate (WER) explained, and why yours is misleading

The definition

Word Error Rate is the edit distance between what a model transcribed and what was actually said, normalised by the length of the reference. Formally: WER = (S + I + D) / N, where S is substitutions, I insertions, D deletions and N the number of words in the reference transcript.

It derives from Levenshtein distance applied at word level. Because insertions are unbounded, WER can exceed 100% — a model that hallucinates freely on a short reference will do exactly that.

What counts as a good WER

There is no universal threshold, and quoting one without context is the most common way WER gets misused. As very rough orientation: under 5% on clean read speech in a high-resource language is strong; 10–15% on spontaneous conversational speech is often production-viable; anything above 30% usually means downstream tasks will struggle.

But those numbers move enormously with language, audio condition and task. A 20% WER on spontaneous Saraiki telephony may represent far better engineering than 8% on read English.

Why your reported WER is probably optimistic

Most reported WER figures come from a held-out split of the same corpus the model trained on. That split shares the training data's speakers, recording conditions, dialect distribution and topic range. It measures interpolation, not generalisation.

Real production audio differs on every one of those axes. The gap between benchmark WER and production WER is not a modelling failure — it is a measurement design failure, and it is entirely avoidable by evaluating on a deliberately different distribution.

What actually moves WER in production

Channel and bandwidth: 8 kHz telephony discards spectral information a wideband model relies on. Dialect: a model tuned on the prestige variety degrades on regional ones. Speaking style: spontaneous speech carries disfluency and overlap that read speech does not. Background: market noise, traffic, other speakers. Device: microphone quality varies enormously across handset tiers.

Each of these is a training-data property before it is a model property. Fixing them means collecting audio that contains them, which is why our specs treat device class and noise floor as quota axes rather than incidental details.

When to use CER instead

Character Error Rate applies the same computation at character level. It is more informative for morphologically rich languages where a single wrong affix marks an entire long word incorrect, overstating the practical severity of the error.

CER is also the better choice for languages without clear word boundaries, and for OCR evaluation generally. Reporting both is usually more honest than choosing whichever flatters the result.