Research
Why AI models fail on low-resource languages
Low-resource languages are a data problem, not a modelling problem. What actually breaks, why speaker count and corpus size are unrelated, and what fixes it.
Key takeaways
- Speaker population and digital corpus size are almost unrelated. Punjabi has roughly 90 million speakers in Pakistan and remains low-resource.
- Architectures and scaling laws transfer across languages. Data does not, because it was never collected.
- Tokenizers trained on English-dominant corpora fragment other scripts badly, inflating cost and degrading quality before training even begins.
- The failure is usually invisible in benchmarks and highly visible in production churn.
“Low-resource” does not mean “few speakers”
The term causes constant confusion. A low-resource language is one with little machine-readable data available, regardless of how many people speak it. Punjabi has roughly 90 million speakers in Pakistan alone and is one of the most spoken languages on earth — and it is unambiguously low-resource in AI terms.
Speaker count and corpus size are driven by completely different forces. Corpus size tracks internet penetration, publishing infrastructure, literacy in the written standard and commercial incentive to digitise. None of those follow population.
The failure is data, not architecture
This is worth stating plainly because a great deal of engineering effort gets wasted on the wrong layer. Transformer architectures transfer across languages. Scaling laws transfer. Training recipes transfer. What does not transfer is data that was never collected in the first place.
There is no Common Crawl of spoken Sindhi. There is no forum archive of Saraiki medical advice. There is no corpus recording how a Pashto speaker actually asks a bank for help. A model cannot learn a distribution nobody ever wrote down.
What breaks first: tokenization
Before training even starts, tokenizers cause damage. A vocabulary fitted mostly to English and other Latin-script languages fragments Perso-Arabic, Devanagari and other scripts into far more tokens per word than it would for English.
The consequences compound: longer sequences for the same content, more compute per example, effectively shorter usable context windows, higher inference cost for those users, and weaker representations because meaning is spread thinly across many sub-word fragments.
What breaks next: the tail
Even where some data exists, it is usually skewed. Open corpora tend to over-represent read speech, formal register, the prestige dialect, urban speakers and good recording conditions — because those are cheapest to collect.
So the model performs well on newsreader-style input and poorly on a farmer in a rural district calling from a budget handset in a regional dialect. Your benchmark, drawn from the same skewed distribution, reports success. Your churn numbers disagree.
What actually fixes it
Quota-controlled collection: define the demographic and acoustic distribution you need before recruiting, and reject out-of-quota submissions even when they are technically clean. Native authoring rather than translation for text, because translated instruction data teaches a model to answer local questions with foreign assumptions.
And realistic expectations about transfer. Related languages help — Urdu speech data measurably improves Hindi ASR — but relatedness in speech does not imply relatedness in script, and a Dravidian language surrounded by Indo-Iranian ones like Brahui gets almost nothing from its neighbours.
Answers
Frequently asked questions
A language with little machine-readable training data available, regardless of speaker population. Corpus size depends on internet penetration, publishing infrastructure and commercial incentive to digitise — not on how many people speak the language. Punjabi, with roughly 90 million speakers in Pakistan, is a standard example.
Partially. Cross-lingual transfer from related high-resource languages gives a real head start, and multilingual pretraining is why fine-tuning now needs hundreds rather than thousands of hours. But transfer plateaus quickly, and the remaining gap is precisely the language-specific behaviour no related language encodes.
Because the benchmark is usually drawn from the same skewed distribution as the training data. If both over-represent urban, educated, clear-audio speakers, the evaluation confirms what the training assumed. Evaluating on a deliberately different distribution is the fastest way to find out.
Which Asian language is your model failing on?
Tell us the language and the failure mode. We will come back with a collection design and an honest view of what data will and will not fix.
Average first response: under 6 business hours. NDAs signed same day.