Language coverage
Urdu training datasets
Urdu has well over two hundred million speakers and a fraction of the machine-readable data that number implies. What exists openly, why it underperforms in production, and what a commissioned Urdu specification has to state.
Key takeaways
- Urdu is not low-resource because it is small. It is low-resource because its speakers write and search in ways the web crawl does not capture.
- Four things break Urdu models: Nastaliq typography, English code-switching, register diglossia, and confusion with Hindi.
- Open Urdu corpora are mostly read speech and scraped news. Production hears spontaneous, code-switched, telephone-band Urdu.
- Roman Urdu is how much of the population actually types. A corpus without it will fail on real user input.
- The transcription convention matters more than the size of the order. Write it down before anyone records anything.
Why Urdu is under-resourced
By speaker count Urdu belongs in the same conversation as the languages every frontier model handles competently. By data volume it does not come close, and the reason is not obscurity. It is that the ordinary written output of Urdu speakers does not land where crawlers look.
A great deal of everyday Urdu is spoken rather than written. A great deal of what is written is written in Roman script in messaging apps that no crawler sees. What remains in Nastaliq on the public web skews heavily towards news, religious material and literary prose, which is a narrow and formal slice of the language. The result is a corpus that over-represents the register a model will almost never be asked to handle, and under-represents the one it will face constantly.
This is why scaling up a web crawl does not close the gap. The missing data was never on the web to begin with. It has to be collected.
The four things that break Urdu models
Nastaliq is not Arabic with different words. The script runs on a sloping baseline, ligatures reshape glyphs according to deep context, and characters overlap vertically in ways that defeat segmentation logic built for Naskh. OCR pipelines that handle Arabic well frequently collapse on printed Urdu, and handwritten Nastaliq is harder again.
Code-switching is the default, not an edge case. Educated urban Urdu mixes English constantly, at the word and phrase level, inside single utterances. A model trained on clean monolingual Urdu will mis-recognise the English fragments and a model trained on clean English will mis-recognise everything around them. Only a corpus that contains the mixture in its natural proportion teaches the right behaviour.
Register diglossia is wide. Formal written Urdu draws on Persian and Arabic vocabulary that a colloquial speaker would not use or always understand. Train on news and literature and you build a model that writes beautifully and misunderstands its users.
Hindi is a helpful prior and a dangerous substitute. Spoken colloquial Hindi and Urdu overlap enough that acoustic transfer works. The scripts share nothing, and the formal lexicons diverge sharply. Using Hindi text as an Urdu substitute produces a model that is subtly wrong in ways evaluation on translated benchmarks will not catch.
What is available openly, and what it is good for
Open Urdu resources are worth using. They are just worth using for the right thing.
| Resource type | Genuinely useful for | Where it fails |
|---|---|---|
| Open read-speech corpora | Bootstrapping an acoustic model, phoneme coverage | Narrow speaker demographics, no spontaneity, no telephone band |
| Web-crawled Nastaliq text | Pretraining, vocabulary coverage, language modelling | News and religious skew, formal register only, duplication |
| Translated benchmark sets | Rough comparability across languages | Translationese; flatters models that never saw native Urdu |
| Community transliteration lists | Roman to Nastaliq mapping as a starting point | Inconsistent conventions, thin coverage of real spellings |
| Multilingual model checkpoints | A prior worth fine-tuning from | Inherits every gap above, and hides it behind fluent output |
The pattern is consistent. Open data gets you to a model that is fluent and unrepresentative. Closing the last distance to production is a collection problem, and it is the only part that has to be commissioned.
Urdu datasets we collect
Everything below is collected in Pakistan by local teams, with consent recorded per contributor and a double-blind quality pass before delivery. Each links to its own specification page.
| Dataset | Modality | Typical use |
|---|---|---|
| Conversational Urdu speech | Speech | Voice agents, call analytics, meeting transcription |
| Pakistani telephony corpus | Speech, 8 kHz | Contact-centre ASR, IVR, fraud and quality analytics |
| Pakistani-accented English | Speech | Accent robustness for English ASR |
| Wake word and command set | Speech | Keyword spotting, on-device voice control |
| Nastaliq and handwriting OCR | Image and text | Document AI, forms, receipts, prescriptions |
| Native Urdu instruction and SFT corpus | Text | Supervised fine-tuning of Urdu-capable LLMs |
| Urdu RLHF preference pairs | Text | Reward modelling, preference optimisation |
| Urdu–English parallel corpus | Text | Machine translation, cross-lingual alignment |
| Safety and toxicity lexicon | Text | Guardrails, moderation, red-teaming |
Writing an Urdu spec that survives QA
Most Urdu projects fail on convention rather than on volume. These are the decisions to make in writing, before recording starts.
- Script policy. Nastaliq, Roman, or both as parallel fields. If both, say which is canonical.
- Code-switch convention. How English fragments are rendered, with at least three worked examples covering a single word, a phrase, and a full clause.
- Register targets. The proportion of colloquial, semi-formal and formal speech you want, rather than whatever the recruiter happens to find.
- Quota axes. Age band, gender, city, urban or rural, and education level where it affects register. State the tolerance on each.
- Acoustic conditions. If production is telephony, specify narrowband capture. Wideband studio audio downsampled after the fact is not the same signal.
- Numbers, dates and named entities. Written as spoken or normalised. Pick one; inconsistency here poisons ASR training quietly.
- Diarisation standard. Whether overlapping speech is labelled, and how turn boundaries are decided.
- Acceptance test. The measurement you will run on delivery, the threshold, and what happens if it is missed.
Eight decisions. Made up front they cost an afternoon; made after the first delivery they cost the delivery.
FAQ
Questions buyers ask about Urdu data
Spoken Hindustani at a colloquial register overlaps heavily, so acoustic models transfer better than people expect. Text does not transfer at all without transliteration, because the scripts differ, and formal registers diverge sharply in vocabulary. Treat Hindi as a useful pretraining prior for Urdu speech and as a poor substitute for Urdu text.
Most open Urdu audio is read speech recorded in quiet conditions by a narrow demographic, and most open Urdu text is scraped news and religious material. Production systems hear spontaneous, code-switched, telephone-band speech from a much wider population. The mismatch is distributional, so more of the same open data does not fix it.
Pick one convention, write it down with worked examples, and hold every transcriber to it. The usual choice is to render English words in Latin script inside the Urdu line rather than transliterating them into Nastaliq. What matters far less than the choice itself is that the corpus is internally consistent, because an inconsistent convention teaches the model noise.
Yes. Roman Urdu is how a large share of Pakistani users actually type, so systems that only see Nastaliq fail on real input. It can be delivered as a parallel field alongside the Nastaliq line so a model can be trained or evaluated on either.
A pilot of a few hours per quota cell for speech, or a few hundred double-labelled items for text and annotation. That is enough to measure agreement and expose spec problems, and small enough to discard if the pilot shows the spec was wrong.
Get Urdu samples in one business day.
Spontaneous, code-switched, telephone-band if you need it, with the consent record attached. Nothing reaches you ungated.