Code-switching breaks speech recognition. Here is how to fix it

What code-switching actually is

Code-switching is alternating between two or more languages within a single conversation, often within a single sentence. “Meeting kal reschedule ho gayi hai, but I'll send the update” is entirely ordinary speech for a bilingual professional in Karachi or Lahore.

It is not error, laziness or incomplete language acquisition. It is a systematic, rule-governed register used by the majority of bilingual speakers worldwide — and any model deployed to them will encounter it constantly.

Why monolingual ASR fails

Three failure modes recur. The language-identification layer flips mid-utterance and the decoder switches acoustic models mid-word, producing garbage at the boundary. Or the model forces English words through the phoneme inventory and language model of the matrix language, producing confident nonsense. Or the language model, trained on monolingual text, assigns near-zero probability to the mixed sequence and the decoder discards the correct hypothesis.

The result is that accuracy degrades most on exactly the utterances that are most natural for your users.

Specify a switch rate, do not hope for one

The critical move is treating code-switching as a quota parameter. Set an explicit target: for example, 30–40% of collected utterances should contain at least one content word from the embedded language, with switches distributed across the utterance rather than clustered at boundaries.

Then design prompts and recruitment to hit it. Left to chance, collection under-represents switching, because contributors in a recording context tend to shift toward a more formal, more monolingual register than they use naturally.

Transcription conventions matter

Code-switched audio is only useful if the transcript preserves the switch points recoverably. That requires a documented convention agreed before collection begins: which script each language is rendered in, whether embedded words are transliterated or kept in their native script, and how language tags are marked.

Consistency matters far more than which convention you choose. An inconsistently tagged code-switched corpus is close to useless for training, and inconsistency is what you get by default when the convention is not written down and enforced in QA.

Elicitation without artificiality

The practical difficulty is that people code-switch less when they know they are being recorded. Reading prompts aloud produces almost none of it.

What works: two-party conversation between people who habitually switch with each other, topics drawn from domains where the embedded language dominates the vocabulary — technology, workplace, medicine, finance — and long enough sessions that participants relax past their initial formality. Role-play scenarios grounded in real tasks also work well.