How to run a speech data collection project

Start from the deployment condition

Before writing anything, describe precisely what the model will hear in production: which languages and dialects, on what devices, over what channel, in what acoustic environment, from which demographic, speaking in what style.

Every subsequent decision derives from that description. Teams that skip it end up specifying studio-clean audio for a product deployed to phone calls, then wondering why production accuracy does not match the benchmark.

Writing the specification

A workable spec names: target languages and dialects with minimum shares; demographic quotas with tolerances; device classes; recording environments with measured dB(A) ranges; speaking style and rate; code-switch rate if relevant; session structure and turn counts; total volume with delivery milestones; transcription convention; and acceptance tolerances.

It should also state explicitly what is not included. Most delivery disputes trace back to an assumption one party held and never wrote down.

Recruitment and consent

Recruit against quota rather than filling volume first and hoping the distribution works out. Community networks and regional partners reach people that ad networks and app-based crowdsourcing simply do not — rural speakers, older speakers, people on feature phones.

Verify identity to prevent one person operating multiple accounts, screen against quota, test for the task, and consent properly before recording anything. Disclose the pay rate in writing before the contributor accepts.

Capture conditions

Match the spec, and record the conditions as metadata rather than assuming them. Sample rate and bit depth; microphone type and distance; room characteristics; measured noise floor per session; device model.

For far-field work, capture at graded distances — 0.3 m, 1 m, 3 m, 5 m — rather than a single nominal one. For telephony, capture the actual codec path rather than downsampling wideband audio, because downsampling does not reproduce what a real GSM channel does to speech.

Transcription standards

Agree the convention before recording starts. Verbatim or cleaned? Are filled pauses transcribed? How are false starts and repairs marked? What happens with unintelligible spans? Which script for each language in code-switched speech? How are numbers, dates and acronyms written?

Then publish it to annotators and enforce it in QA. For languages with contested orthography — Saraiki, Balochi, Brahui — you must agree a house standard, because there is no external one to default to. Consistency matters more than which convention you pick.

QA gates and acceptance

Automated gates first, within minutes of upload: SNR floor, clipping, DC offset, dropout, silence ratio, duplicate fingerprint, language ID confidence, PII keyword scan.

Then human QA: a second native annotator checking transcripts blind to the first, with disagreements above threshold routed to a senior adjudicator whose rulings update a versioned rubric.

Then acceptance: a defined window against the written tolerances, with out-of-spec work re-collected at the vendor's cost and not invoiced. Without written tolerances this stage becomes an argument rather than a test.

Deliver early and often

The single highest-value process decision is delivering a small batch in week two or three rather than everything at the end.

Specs are always slightly wrong on first contact with reality. Finding that out in week two costs a spec amendment; finding out in week twelve costs a re-collection. Weekly or biweekly incremental delivery makes course correction cheap.