Automatic speech recognition (ASR) is the technology that converts spoken language into written text. Feed a model an audio signal (a voice memo, a call-center recording, a command spoken to a car dashboard) and it returns the words that were said. That is the whole ASR meaning in one sentence. The engineering behind it, and especially the data behind it, is where projects succeed or fail.
If you searched for an ASR definition, here it is in practical terms: ASR is a sequence-to-sequence machine learning problem. The input is a continuous acoustic signal sampled thousands of times per second; the output is a discrete sequence of characters, subword tokens, or words. Everything else (voice assistants, dictation, subtitles, IVR systems, meeting notes) is an application built on top of that mapping.
This guide explains how modern speech recognition works, how accuracy is measured, and what kind of training data actually moves the needle. It's written for the ML engineers and data leads who have to ship these systems, not just read about them.
ASR Definition: What the System Actually Does
An ASR system takes raw audio (typically 8 kHz or 16 kHz PCM) and produces a transcript. Classic systems from the 2000s did this with three separate components: an acoustic model (GMM-HMM, later DNN-HMM) mapping audio frames to phonemes, a pronunciation lexicon mapping phonemes to words, and an n-gram language model scoring word sequences. Each part was trained separately and stitched together with a decoder.
That architecture is largely gone. Since roughly 2020, production ASR is dominated by end-to-end neural models that learn the entire audio-to-text mapping in one network. OpenAI's Whisper, Google's Conformer-based models (used in the Universal Speech Model), NVIDIA's Canary and Parakeet, and Meta's wav2vec 2.0 / MMS family are all end-to-end. You still see hybrid systems in latency-critical embedded deployments, but if you're building today, you're almost certainly fine-tuning or training an end-to-end model.
How Does Speech Recognition Work? The Modern Pipeline
Modern ASR has three conceptual stages, even when they live inside a single network:
1. Acoustic front-end. Raw waveform is converted into features, usually 80-dimensional log-Mel spectrograms computed over 25 ms windows with a 10 ms hop. Some models (wav2vec 2.0) learn features directly from the waveform with convolutional layers instead.
2. Encoder. A deep network turns the feature sequence into contextual representations. Today that network is almost always a Transformer or a Conformer, a Transformer variant that adds convolution modules to capture local acoustic patterns. Whisper large-v3 uses a 32-layer Transformer encoder; Conformer encoders dominate the leaderboards for streaming ASR.
3. Decoder. The representations are converted into text. Here there are two main families:
| CTC | Attention / seq2seq (incl. RNN-T) | |
|---|---|---|
| How it decodes | Frame-level predictions, collapses repeats and blanks | Autoregressive decoder attends to encoder output |
| Alignment | Monotonic, implicit | Learned, flexible |
| Streaming | Natural fit (frame-synchronous) | Needs RNN-Transducer or chunked attention |
| Language modeling | Weak internally; often paired with external LM | Strong internal LM behavior |
| Typical use | Low-latency, on-device, keyword systems | Server-side transcription (Whisper is attention-based) |
Many production systems hedge: they train a joint CTC + attention objective (the ESPnet/Icefall recipe) or use an RNN-Transducer, which combines an encoder with a lightweight prediction network and is the workhorse of streaming recognition at Google and Microsoft.
Whisper deserves a specific mention because it shifted expectations. Trained on 680,000 hours of weakly supervised audio scraped from the web, it showed that scale and diversity of training data matter as much as architecture. Its zero-shot robustness on accented and noisy speech, where previous models collapsed, is a data story, not an architecture story.
WER Explained: How ASR Accuracy Is Measured
Word error rate (WER) is the standard metric. It counts the minimum number of substitutions, insertions, and deletions needed to turn the hypothesis into the reference transcript, divided by the number of reference words:
WER = (S + I + D) / N
A WER of 10% means roughly one error per ten words. Three things practitioners should internalize:
- WER is corpus-relative. A model with 2% WER on LibriSpeech test-clean (read audiobooks) can post 25%+ on real call-center audio in the same language. Published benchmark numbers tell you little about your domain.
- Normalization changes everything. Casing, punctuation, number formatting ("25" vs "twenty-five") can swing WER by several points. Always compare systems under the same text normalization.
- For agglutinative or non-space-delimited languages (Finnish, Japanese, Mandarin), character error rate (CER) is often more meaningful than WER.
For voice AI products, also track entity-level accuracy (names, addresses, amounts) and latency. A 9% WER model that nails phone numbers may beat a 7% WER model that garbles them.
What Training Data Does ASR Need?
Supervised ASR training needs paired audio and accurate transcripts. The practical requirements:
- Volume. Fine-tuning Whisper or a Conformer on a new domain typically starts showing gains at 50-100 hours; robust production models for a new language or domain usually want 500-2,000 hours of transcribed speech. Pretraining from scratch needs far more.
- Speaker diversity. 50-200 distinct speakers is a reasonable floor for a model to generalize across voices rather than memorize individuals. Age, gender, dialect, and regional accent coverage matter more than raw hours beyond a point.
- Acoustic diversity. Device types (smartphone, headset, far-field mic), environments (quiet, street, in-car), and sample rates should mirror deployment conditions.
- Transcript quality. Training on transcripts with 5%+ error effectively caps your model's accuracy. Verbatim conventions (fillers, false starts, overlaps) should be consistent across the corpus.
- Clean rights. Every hour needs an unbroken chain of consent and a license that actually permits commercial model training, a growing legal and procurement concern that web-scraped and much open data cannot satisfy. Our guide to buying speech data covers how to evaluate this before you sign.
This is exactly the gap off-the-shelf conversational datasets fill: our catalog covers 60 languages with 500-2,000 hours each, 50-200 native speakers per language, transcribed and consented for commercial ASR training, from US English to Hindi and Swahili.
Common ASR Use Cases
- Voice assistants and IVR. Command recognition and slot filling, latency-sensitive, usually streaming RNN-T.
- Meeting and call transcription. Long-form, multi-speaker, paired with diarization.
- Contact-center analytics. Transcribe 100% of calls for QA, compliance, and intent mining.
- Captioning and subtitling. Broadcast, education, accessibility compliance.
- Medical and legal dictation. Domain vocabulary is the hard part; specialty terms dominate errors.
- In-car voice control. Far-field mics, road noise, strict on-device latency budgets.
- Voice agents. ASR as the front-end of speech-to-speech LLM pipelines, where a single misrecognized entity derails the whole interaction.
Why Conversational Data Beats Read Speech
Most classic training corpora are read speech: someone reads a prompt into a microphone. Read speech is cheap to collect and easy to align, but it is acoustically and linguistically unlike how people actually talk. Spontaneous conversation has:
- Disfluencies. Fillers ("uh," "you know"), repairs, false starts, which read speech almost never contains.
- Faster, reduced articulation. Coarticulation and dropped syllables ("gonna," "d'you") that read-speech models mistranscribe.
- Turn-taking and overlap. Two speakers talking at once, backchannels ("mm-hm") interleaved with content.
- Natural prosody and code-switching. Especially in multilingual markets like India or the Maghreb, where speakers mix languages mid-sentence.
The result: models trained predominantly on read speech (LibriSpeech-style corpora) degrade sharply on conversational input. This is the well-documented gap between test-clean scores and real-world performance. If your product listens to people talking naturally (calls, meetings, voice agents), you need spontaneous conversational training data in the target language and accent. That's the data type our datasets are built on; when your spec is unusual (specific dialects, domains, or acoustic conditions), custom collection fills the rest.
Need training data?
If you're fine-tuning or training ASR models, we offer transcribed, consented conversational speech datasets in 60 languages, priced at $60-95 per hour. Browse the dataset catalog or contact us for samples and language availability.