Voice activity detection (VAD) is the task of deciding whether a given slice of audio contains human speech. A VAD model consumes short frames, typically 10-100 ms of audio, and outputs a binary label or a speech probability for each one. Stitched together, these frame-level decisions tell a system exactly where speech starts and stops in a recording or a live stream.
That sounds almost trivially simple, and that is exactly why VAD is everywhere: it is the first gate in nearly every speech pipeline. Automatic speech recognition (ASR) systems use it to skip silence and cut compute costs. Voice agents use it to detect when a caller starts and stops talking. Telephony stacks use it to save bandwidth by not transmitting silence. Diarization and wake-word systems use it as a front-end filter.
This guide covers how voice activity detection works, from energy heuristics to neural models like Silero VAD, where it sits in ASR and voice-agent pipelines, how it's evaluated, and why the quality of a VAD model comes down almost entirely to the audio it was trained on.
What is voice activity detection?
Formally, VAD is a frame-level binary classification problem: speech vs. non-speech. "Non-speech" is the hard part, because it includes everything else: silence, room tone, keyboard clicks, dogs barking, music, and the genuinely difficult cases like background TV chatter or a second conversation across the room, which is speech but not the speech you want.
A practical VAD has three components:
- A feature extractor that converts raw audio frames into something a classifier can use (energy, spectral features, or learned representations).
- A classifier that outputs a speech probability per frame.
- Post-processing (hangover/hysteresis logic) that smooths flickering frame decisions into stable speech segments. For example: don't end a segment until you've seen 200 ms of continuous silence, and pad segment boundaries slightly so you don't clip word onsets.
That third step matters more than people expect. A VAD that chops off the first 80 ms of an utterance will silently degrade ASR accuracy on plosives and short function words, and no amount of downstream modeling fixes it.
How does voice activity detection work? Heuristics vs. ML
Energy and spectral heuristics
The oldest VADs threshold on short-term energy: if a frame is louder than an adaptive noise-floor estimate, call it speech. Refinements add zero-crossing rate (voiced speech has low ZCR; fricatives and noise have high ZCR), spectral flatness, and pitch detection. Standards like ITU-T G.729 Annex B and the GSM/AMR VADs used exactly this family of features for decades of telephony.
Heuristic VADs are essentially free computationally and easy to reason about, but they degrade badly in low signal-to-noise conditions. Loud music looks like speech; quiet, breathy speech looks like silence. WebRTC VAD, the GMM-based detector extracted from Google's WebRTC stack, is the best-known member of this generation. It runs in microseconds per frame, but published comparisons consistently show it trading far more false positives for the same sensitivity than modern neural VADs; in one widely cited benchmark, at a fixed 5% false-positive rate WebRTC VAD caught only around half of speech frames while Silero VAD caught close to 88%.
ML-based VAD
Modern VADs are small neural networks trained on large amounts of labeled speech/non-speech audio:
- Silero VAD has become the de facto open-source default: a model under ~2 MB that processes audio in ~30 ms chunks, runs faster than real time on a single CPU thread, and holds up well on noisy, far-field, and telephony audio. Recent versions (v4/v5) specifically improved robustness to music and whispered speech.
- pyannote.audio's segmentation model goes further: it's trained not just to find speech but to handle overlapping speech, which is why it anchors the pyannote speaker diarization pipeline.
- Most commercial ASR APIs and voice-agent frameworks embed a proprietary neural VAD; several newer "semantic VAD" or turn-detection models also look at what is being said to judge whether the speaker is actually done.
| Approach | Examples | Compute | Noise robustness | Typical use |
|---|---|---|---|---|
| Energy/spectral heuristics | G.729B, AMR VAD | Negligible | Poor | Legacy telephony, bandwidth saving |
| Statistical (GMM) | WebRTC VAD | Very low | Moderate | Embedded, latency-critical filters |
| Neural frame classifiers | Silero VAD, Picovoice Cobra | Low (CPU real-time) | Good | ASR pre-processing, voice agents |
| Neural segmentation w/ overlap | pyannote segmentation | Moderate | Very good | Diarization, meeting transcription |
Where VAD sits in ASR and voice-agent pipelines
In an offline transcription pipeline, VAD is a pre-processor: it splits long recordings into speech segments so the ASR model only transcribes actual speech. This is both a cost and a quality lever. Whisper-family models famously hallucinate text on long silences, and running them only on VAD-detected speech (as WhisperX does with Silero VAD) largely eliminates that failure mode while speeding up batch transcription dramatically.
In a real-time voice agent, VAD does three jobs, and each one has a latency budget:
- Endpointing. The agent needs to decide the user has finished their turn. The standard approach: wait for N milliseconds of VAD silence (commonly ~300-800 ms) before finalizing the utterance. Set N too low and you cut users off mid-sentence; too high and the agent feels sluggish, because endpoint delay is added directly to total response latency on every single turn.
- Barge-in. When the user starts talking over the agent's TTS output, VAD (combined with echo cancellation so the agent doesn't detect its own voice) triggers the agent to stop speaking and listen. Barge-in detection typically needs to fire within ~100-300 ms to feel natural.
- Cost gating. Streaming ASR and LLM tokens are billed by usage; VAD ensures you only stream audio that contains speech.
The consequence: a voice agent's perceived snappiness is often limited less by the LLM than by VAD/endpointing tuning. Teams routinely spend more time tuning silence thresholds against real call audio than tuning prompts.
How VAD is evaluated: false alarms vs. missed speech
VAD evaluation is a tradeoff between two error types:
- False alarm rate (false positives): non-speech frames labeled as speech. Cost: wasted ASR compute, hallucinated transcripts, agents that interrupt themselves because of a door slam.
- Missed speech rate (false negatives): speech frames labeled as silence. Cost: clipped words, dropped utterances, an agent that never hears the user's quiet "yes."
Because you can trade one for the other by moving the decision threshold, honest comparisons report ROC/DET curves or fix one rate and compare the other, rather than quoting a single "accuracy" number. Segment-level metrics also matter: boundary precision (how close detected onsets/offsets are to true ones, usually within a tolerance collar of ±100-200 ms) is what determines whether words get clipped.
Which error is worse depends entirely on the application. A wake-word front-end should be miss-averse; a transcription cost filter can tolerate misses of marginal frames but not false alarms on an hour of music.
Why VAD models live and die by their training data
Every VAD failure mode in production traces back to a distribution gap in training data:
- Conversational speech isn't read speech. Spontaneous dialogue is full of hesitations, fillers ("uhh", "mm-hmm"), laughter, trailing off, and turn overlaps. A VAD trained mostly on clean read speech will mis-time boundaries on all of these, and boundary timing is the whole product. The differences are big enough that we wrote a full comparison of conversational vs. scripted speech data.
- Boundary labels must actually be accurate. VAD training targets are the speech/non-speech boundaries themselves. Transcripts that are time-aligned only at the utterance level, or aligned sloppily, teach the model sloppy endpointing.
- Channel and language diversity. Telephony codecs, far-field reverb, and language-specific phonetics (e.g., different pause patterns and phrase-final devoicing across languages) all shift what "speech" looks like acoustically.
This is where dataset design matters. Dual-channel conversational recordings, one channel per speaker as in our off-the-shelf datasets, give you clean per-speaker signals from which precise speech/non-speech boundaries and overlap regions can be derived, across languages from US English to Hindi and Arabic (MSA). If you need a specific acoustic condition (a particular telephony stack, in-car audio, specific noise profiles), custom collection exists precisely to close that gap.
Choosing a VAD: practical guidance
- Default choice: Silero VAD. It's free, small, fast on CPU, and accurate enough for most ASR pre-processing and voice-agent endpointing.
- Extreme resource constraints (MCUs, per-packet telephony decisions): WebRTC VAD or a similarly tiny detector, accepting the accuracy hit.
- Meetings and overlapping speech: pyannote's segmentation model, usually as part of a full diarization pipeline.
- Voice agents: whatever VAD you pick, budget real tuning time for endpoint silence thresholds and barge-in behavior against recordings that match your production audio, and evaluate on conversational test data, not clean clips.
Need training data?
If you're training or evaluating VAD, endpointing, or full speech pipelines, we license conversational speech datasets in 60 languages: dual-channel audio with time-aligned transcripts, speaker metadata, and a full consent chain. Browse the dataset catalog or contact us to discuss your target languages and acoustic conditions.