Speaker diarization is the task of taking an audio recording and answering "who spoke when": partitioning the timeline into segments and assigning each segment a speaker label. The labels are anonymous (Speaker A, Speaker B) because diarization does not need to know who the speakers actually are, only that they are different people.
That distinction separates diarization from speaker identification (matching a voice against known, enrolled speakers) and speaker verification (confirming a claimed identity). Diarization is unsupervised at inference time: the system usually doesn't even know how many speakers are present.
Diarization is what turns a raw transcript into a usable record of a conversation. Meeting notes, call-center analytics, medical scribing, and conversational training-data preparation all depend on it. It also remains meaningfully harder than transcription itself, because natural conversation is full of overlaps, interruptions, and half-second back-channels.
What is speaker diarization? Definition and output
Given an audio file, a diarization system outputs a list of (start time, end time, speaker label) tuples, e.g.:
0.00-4.20 SPEAKER_00
4.05-9.80 SPEAKER_01
9.80-10.30 SPEAKER_00
Note the overlap between 4.05 and 4.20: both speakers active at once. Whether a system can even represent overlap is one of the main dividing lines between architectures. Combined with an ASR transcript, this yields speaker-attributed transcription: who said what, when.
The classic pipeline: VAD → embeddings → clustering
Most production diarization systems are modular pipelines with four stages:
- Voice activity detection. Find speech regions and discard silence and noise (see our VAD guide).
- Segmentation. Cut speech into short chunks, ideally at speaker-change points; modern systems (pyannote) use a neural segmentation model that also detects overlapping speech.
- Speaker embeddings. Map each chunk to a fixed-length vector that encodes voice identity. The historical progression: i-vectors → x-vectors (TDNN-based, 2018) → ECAPA-TDNN, the current workhorse, trained on large speaker-verification corpora like VoxCeleb so that same-speaker chunks land close together in embedding space.
- Clustering. Group embeddings by speaker, typically with agglomerative hierarchical clustering or spectral clustering, estimating the number of clusters from the data unless the caller supplies it.
This pipeline is robust and interpretable, but it has a structural weakness: steps 3-4 assume each chunk contains exactly one speaker. Overlapping speech violates that assumption, which is why classic pipelines historically just lost the overlapped speaker.
End-to-end neural diarization (EEND)
End-to-end neural diarization replaces the pipeline with a single network that ingests audio and directly outputs, for each frame, the activity of each speaker. Overlap becomes a first-class citizen rather than an afterthought. Training uses permutation-invariant loss, since speaker label order is arbitrary.
In practice the field has converged on hybrids: a local EEND-style model handles segmentation and overlap within short windows, and clustering stitches speaker identities across the full recording. pyannote 3.x works this way (its "powerset" segmentation model classifies each frame into speaker combinations, including overlap sets), and NVIDIA's Sortformer (in NeMo) is a prominent end-to-end model that resolves the permutation problem by sorting speakers by arrival time. Pure EEND still struggles with long recordings and unbounded speaker counts, which keeps hybrid designs dominant.
Speaker diarization tools in 2026
| Tool | Approach | Strengths | Notes |
|---|---|---|---|
| pyannote.audio 3.x | Neural segmentation (powerset, overlap-aware) + ECAPA embeddings + clustering | Best open-source accuracy/effort ratio; large community | ~10-20% DER on common benchmarks; GPU recommended; gated model access via Hugging Face |
| NVIDIA NeMo | Clustering pipelines, MSDD, Sortformer (end-to-end) | Strong overlap handling, GPU-optimized, streaming variants | Heavier engineering lift |
| WhisperX | Whisper ASR + Silero VAD + pyannote diarization | One-command speaker-attributed transcripts; fast batched inference | Diarization quality is pyannote's; alignment adds word-level timestamps |
| Commercial APIs (AssemblyAI, Deepgram, Speechmatics, Google, AWS) | Proprietary | No infrastructure; diarization bundled with ASR | Speaker-count limits and overlap behavior vary |
Does Whisper support speaker diarization?
No. Whisper does not diarize. This is one of the most common misconceptions about it. Whisper is a sequence-to-sequence transcription model: audio in, text (with timestamps) out. It has no speaker representation anywhere in its architecture and will happily merge five speakers into one undifferentiated transcript.
The standard workaround is to run diarization alongside Whisper and merge the outputs: run pyannote to get speaker-labeled time segments, run Whisper to get word-level timestamps (Whisper's native segment timestamps are too coarse, so tools force-align them), then assign each word to the speaker segment it falls inside. WhisperX packages exactly this (batched Whisper inference, forced alignment for accurate word timing, and pyannote diarization) and is the most popular way to get "Whisper with speakers."
Measuring accuracy: Diarization Error Rate (DER)
DER is the fraction of audio time attributed to the wrong speaker, computed against a reference annotation:
DER = (missed speech + false alarm speech + speaker confusion) / total reference speech time
- Missed speech: reference has speech, system says nothing (overlaps are the usual culprit).
- False alarm: system claims speech where there is none.
- Speaker confusion: speech detected, but assigned to the wrong speaker.
Scoring conventions matter enormously when comparing published numbers: many older results used a ±250 ms "collar" around reference boundaries and excluded overlap regions, both of which flatter the score. Modern benchmarks (DIHARD) score with no collar and include overlap. On conversational benchmarks like CallHome, AMI, and DIHARD III, state-of-the-art systems typically land somewhere around 10-20% DER, meaning even good systems misattribute one word in every six to ten on hard audio. There is also JER (Jaccard Error Rate), which weights all speakers equally rather than by talk time.
Why natural conversation is the hard case
Diarization demos look great on podcasts: two speakers, good microphones, polite turn-taking. Real conversation breaks every one of those assumptions:
- Overlap is constant. In spontaneous dialogue, speakers overlap during turn transitions and back-channels; depending on the corpus, overlapped speech is commonly on the order of 5-20% of speech time. Single-channel systems must detect and separate it; many simply miss it.
- Back-channels are short. "mm-hmm", "yeah", "right" last 200-500 ms, often shorter than the analysis window used for embeddings, so they get absorbed into the other speaker's turn.
- Similar voices confuse clustering. Two same-gender speakers with similar vocal characteristics on a narrowband phone line push embedding clusters together.
- Channel effects. Telephony codecs strip the spectral detail that embeddings rely on; far-field reverb smears speaker cues.
This is why evaluation on scripted or read speech tells you nearly nothing about production performance, a theme we cover in depth in conversational vs. scripted speech data.
Practical tips for better diarization results
A few adjustments consistently move DER more than swapping models:
- Pass the speaker count if you know it. Estimating the number of speakers is one of the largest error sources in clustering; pyannote and most APIs accept
num_speakers(or min/max bounds) and improve markedly when given them. - Feed the best channel you have. If your source is dual-channel, diarize before mixing down. Better yet, skip diarization entirely, since channel separation already answers "who spoke when."
- Mind the audio front-end. Aggressive noise suppression and AGC can erase the speaker cues embeddings depend on; diarize the least-processed audio available.
- Evaluate on your own audio. Published DERs come from specific benchmarks with specific scoring rules. Twenty minutes of hand-labeled production audio, scored with
pyannote.metricsordscore, tells you more than any leaderboard. - Reconcile word boundaries carefully when merging with ASR. Most "diarization errors" users see in speaker-attributed transcripts are actually word-to-segment assignment errors at turn boundaries, exactly where timestamps from both systems are least reliable.
The training-data angle: dual-channel audio is ground truth
Building or fine-tuning diarization systems runs into a practical problem: the training labels (precise per-speaker activity, including overlaps) are exactly what's expensive to annotate by hand. Human annotators marking speaker turns on a single mixed channel make the same mistakes the models do, especially on overlap boundaries.
Dual-channel recording solves this at the source. When each speaker is captured on their own channel, as in telephone conversations recorded with per-side channels, per-speaker voice activity can be derived directly from each channel, giving frame-accurate "who spoke when" labels including overlap regions, with no annotator judgment calls. Mix the channels down and you get realistic single-channel diarization training input with near-perfect reference labels attached.
This is precisely how our datasets are built: spontaneous two-party conversations, one channel per speaker, with time-aligned transcripts and speaker metadata, across 60 languages, from US English to Hindi and Arabic (MSA). If you're deciding how much of it you need, our guide on how much audio data you need is a good starting point.
Need training data?
If you're training or benchmarking diarization, dual-channel conversational audio gives you per-speaker ground truth that manual annotation can't match, and we license it off the shelf in 60 languages, 500-2,000 hours per language. Explore the dataset catalog or get in touch to talk through your setup.