Speech synthesis is the artificial production of human speech: a system takes text (or another representation of language) and generates an audio waveform that sounds like a person saying it. The most common form is text-to-speech (TTS), the technology behind screen readers, voice assistants, audiobook narration, IVR systems, and the speaking half of every voice agent. The field also includes voice conversion and singing synthesis, but when engineers say "speech synthesis," they almost always mean TTS.
For most of its history, synthesis was an engineering problem: hand-built rules, spliced recordings, clever signal processing. Since roughly 2016 it has become a machine learning problem, and today's best neural TTS systems reach human-level naturalness on read sentences. That shift changed what determines quality. The bottleneck is no longer the vocoder or the acoustic model (architectures are largely commoditized) but the speech data the model learns from.
This post walks through how speech synthesis works, how it evolved, how modern neural TTS is built and evaluated, and why the data requirements changed so dramatically with each generation.
A Short History of Speech Synthesis
Four eras cover ninety years of the field, and each one is defined by how it represents speech:
Formant synthesis (1930s-1980s). The earliest electronic systems, from Bell Labs' Voder demonstrated in 1939 to Dennis Klatt's work that became DECtalk in the 1980s, generated speech from acoustic rules. Engineers modeled the resonant frequencies (formants) of the vocal tract and drove an oscillator-and-filter pipeline with hand-written rules per phoneme. No recordings required, and it showed. Formant synthesis is intelligible but unmistakably robotic (it is the voice Stephen Hawking used). Its virtues were a tiny footprint and unlimited flexibility, which kept it alive in embedded devices for decades.
Concatenative synthesis (1990s-2000s). Instead of modeling speech, store it. Concatenative systems recorded one voice actor reading a large scripted corpus, chopped the audio into units (diphones, later variable-length units), and stitched the best-matching units together at runtime: "unit selection." At its best it sounded genuinely human, because it was human audio. But quality collapsed near database gaps, joins produced audible glitches, and every new voice meant recording and annotating a new studio corpus. Prosody control was minimal. You got the intonation the actor happened to record.
Statistical parametric synthesis (mid-2000s-2015). HMM-based systems (the HTS toolkit era) learned statistical models of acoustic parameters (spectrum, F0, duration) from data, then generated parameters and ran them through a vocoder. Voices were smaller, more flexible, and adaptable to new speakers with little data, but the averaging inherent in the statistics produced a characteristic muffled, buzzy quality. Parametric systems traded naturalness for control.
Neural synthesis (2016-present). DeepMind's WaveNet (2016) showed that a neural network generating raw waveform samples could dramatically outperform both concatenative and parametric systems in listening tests. Everything since builds on that result.
| Era | Representative systems | How it works | Data needed |
|---|---|---|---|
| Formant | Voder, DECtalk, Klatt synthesizer | Hand-written acoustic rules drive oscillators/filters | None (rules, not recordings) |
| Concatenative | Festival, AT&T Natural Voices | Splices units of recorded speech at runtime | 10-50+ hrs, one studio speaker, heavily annotated |
| Statistical parametric | HTS (HMM-based) | Learns statistics of acoustic features, vocoder resynthesis | A few hours per voice |
| Neural | WaveNet, Tacotron 2, VITS, flow/diffusion TTS | Deep networks map text to spectrogram/waveform | 10-40 hrs per voice; 10k-100k+ hrs for zero-shot systems |
How Modern Neural TTS Works
The classic neural TTS pipeline has two stages:
1. Acoustic model: text → mel-spectrogram. A sequence-to-sequence network converts a phoneme or character sequence into a mel-spectrogram, a time-frequency representation of the target speech. Tacotron 2 (2018) is the canonical example: an attention-based encoder-decoder that learns alignment between text and audio frames directly from paired data. Later models such as FastSpeech 2 made this stage non-autoregressive, predicting phoneme durations explicitly and generating all frames in parallel, trading a little prosodic spontaneity for speed and stability.
2. Vocoder: mel-spectrogram → waveform. A separate neural network renders the spectrogram as audio. WaveNet filled this role first; GAN-based vocoders like HiFi-GAN (2020) made it fast enough for real-time synthesis on CPU with near-indistinguishable quality.
Three developments moved the field past this two-stage design:
- End-to-end models. VITS (2021) trains text-to-waveform as a single model, combining a variational autoencoder, normalizing flows, and adversarial training. It removed the acoustic-model/vocoder mismatch and became the default open-source baseline for years.
- Diffusion and flow-matching models. Systems like Grad-TTS, NaturalSpeech, Meta's Voicebox, and the flow-matching family (E2-TTS, F5-TTS) treat generation as iteratively denoising toward a speech representation. They currently deliver the strongest combination of quality, diversity, and controllability, and flow matching brought inference down to a handful of steps.
- Zero-shot voice cloning. VALL-E (2023) reframed TTS as language modeling over discrete neural audio codec tokens: give it a few seconds of a reference voice, and it continues in that voice. The current generation of open zero-shot systems (XTTS, CosyVoice 2, F5-TTS, IndexTTS-2 among them) clone timbre, accent, and speaking style from a short prompt with no fine-tuning, and commercial platforms have made cloning a product feature. These models are trained once on tens of thousands of hours spanning thousands of speakers.
How Speech Synthesis Is Evaluated
Synthesis has no single ground truth, since many waveforms are valid renditions of one sentence, so evaluation leans on human judgment plus a growing set of proxies:
- MOS (Mean Opinion Score). Listeners rate naturalness from 1 to 5; scores above ~4.3 sit in "hard to distinguish from human" territory on read speech. Comparative MOS (CMOS), where listeners compare two systems directly, is more sensitive for closely matched models. MOS numbers are not comparable across papers, because listener pools, instructions, and audio conditions differ. Treat published MOS as within-study evidence only.
- Speaker similarity. For cloning systems: cosine similarity between speaker embeddings (from a verification model such as ECAPA-TDNN or WavLM-based encoders) of the synthesized audio and the reference, often alongside human similarity MOS.
- Intelligibility via ASR. Run a strong speech recognition model over the synthesized audio and measure WER against the input text. Cheap, automatic, and catches the failure modes (skipped words, babble) that autoregressive models are prone to.
- Predicted MOS. Neural MOS predictors like UTMOS approximate human ratings for fast iteration, with the usual caveat that they can be gamed and drift out of domain.
Data Requirements: How Neural TTS Shifted the Bottleneck
Each era's data appetite tells you where the engineering effort lived. Formant synthesis needed zero recordings and years of expert rule-writing. Concatenative systems needed one immaculate, exhaustively segmented studio corpus per voice; the database was the product. Parametric models got by on a few hours but capped quality.
Neural TTS inverted the equation. Architectures are published, implementations are open source, and training a competitive voice is a known recipe. What separates a great voice from a mediocre one is almost entirely the corpus:
- Single-voice models want 10-40 hours of consistent, clean recordings from one speaker (LJSpeech, the classic open benchmark, is ~24 hours). Recording-condition consistency matters as much as volume. A model faithfully reproduces room tone, mic changes, and fatigue.
- Zero-shot and multi-speaker models are trained on 10,000-100,000+ hours across thousands of speakers. At that scale, coverage becomes the constraint: accents, ages, speaking styles, emotional range, and languages the model has never seen simply cannot be cloned well. This is acute for lower-resource languages, where large open TTS corpora barely exist. See our guide to speech data for low-resource languages.
- Every model needs accurate text-audio pairing. Transcription errors become pronunciation errors; misaligned segments become unstable attention and skipped words.
What to look for in a TTS corpus (sample rate, alignment quality, phonetic coverage, licensing) is a topic of its own; we cover it in depth in our guide to text-to-speech datasets.
Prosody: Why Conversational Data Matters
The last unsolved percentage points in speech synthesis are prosody: intonation, rhythm, stress, pausing, disfluency. A modern model reading a news sentence is essentially solved. The same model voicing a dialogue turn ("wait, no, yeah, that actually makes sense") usually is not, because it was trained on read speech.
Read-aloud corpora have narrow, declarative prosody: even pacing, falling terminal pitch, no hesitations, no listener-directed intonation. Conversational speech is a different distribution (backchannels, pitch resets, lengthening, laughter, turn-final rises) and a model cannot generate what it never observed. This is why TTS voices destined for voice agents, dubbing, or companion apps increasingly train or fine-tune on spontaneous conversational recordings rather than scripts; the difference between the two data types is bigger than most teams expect, as we break down in conversational vs. scripted speech data.
The practical upshot: if your synthesis target is dialogue, source dialogue data. That means real two-party conversations with accurate, time-aligned transcripts, cleared for commercial training use. That combination is rare in open datasets and is exactly where purpose-built corpora earn their keep.
Need training data?
SpeechData.ai provides conversational speech corpora in 60 languages: dual-channel audio, time-aligned transcripts, speaker metadata, and a full consent chain, licensed for commercial TTS and voice-cloning training. Browse our datasets or contact us to discuss your target voices and languages.