A text-to-speech dataset is a set of audio recordings paired with exactly matching transcripts, used to train a model to generate speech. What separates a usable TTS dataset from an ASR corpus is a single principle: the model will imitate everything in the audio. The voice, but also the room, the microphone, the compression artifacts, and the reading style. ASR training data should be as messy as the real world; TTS training data should sound the way you want your product to sound.
That principle drives every practical decision: whether you need studio or found audio, single- or multi-speaker corpora, how many hours per voice, and why phonetic coverage and prosody matter more than raw volume. It also drives the legal side. Voice cloning has made voice-rights consent the single most important line item in any TTS data purchase.
This guide covers how requirements differ across model generations (from VITS-style single-speaker models to Tortoise-style and modern zero-shot cloning systems), realistic hour counts per voice, and why conversational data is increasingly part of the recipe for natural prosody.
What makes TTS data different from ASR data
| Dimension | ASR training data | TTS training data |
|---|---|---|
| Acoustic quality | Diverse, noisy, real-world channels | Clean, consistent, high SNR (typically >30 dB for studio-grade) |
| Speakers | The more the better (hundreds+) | Depends on goal: 1 for a product voice, hundreds to thousands for cloning models |
| Transcript accuracy | Small error rates tolerable | Must be verbatim-exact; mismatches cause audible artifacts |
| Sample rate | 8-16 kHz often fine | 22.05-48 kHz; vocoders expose low-bandwidth audio |
| Disfluencies | Valuable (models must recognize them) | Usually harmful for scripted voices, valuable for conversational style |
| Recording consistency | Irrelevant | Critical; mic changes and room changes mid-corpus degrade output |
The classic open references illustrate the target: LJSpeech (~24 hours, one speaker, consistent recording) remains the benchmark for single-speaker training, and VCTK (109 speakers, ~44 hours total) for multi-speaker work. Both are read speech in controlled conditions, which is exactly why models trained only on them sound like someone reading aloud.
Two properties matter beyond cleanliness:
Phonetic coverage. The corpus should contain every phoneme of the language in varied contexts, ideally verified with a phoneme histogram, and for smaller corpora, scripts designed for diphone/triphone coverage. Gaps show up as mispronunciations on unseen words. This matters double in languages with sounds English-centric pipelines miss: tonal contrasts in Yoruba, pitch accent in Japanese, emphatic consonants in Arabic.
Prosodic range. If every sentence is a flat declarative read at constant pace, the model can only produce flat declaratives. You want questions, exclamations, varied sentence lengths, and (for assistant-style products) genuinely conversational intonation.
Data requirements by model generation
Single-speaker end-to-end models (VITS, FastSpeech 2 + HiFi-GAN). Training from scratch: roughly 10-24 hours of one speaker in consistent studio conditions. Fine-tuning a pretrained checkpoint to a new voice: commonly 30 minutes to 2 hours for a recognizable, usable voice, with quality still improving up to ~5 hours. Transcript exactness and consistent recording conditions matter more than adding a 25th hour.
Tortoise-style autoregressive + diffusion models. These decouple "learning to speak" from "learning a voice": a large multi-speaker pretraining corpus (Tortoise used tens of thousands of hours, largely audiobook-derived) teaches general speech, and short reference clips condition the voice at inference. If you're building this class of model, your data problem is breadth (many speakers, many styles) with per-file quality filtering rather than uniform studio capture.
Zero-shot voice cloning (XTTS, VALL-E-style, F5-TTS and successors). At inference these need 3-15 seconds of reference audio. But pretraining them well requires thousands to tens of thousands of hours across thousands of speakers, and the pretraining distribution defines the cloning ceiling: a model pretrained only on read English audiobooks clones read English voices well and struggles with conversational delivery, other languages, and accents outside its distribution. This is where multilingual conversational corpora earn their keep. Adding, say, 1,000 hours of Brazilian Portuguese or Hindi conversation measurably improves cloning fidelity for those speaker populations.
Hours needed per voice: quick reference
| Goal | Audio needed | Notes |
|---|---|---|
| Zero-shot clone (pretrained model) | 3-15 seconds reference | Quality bounded by pretraining data coverage |
| Fine-tune existing model to a new voice | 30 min - 2 hrs | Studio-clean, verbatim transcripts |
| High-quality dedicated product voice | 5-20+ hrs | Include style/emotion variants |
| Single-speaker model from scratch | 10-24 hrs | LJSpeech-scale |
| Multi-speaker model from scratch | 100+ hrs, 50+ speakers | VCTK-scale and up |
| Zero-shot cloning model pretraining | 10k-100k+ hrs, 1k+ speakers | Diversity of style and language is the constraint |
Why conversational data matters for natural prosody
The most common complaint about production TTS is not audio quality (modern vocoders solved that) but that the voice sounds like it's reading. That's a data problem. Read-speech corpora teach read-speech prosody: narrow pitch range, uniform pacing, list-like intonation.
Spontaneous conversation carries what scripted reads can't: natural phrase-final lengthening, backchannel intonation ("mm-hm," "right"), genuine question contours, hesitation patterns, and the pitch resets that mark real turn-taking. Modern conversational TTS systems (the style popularized by models like CSM and Dia in 2024-25) get their realism precisely from training on conversational audio, sometimes deliberately keeping disfluencies so the model can produce natural-sounding "um"s and restarts on demand.
The practical recipe for a voice-agent product is a hybrid: a studio-quality scripted core for the target voice, plus conversational data, either from the same speaker or in pretraining, to widen the prosodic distribution. For the tradeoffs in detail, see our comparison of conversational vs. scripted speech data. Our catalog is conversational by design for this reason; where a project needs a bespoke scripted voice on top, that's a custom collection job.
Licensing and voice rights: the part that can kill a product
TTS licensing is stricter than ASR licensing because the output is the input's voice. Check four things before buying any voice cloning dataset or TTS corpus:
- Consent must name synthesis. A contributor who consented to "speech technology research" or ASR training has not consented to having their voice cloned. The consent language must explicitly cover training models that generate or synthesize speech. Ask to see it.
- Right of publicity and biometric law. Voices are protected as biometric identifiers under GDPR and laws like Illinois BIPA, and voice-specific statutes are spreading. Tennessee's ELVIS Act (2024) explicitly covers AI voice simulation. Consent documentation is your defense.
- Output rights. Confirm the license covers commercial use of generated audio, not just the training run. Some licenses are silent on outputs; get it in writing.
- Revocation terms. If a contributor withdraws consent, what happens to your trained model? A well-drafted license answers this; most bad ones don't. Our speech data licensing guide walks through the clauses in detail.
A vendor who cannot produce per-speaker consent records for synthesis use is selling you risk, whatever the audio sounds like.
How to evaluate a TTS dataset before buying
Run this on a sample you select:
- Listen. Random files, decent headphones. Room tone changes, clipping, breath artifacts, de-essing damage.
- Measure. SNR distribution, sample rate (confirm it's native, not upsampled), loudness consistency (LUFS variance across files).
- Verify transcripts. Word-exact check on 30+ random utterances, including numbers, punctuation conventions, and non-speech markers.
- Check coverage. Phoneme histogram against the language's inventory; sentence-type distribution (declarative/interrogative ratio).
- Check speaker metadata. Consistent speaker IDs, demographics, and above all consent scope per speaker.
- Train a probe. Fine-tune a small model on 30 minutes of one speaker; audible artifacts surface data problems faster than any spreadsheet.
Need training data?
SpeechData.ai provides consented, transcribed conversational speech in 60 languages: 500-2,000 hours per language, 50-200 native speakers, licensed for commercial TTS and voice-AI training at $60-95/hour. Explore the catalog or get in touch for samples and voice-rights documentation.