How Neural Text to Speech Works, Step by Step
Neural text to speech works in stages: normalize the text, predict pronunciation, generate a spectrogram, then a vocoder makes the audio. Each step explained.
Neural text to speech uses deep learning models to turn text into audio in stages. Most systems normalize the text, predict its pronunciation, generate a mel spectrogram that describes the sound over time, and then pass it to a vocoder that produces the waveform. Some newer designs merge these steps into one model.
What makes text to speech "neural"?
Older systems assembled recorded fragments or used hand-tuned parameters. Neural systems learn the mapping from text to speech from pairs of text and recorded audio. The Tacotron paper, first submitted in March 2017, describes an "end-to-end generative text-to-speech model" that produces speech directly from characters and can be trained from scratch on text-audio pairs.
If you are new to the topic, the basics of text to speech cover the terms used here.
The pipeline at a glance
| Stage | Input | Output | Example | |---|---|---|---| | Text normalization | Raw text | Spoken-form text | "Henry VIII" becomes "Henry the Eighth", but "Chapter VIII" becomes "Chapter Eight" | | Pronunciation (grapheme-to-phoneme) | Words | Phonemes | Dictionary lookup first, rules for unknown words | | Acoustic model | Characters or phonemes | Mel spectrogram | Tacotron 2 maps character embeddings to mel-scale spectrograms | | Vocoder | Spectrogram | Waveform | WaveNet, or HiFi-GAN in later systems |
This is the classic two-stage design. Many systems use it, but not all of them. VITS and codec language models work differently, as covered below.
Step 1: Text normalization
Normalization expands numbers, abbreviations and heteronyms, and context decides the answer. "1325" can be "thirteen twenty-five", and "St." can mean Saint or Street. Homographs are often resolved with neighboring words and part-of-speech tagging. Wikipedia's speech synthesis article reports error rates for this step "usually below five percent."
Step 2: Pronunciation
Grapheme-to-phoneme (G2P) conversion turns spelling into sounds. Dictionary lookup is quick and accurate, but it fails on words it does not know. Rules can handle any input, but they become complicated quickly. English leans on dictionaries because its spelling is irregular, and nearly all systems combine the two approaches.
The front end also divides text into prosodic units such as phrases, clauses and sentences. Statements, questions and exclamations each get a different pitch contour.
Step 3: The acoustic model predicts a spectrogram
The acoustic model predicts a mel spectrogram. A spectrogram is a picture of sound energy over time, and the mel scale spaces frequencies the way listeners perceive pitch. The Tacotron 2 paper uses a recurrent sequence-to-sequence network to map character embeddings to mel-scale spectrograms.
FastSpeech 2 takes a different route. It is non-autoregressive, which means it predicts the whole sequence at once, and it adds pitch, energy and more accurate duration as conditional inputs. Its paper reports a 3x training speed-up over FastSpeech.
Step 4: The vocoder turns the spectrogram into audio
A vocoder converts the spectrogram into a waveform. WaveNet was built to model raw audio, which is "typically 16,000 samples per second or more," one sample at a time, with each sample predicted from all the samples before it. Tacotron 2 used a modified WaveNet as its vocoder, and its authors found that mel spectrograms work well as WaveNet input.
HiFi-GAN is a GAN vocoder that models periodic patterns in audio. Its paper reports that it generates 22.05 kHz audio 167.9 times faster than real time on a single V100 GPU. A small-footprint version runs 13.4 times faster than real time on a CPU.
Step 5: End-to-end and codec language models
Not every system follows the two-stage design. VITS is a parallel end-to-end model that combines variational inference, normalizing flows, adversarial training and a stochastic duration predictor. On the LJ Speech dataset, its authors report MOS "comparable to ground truth."
VALL-E treats TTS as conditional language modeling over discrete neural audio codec codes. It was trained on 60K hours of English speech and can synthesize a speaker from "only a 3-second enrolled recording." It is a research system. None of the vendor pages checked for this article describes its internal architecture, so do not assume a commercial product uses any of these components.
Milestones in neural TTS
| Date | System | Key idea | Reported result | |---|---|---|---| | Sep 2016 | WaveNet | Raw audio, one sample at a time | MOS 4.21 for US English in DeepMind's 2016 test | | Mar 2017 | Tacotron | End-to-end from characters | MOS 3.82 for US English in the paper | | Dec 2017 | Tacotron 2 | Spectrogram plus WaveNet vocoder | MOS 4.53, against 4.58 for professional recordings | | Jun 2020 | FastSpeech 2 | Non-autoregressive, pitch and energy inputs | 3x training speed-up over FastSpeech | | Oct 2020 | HiFi-GAN | GAN vocoder | 167.9x real time on one V100 GPU | | Jun 2021 | VITS | Parallel end-to-end | MOS comparable to ground truth on LJ Speech | | Jan 2023 | VALL-E | Codec language model | Trained on 60K hours; 3-second enrolled recording |
Each score comes from a different paper with its own test set, so do not rank these systems by MOS alone.
Why the same text sounds a little different each time
Neural models are nondeterministic. ElevenLabs states that its models are nondeterministic and offers a seed parameter that improves consistency, but the seed does not guarantee identical output. Microsoft says its HD voices "introduce slight variations in each output to enhance realism," and a temperature setting trades stability for variety. The documented defaults for Dragon HD Omni are below.
| Parameter | Default | Range | |---|---|---| | temperature | 0.7 | 0.3 to 1.0 | | top_p | 0.7 | Not listed on the page | | top_k | 22 | 1 to 50 | | cfg_scale | 1.4 | 1.0 to 2.0 |
Higher temperature, top_p and top_k values give more expressive but less stable output.
How much data does a neural voice need?
Tacotron 2 "required substantial training data, typically tens of hours of audio." According to Wikipedia, a version trained on only 24 minutes "failed to produce intelligible speech." Newer systems vary widely. Kokoro-82M, an open-weight model with 82 million parameters, describes its v1.0 training data as "a few hundred hours" on its model card. Zero-shot research systems such as VALL-E use a few seconds of reference audio to imitate a speaker, which is a different job from training a studio voice.
What this means when you write a script
The pipeline explains why script details matter. Normalization decides how numbers and abbreviations are read, so write them the way you want them heard. Pronunciation and prosody depend on punctuation and sentence shape, so clean punctuation helps. Because neural output varies from run to run, a short test pass is worth doing before you generate a long file.
- Write numbers, dates and abbreviations in the form you want spoken.
- Use clear punctuation, since phrase boundaries shape pitch and pauses.
- Test a short passage first, then generate the full script.
OndaLabs text to speech lets you control delivery by describing it in plain words and with supported expressive tags in the script. Onda Lite is on all plans at 1 credit per character. Onda Flash, the most expressive model, is on paid plans at 1.5 credits per character. For sentence-level habits, read how to make an AI voice sound natural, then try a passage in the text to speech studio.
Frequently asked questions
How does neural text to speech work step by step?
It normalizes the text, predicts pronunciation and prosody, generates a spectrogram with an acoustic model, and converts that spectrogram into audio with a vocoder. Some newer systems, such as VITS, combine these steps into one end-to-end model. The stages are still a useful way to understand the process, even when a product does not disclose its internals.
What is a vocoder and what does it do?
A vocoder converts an intermediate representation, usually a spectrogram, into a waveform that a speaker can play. WaveNet and HiFi-GAN are two examples covered in this article. The vocoder is the last stage that produces actual sound.
What is a mel spectrogram?
A mel spectrogram is a picture of sound energy over time, with frequencies arranged on the mel scale, which spaces pitches the way listeners hear them. Acoustic models predict it from text, and vocoders turn it back into audio. Tacotron 2 is a well-known example of a model that predicts mel-scale spectrograms.
What is the difference between concatenative and neural TTS?
Concatenative TTS joins short recorded fragments from one speaker, so changing the voice or style means recording a new database. Neural TTS learns the mapping from text to audio from paired recordings and can produce sound that was not in any recording. Neural systems scored higher than concatenative ones in the published listening tests cited here.
What is the difference between Tacotron, WaveNet, FastSpeech and VITS?
Tacotron predicts spectrograms from characters, and Tacotron 2 pairs that model with a WaveNet-based vocoder. WaveNet is a sample-by-sample audio model that was also used as a vocoder. FastSpeech 2 is non-autoregressive and adds pitch and energy inputs, while VITS is a parallel end-to-end model. Each one addresses a different part of the problem.
Why does the same text sound slightly different each time?
Neural models are nondeterministic, so they can produce slightly different output on each run. Vendors offer controls such as seeds and temperature settings to improve consistency. ElevenLabs says its seed parameter does not guarantee identical output, so generate a take once and keep the file if you need a fixed version.