Speech to Speech Model vs STT-LLM-TTS Pipeline
A speech to speech model answers in audio, while a pipeline chains speech-to-text, an LLM and text-to-speech. Compare latency, control and cost, October 2026.
A speech to speech model takes audio in and produces audio out inside one model. A pipeline chains three stages instead: speech-to-text, a text language model and text-to-speech. OpenAI frames the choice as one model that interprets audio and responds in speech, versus stages you can inspect and replace. As of October 2026, major vendors offer both designs.
How a pipeline works, and what you control
A pipeline runs its stages in order. Speech recognition turns incoming audio into text while the caller is still speaking. A language model turns the transcript into a streaming reply. Speech synthesis turns that reply into audio. Pipecat's documentation notes that the stages run in parallel, so later parts of a reply are generated while earlier parts are being spoken.
The main advantage of a pipeline is control. Each stage is visible and replaceable. You can store the transcript, check it before the language model answers, or swap one speech model for another. The range of components is wide. Deepgram's documentation gives about 260 milliseconds for end-of-turn detection in its Flux model. Cartesia's Sonic 3.6 speaks 44 languages according to its documentation, and ElevenLabs Flash v2.5 lists 32.
How a speech to speech model works
A speech to speech model processes audio directly. OpenAI's Realtime guide says the model "works with audio directly, keeps conversation state, and can call tools." There is no separate text step that the caller's words must pass through, so the model's reply can start from the audio itself.
Some designs are full duplex, which means the model listens while it speaks. The Moshi paper from Kyutai describes a speech-text model that models its own speech and the user's in parallel streams. The authors claim a theoretical latency of 160 milliseconds and about 200 milliseconds in practice. Those are the authors' figures, not independent measurements, and the arXiv paper sets out the design.
OpenAI's GPT-Live-1 is also full duplex. OpenAI calls it "our premier model for natural, expressive voice conversations with smooth interruption handling." It can hand reasoning and tool use to a backend agent while it keeps talking, and the backend is billed separately. Google describes Gemini 3.8 Live as "the default option for most low-latency voice agent experiences." Both are vendor descriptions.
Side by side
| Question | Pipeline | Speech to speech model |
|---|---|---|
| Can you see the text between stages? | Yes. Each stage produces text you can log or check | A transcript is available, but it is generated alongside the audio, not used as the model's input the way it is in a pipeline |
| Can you swap one part? | Yes, each stage independently | You replace the model as a whole |
| Turn detection | You choose recognition endpointing, a turn-detector model or voice activity detection, according to LiveKit's turn guide | Server-side detection; LiveKit can use it with turn_detection="realtime_llm" |
| Pricing unit | Per component. LiveKit's calculator estimates $0.0479 per minute for a phone-call agent | Per token or per minute. GPT-Live-1 is $0.05 per minute |
| Languages in vendor docs | Set by each component, for example Cartesia's 44 | Gemini's Live API lists 70 languages; Nova 2 Sonic lists 7; OpenAI's pages give no count |
| Feature support | Pick any text model that has the features you need | Gemini 3.8 Live does not support caching, structured outputs, file search or the Batch API |
Which is faster? What the numbers show
Artificial Analysis measures time to first audio, or TTFA, for speech-to-speech models: the seconds needed to generate the first token of audio output. It also reports a benchmark cost per hour. The table shows a selection of scored models as of October 2026.
| Model | Index score | TTFA (seconds) | Benchmark cost per hour (USD) | |---|---|---|---| | Grok Voice 3.0 | 86.9 | 0.81 | 4.80 | | GPT-Live-1 (Astra label, medium) | 83.2 | 1.34 | 5.83 | | Gemini 3.8 Live, Extended Thinking (High) | 82.6 | 1.35 | 3.50 | | Gemini 3.8 Live | 76.0 | 1.18 | 0.84 | | GPT-Realtime-2 (High) | 73.6 | 1.14 | 4.14 |
Three caveats apply. The page's description of how its index is weighted contradicts itself, so this article does not describe the weighting. TTFA measures the first audio, not the full reply. And the page gives no latency or cost for the four default cascaded systems it lists, so it cannot say whether a native model beats a pipeline.
Settings matter within a single model family. Gemini 3.1 Flash Live measured 2.99 seconds at its High setting and 0.96 seconds at Minimal. Vendor claims also need checking. xAI says Grok Voice 3.0 offers "class-leading latency at under one second," which is consistent with the 0.81-second TTFA above. The latency guide explains why numbers from different sources do not line up.
What each one costs
Token prices and per-minute prices are hard to compare directly. Here is a rough check for audio only, with a 50/50 split between caller and agent audio. OpenAI's cost guide counts user audio at 10 tokens per second and assistant audio at 20 tokens per second. On that basis:
- gpt-realtime-2.1 comes to about $0.048 per minute, at $32 per million input tokens and $64 per million output tokens.
- gpt-realtime-2.1-mini comes to about $0.015 per minute, at $10 and $20 per million tokens.
- Gemini 3.8 Live comes to about $0.0115 per minute, from its per-minute audio rates of $0.005 in and $0.018 out.
These figures understate a real bill. Each response resends the conversation, so later turns cost more, and input transcription is billed separately. OpenAI says prompt caching is automatic but best effort. On gpt-realtime-2.1, cached audio input costs $0.40 per million tokens, against $32 uncached.
Pipeline costs add up by component. Deepgram's bundled Voice Agent API costs $0.075 per minute on its Standard tier, and $0.050 per minute if you bring your own language model and speech synthesis. The API comparison lists prices for each provider.
Where speech-to-speech models still lag
Voice benchmarks are harsh. In the tau-Voice study, voice agents completed 31 to 51 percent of customer-service tasks under clean conditions, against 85 percent for a text agent. Sierra's later snapshot put the best voice result at 67 percent by April 2026, and the gap is narrowing. The study tested native speech-to-speech voice agents, so it does not show how pipelines compare.
Feature gaps are real too. Gemini 3.8 Live's model page lists no support for caching, structured outputs, file search, code execution, Google Maps grounding or the Batch API. A pipeline lets you pick any text model that has the features you need.
Hybrids sit between the two
Some designs combine the approaches. GPT-Live supports two delegation modes. In Responses delegation, OpenAI runs a supported Responses model. In client delegation, you connect your own model, agent or service.
Ultravox takes the speech-native route from the other side. It accepts speech directly, without a separate recognition step, by attaching a multimodal projector to an open-weight language model. Its default model is built on Llama 3.3 70B, and the project is MIT licensed. It outputs streaming text, so you still pair it with speech synthesis.
How to choose
- Choose a pipeline if you need to see, store or check every intermediate text before the model replies.
- Choose a speech to speech model if you want one model to handle audio and tool calls, and you accept that you swap the whole model rather than one part.
- Check languages against each vendor's own documentation, not a summary.
- Estimate cost with your real call length, context size and caching, not one minute of ideal audio.
- Test with your own audio, including noise and accents, because benchmark conditions are cleaner than most phone lines.
- Keep fixed prompts as recorded audio, whichever architecture you choose.
Where recorded prompts fit
OndaLabs is neither a pipeline component nor a speech-to-speech model. It has no real-time mode and no API. It can produce the fixed audio that either design still needs: greetings, disclosure lines and hold messages. In the text to speech studio, Onda Lite costs 1 credit per character, and Onda Flash, on paid plans, costs 1.5 credits per character. A 200-character greeting therefore costs 200 credits on Onda Lite or 300 on Onda Flash. Use the dialogue tool to script sample exchanges between a caller and an agent before you record anything.
Frequently asked questions
Is a speech to speech model always faster than a pipeline?
No. The published benchmark data does not show that. Artificial Analysis reports first-audio times between 0.81 and 1.35 seconds for the native models in its table, and it gives no scores for the default cascaded systems it lists. Measure both designs with your own audio and your own tools.
Can I see the transcript with a speech to speech model?
In most cases, yes. OpenAI's Realtime API offers input transcription, billed separately, and Gemini's Live API lists transcripts of input and output. The transcript is generated alongside the audio rather than being the model's input, which is the key difference from a pipeline.
Is speech to speech more expensive than a pipeline?
It depends on the stack. gpt-realtime-2.1 works out to about $0.048 per minute on audio alone, before resent context and transcription. A pipeline's cost is the sum of its parts, and bundled per-minute platforms charge yet another way. Compare the full cost of each design for your call length.
Do speech to speech models support tool calling?
Many do. OpenAI describes gpt-realtime-2.1 as a reasoning model with tool use, and Gemini 3.8 Live supports function calling and asynchronous function calling. Support differs by model, so check the documentation for the exact model you plan to use.
What is full-duplex voice AI?
Full-duplex voice AI listens and speaks at the same time, so the model can handle interruptions without waiting for a turn to end. OpenAI describes GPT-Live-1 as a full-duplex model, and the Moshi paper describes one built on parallel audio streams.