What Is an AI Voice Agent? How It Works in 2026
An AI voice agent talks with people in real time using speech recognition, a language model and speech synthesis. How it works, what it costs and its limits.
An AI voice agent is software that holds a spoken conversation with a person. It listens to speech, uses a language model to decide what to say or do, and answers in synthesized speech, often in real time and often over a phone line. As of October 2026, most builders assemble one from speech recognition, a language model and speech synthesis, or use a single speech-to-speech model.
How an AI voice agent works
Every turn in a conversation runs the same loop: listen, decide, speak. The caller's audio is captured and either turned into text or passed to a model directly. The agent decides on a reply, which may include a tool call such as checking a calendar. Then the reply is spoken back. The loop has to repeat in a fraction of a second, or the call starts to feel wrong.
The LiveKit agents framework describes this as streaming audio through an STT-LLM-TTS pipeline, and it also supports realtime APIs. LiveKit's agents documentation lists turn detection and interruption handling among its core capabilities. Other vendors name the same parts. The table maps each component to its job and to an example from vendor documentation.
| Component | What it does | Example from vendor docs |
|---|---|---|
| Speech recognition | Turns caller audio into text | ElevenLabs Agents lists a speech recognition model as one of its four components |
| Language model | Decides the reply and can call tools | ElevenLabs lets you choose a supported or custom model; Deepgram calls functions mid-conversation |
| Speech synthesis | Turns the reply into audio | ElevenLabs Agents includes a text-to-speech model |
| Turn-taking | Decides when the caller has finished | ElevenLabs uses a proprietary turn-taking model; OpenAI offers server_vad and semantic_vad |
| Telephony | Connects the agent to phone calls | ElevenLabs offers SIP trunking and a native Twilio integration |
| Data and tools | Looks up records, books slots, answers from documents | ElevenLabs supports a knowledge base with retrieval-augmented generation |
Pipeline or speech-to-speech
There are two ways to build the loop. OpenAI's voice agents guide describes a speech-to-speech path, where one model interprets audio, decides what to do and responds in speech. It also describes a chained path, which runs speech-to-text, a text model and text-to-speech in sequence. The chained path lets you inspect or transform the intermediate text and replace each component independently, for example by running a policy check before the text agent responds.
Each path has different trade-offs in control, latency and cost. The comparison of speech-to-speech models and pipelines works through them in detail.
What makes a voice agent feel natural
Natural conversation depends on timing. The Stivers et al. study in PNAS (2009) looked at conversation in 10 languages. The mean gap between one speaker's turn ending and the next beginning was +208 milliseconds, with a mode of 0 milliseconds and a median of +100 milliseconds. Each language's mean sat within about 250 milliseconds of the cross-language average. A 200-millisecond silence was judged a delay in most languages, but it was still considered on time.
Turn-taking is a design problem in its own right. OpenAI exposes two voice activity detection modes. server_vad detects the end of speech from silence and is the default for speech-to-speech sessions. semantic_vad uses a classifier that judges whether the user has finished based on their words, and OpenAI says it is less likely to interrupt the user or split a transcript early. Interruption handling matters as much. Callers talk over agents, and the agent has to stop and listen.
Speed is the other half of the feel. The latency guide covers the numbers and where the delay comes from.
How voice agents differ from IVR and chatbots
An IVR, or interactive voice response system, plays prompts and follows a fixed menu. The caller presses keys or says one of a few keywords. A voice agent accepts open speech and decides what to do with it, which makes it closer to a chatbot that talks. The difference from a text chatbot is timing. A voice agent has to produce its reply while the caller waits on the line, and it has to handle interruptions that text users rarely make.
A hybrid design is one option: fixed menu prompts for routing, with an agent for open questions.
What can voice agents do today, and where do they fail?
Benchmarks give the most grounded picture. Sierra's tau-Voice benchmark uses 278 customer-service tasks, with the same tasks and evaluator for text and voice agents. In an arXiv paper submitted in March 2026, a reasoning text agent completed 85 percent of the tasks. Voice agents managed 31 to 51 percent under clean conditions, and 26 to 38 percent with noise and diverse accents. The authors also found that 79 to 90 percent of failures came from agent behavior rather than from the audio.
Newer numbers exist. Sierra's blog, dated May 1, 2026, says voice task completion rose from 30 percent for OpenAI's gpt-realtime-1.0 in August 2025 to 67 percent for xAI's grok-voice-think-fast-1.0 in April 2026. The two sources are different snapshots, so quote each with its date. Sierra also notes that its accents are generated with ElevenLabs personas, not recorded human speakers.
Vendors publish their own results. In its October 14, 2026 post on Grok Voice 3.0, xAI says its Starlink call centers handle about 20,000 calls a day, with roughly 70 percent resolved without a human. Treat that as a vendor claim until it is checked independently.
What does a voice agent cost?
Most platforms publish per-minute prices, but the prices are not like for like. Some cover the model only, and you add recognition, speech synthesis and telephony. Others bundle several parts. The per-hour column is the per-minute price times 60.
| Service | Published price | Per hour | What it covers | |---|---|---|---| | OpenAI GPT-Live-1 | $0.05 per minute, billed per second | $3.00 | Voice session; backend model and tool use billed separately | | xAI Grok Voice 3.0 | $0.08 per minute, plus $0.004 per text input | $4.80 | Speech to speech | | Deepgram Voice Agent, Standard | $0.075 per minute | $4.50 | Bundled recognition, language model and speech synthesis | | ElevenLabs Agents | $0.08 per minute beyond plan minutes | $4.80 plus LLM | Orchestration, recognition, synthesis and turn-taking; LLM billed separately | | Ultravox | $0.05 per minute after 30 free minutes | $3.00 | Speech-native voice AI | | Vapi | $0.05 per minute hosting | $3.00 plus providers | Orchestration; speech and model costs at provider cost | | Retell AI | $0.07 to $0.31 per minute | Not applicable | Voice engine, language model, synthesis and telephony priced separately |
Retell's pricing page shows how a call breaks into parts: a $0.055 per minute voice engine, $0.015 per minute for most synthesis voices, $0.015 per minute for listed telephony carriers, and a language model priced by model. Its default estimator, which does not name its model, comes to $0.11 per minute.
The API comparison lists more providers and their terms.
Rules to know before you deploy one
Disclosure is a legal question as well as a design one. On February 8, 2024, the US Federal Communications Commission ruled that calls made with AI-generated voices are "artificial" under the Telephone Consumer Protection Act, and that the same prior express consent standard applies as to other artificial or prerecorded voices. The ruling concerns robocalls, so check how it applies to your own call types.
In the EU, Article 50(1) of the AI Act requires providers to design AI systems that interact directly with people so that those people are told they are dealing with AI, unless that is obvious to a reasonably well-informed person. The Article 50 transparency duties apply from August 2, 2026. This is general information, not legal advice.
Where recorded voices still fit
OndaLabs is not a voice-agent platform. It has no real-time mode, no phone service and no API. It is a studio for recorded voices, and a voice agent still needs a good deal of fixed audio: a greeting, a hold message, a voicemail greeting, menu prompts and a disclosure line such as "You are speaking with an automated assistant." Those lines do not change from call to call, so they can be produced once and reviewed before a launch.
The studio produces them with text to speech, using 30 prebuilt voices that work in every supported language. A consistent brand voice can be designed from a text description or cloned from a recording you have permission to use, at 1,500 credits per voice, through custom voices. The dialogue tool can script a sample caller and agent exchange for reviewing a call script.
The Free plan includes 2,000 credits a month, about two minutes of audio. Commercial use is included from the Starter plan, which costs $9 a month. The pricing page lists every plan.
Frequently asked questions
What is the difference between an AI voice agent and an IVR?
An IVR follows a fixed menu of prompts and keypresses. An AI voice agent accepts open speech, decides on a reply with a language model and can call tools to act on the request. The agent also has to respond in the time a caller expects, which makes latency part of the design.
Does an AI voice agent have to say it is an AI?
In the EU, Article 50(1) of the AI Act requires disclosure unless it is obvious to a reasonably well-informed person, and those duties apply from August 2, 2026. In the United States, the FCC's 2024 ruling covers AI-generated voices in robocalls, and some state laws add their own rules. A clear disclosure line is the safe default, and you should check the rules for each market you call.
How much does an AI voice agent cost per minute?
Published per-minute prices run from $0.05 to about $0.31, depending on what the price covers. Model-only prices leave out recognition, synthesis and telephony. Bundled prices, such as Deepgram's $0.075 per minute on its Standard tier, cover more of the stack. Add your language model and telephony costs before you compare.
Can an AI voice agent make phone calls?
Yes. OpenAI's Realtime API accepts phone calls through a SIP trunking provider such as Twilio. ElevenLabs Agents offers SIP trunking, batch outbound calls and a native Twilio integration, and LiveKit agents can place and receive calls through its SIP integration. Outbound calls are subject to consent rules, such as the FCC's TCPA ruling in the United States.
How fast does a voice agent need to respond?
People leave a gap of about 200 milliseconds between turns on average, based on the 10-language study. Benchmarks published in 2026 put default platform stacks at about 1.3 to 1.7 seconds median, measured from the caller's side. The latency guide explains what to measure and why the published numbers differ.