OpenAI released GPT-Live-1 on September 10 as a full-duplex voice model that can listen and speak at the same time. OpenAI says the model cut interruptions by almost 80% in an early language-tutor evaluation. It also reports that one healthcare company removed 23,000 lines when it replaced a cascaded voice stack.
That does not make native speech the automatic choice for every voice chatbot. Architecture decides which latency you can remove, which components you can inspect, and how safely a spoken correction reaches a business action. Use this scorecard before you compare voices or run a polished demo.
| Requirement | Native speech | Cascaded STT-LLM-TTS | Hybrid |
|---|---|---|---|
| Natural overlap and interruption handling | Strongest default | Requires careful orchestration | Native model manages the conversation |
| Independent speech-provider choice | Limited to platform options | Choose and replace each component | Usually limited at the front end |
| Transcript-first inspection | Available, but verify event timing | Explicit text boundary at every turn | Text backend creates a clear action record |
| Complex reasoning and tool calls | Possible, model dependent | Full control over text model and tools | Delegate hard work to a text model |
| Small engineering team | Fewer components to operate | More moving parts | Moderate complexity with a clean boundary |
| Strict action controls | Keep authorization outside the model | Easier to gate at the text boundary | Best balance for many support workflows |
The useful default for support is often hybrid. Let a native speech model handle timing, pauses, and interruptions. Send stable intent and confirmed fields to a text-based service that owns retrieval, policy checks, tool calls, and approvals.
Name the three architectures
A voice chatbot has to hear audio, decide what the caller means, produce an answer, and play that answer. The architecture question is where those jobs meet.
Cascaded voice chatbot
The classic pipeline has three named stages:
caller audio -> speech-to-text -> language model -> text-to-speech -> caller
This design exposes useful boundaries. You can save the final transcript, replace the speech recognizer without changing the reasoning model, send product names as vocabulary hints, and inspect the exact text that authorized a tool call.
Each boundary also costs time. The speech recognizer must decide that a turn ended. The application waits for model output. The speech synthesizer then waits for enough text to start speaking. Streaming reduces the delay, but it creates new questions about what happens when the caller interrupts halfway through the generated sentence.
Twilio's Conversation Relay documentation shows a managed version of this pattern. Twilio handles speech recognition, speech synthesis, session state, and low-latency communication while the application receives transcript events and returns text. A cascaded system does not have to mean building every media component yourself.
Native speech chatbot
A native speech model accepts audio and produces audio inside one conversational loop:
caller audio -> speech model -> caller audio
The model can hear a caller start speaking while its own answer is still playing. That makes barge-in, short acknowledgements such as "right," and thinking pauses part of the same stream rather than special events glued onto a text turn.
The shorter diagram hides important application work. You still need authentication, knowledge retrieval, tool schemas, policy rules, action logs, handoff logic, and a call provider. You also need a transcript or structured event record for review. Fewer model handoffs do not remove the business system around the call.
Hybrid voice chatbot
The hybrid pattern splits conversational timing from business reasoning:
caller <-> native speech model <-> text reasoning and action service
The speech model can acknowledge the caller, wait through a pause, and stop when interrupted. It delegates work that needs deeper reasoning or tools. OpenAI describes this exact separation for GPT-Live-1, which can hand work to a backend text model while the voice conversation continues.
This pattern is appealing because floor control and business control have different needs. Floor control must react in milliseconds to speech. Business control should slow down long enough to validate an order ID, check permissions, and record why an action was allowed.
Build a latency budget by component
"The call feels slow" is too vague to debug. Record the delay between events that your team can change.
For a cascaded system, capture at least four values:
end_of_user_speech_to_final_transcript_ms final_transcript_to_first_model_token_ms first_model_token_to_first_audio_ms end_of_user_speech_to_first_audio_ms
The last value is the caller's wait. The first three explain it. Twilio's voice insights metrics separate network, speech-to-text, application, and text-to-speech latency for this reason.
A native system needs a different trace. The model receives audio continuously, so a single "request started" timestamp misses the moments that shape the call:
user_speech_started user_speech_ended agent_speech_started user_interrupted_agent agent_audio_stopped new_intent_committed
Measure the median and the slowest five percent. Then separate ordinary responses from tool-using turns. A fast greeting can conceal a six-second account lookup.
Do not optimize silence away blindly. A caller may pause to read a confirmation number or think through a date. Response speed and interruption rate belong beside each other. If faster endpointing makes the agent speak over customers, the lower latency is a regression.
Keep authorization outside the audio loop
Voice makes an action sound casual. "Sure, I'll cancel that" takes a second to say and may trigger an irreversible change.
The action service should receive a structured request after the speech layer has stabilized the caller's intent. It should apply the same authentication, permission, confirmation, and idempotency rules used by a text interface. The voice model can ask for confirmation. It should not define what counts as enough confirmation.
Use a payload that preserves the decision:
{
"intent": "cancel_subscription",
"account_id": "acct_4821",
"effective_date": "2026-10-01",
"caller_confirmation": {
"transcript": "Yes, cancel it at the end of this billing period",
"captured_at": "2026-09-14T10:42:18Z"
},
"source_turn_id": "turn_19",
"idempotency_key": "call_739-turn_19-cancel"
}
Now an interruption has somewhere safe to land. If the caller says, "Wait, keep it active," before execution, the application can invalidate turn_19 and refuse the old payload. A raw audio stream with no stable turn or action ID makes that recovery much harder to prove.
The chatbot tool permissions checklist covers the same control boundary for web chat. Read access, write access, approval, and audit requirements stay the same when audio replaces typing.
Work through a correction, not a demo prompt
Consider a customer calling to move a delivery. The agent has already started its answer when the customer changes the request.
Customer: Move my Tuesday delivery to Friday. Agent: I can move order 1842 to Fri... Customer: Wait, Thursday. I will not be home Friday. Agent: Thursday, September 17. Should I make that change? Customer: Yes. Agent: Done. Order 1842 is scheduled for Thursday, September 17.
The audio sounds simple. The event trace is the real test:
10:42:11.080 intent_candidate delivery_date=2026-09-18 10:42:11.310 agent_speech_started 10:42:11.860 caller_barge_in 10:42:11.940 agent_audio_stopped 10:42:12.440 intent_revised delivery_date=2026-09-17 10:42:13.020 confirmation_requested 10:42:14.670 confirmation_received 10:42:14.720 update_delivery submitted idempotency_key=call_739-turn_19 10:42:15.080 update_delivery succeeded
Pass the test only if the Friday candidate never reaches the delivery API, playback stops promptly, the revised date is repeated, and one confirmed write occurs. The final transcript alone cannot prove any of those facts.
This is also why critical fields need their own test set. The voice transcription testing guide shows how to score names, order IDs, dates, and self-corrections without hiding a dangerous error inside average word accuracy.
Test conversation behavior and task completion together
A pleasant voice can still fail the job. A correct backend can still produce a call that customers abandon because it interrupts them.
Full-Duplex-Bench v3 evaluates real human audio with hesitations, self-corrections, and multi-step tool use. Its reported results expose a tradeoff: one system had the fastest latency but the lowest turn-take rate, while the cascaded baseline took the turn reliably and had the highest latency.
The broader task gap is more sobering. The Tau-Voice benchmark tested 278 grounded tasks. Its authors report 85% completion for a text baseline, compared with 31% to 51% for voice agents in clean conditions and 26% to 38% with realistic noise and varied accents.
Your release suite should therefore score both layers:
| Measure | What it catches | Release question |
|---|---|---|
| Task completion | Correct answer or completed action | Did the caller get the intended result? |
| Agent interruption rate | Agent speaks before the caller finishes | Does faster endpointing create talk-over? |
| Barge-in stop time | Delay before agent playback stops | Can the caller regain the floor? |
| Correction survival | Revised value replaces the abandoned value | Did "Thursday, not Friday" reach the tool? |
| Tool-call latency | Time spent after intent is stable | Is the backend causing dead air? |
| Silent call rate | Calls with broken input or output audio | Did the media path fail completely? |
Run the same calls after a model, speech provider, endpointing setting, prompt, codec, or tool change. Ten studio recordings will not cover a phone launch. Include mobile networks, speakerphone, background conversation, accents, brief acknowledgements, long pauses, and callers who revise a request twice.
The voice AI phone checklist is a useful launch companion. It covers job boundaries, short spoken answers, handoffs, and evidence storage. Architecture testing goes one level lower by proving that the media and action paths preserve those rules.
Choose based on the failure you can operate
Choose a cascaded architecture when you need provider choice, explicit text boundaries, custom vocabulary, or detailed control over every recognition and synthesis step. It suits teams that already know how to operate streaming services and can own the latency budget.
Choose native speech when natural overlap is central, your supported languages and voices fit the provider, and a smaller engineering team needs to remove orchestration work. Verify the transcript and event hooks before assuming the simpler diagram creates simpler audits.
Choose hybrid when the call needs natural turn-taking plus consequential business actions. Keep voice responsive at the front. Keep policy, tools, approvals, and logs in a backend that can be tested like any other production service.
Architecture does not rescue weak sources or vague workflow rules. It decides where you can see mistakes and stop them. Pick the design whose failures your team can measure, replay, and contain, then test it with corrections and interruptions before customers do.
No credit card required.
