Loading…
Agent Architecture
Most voice agents today follow a similar fundamental pipeline — audio in, transcribe, think, speak, audio out. However, newer architectures are beginning to challenge this model by collapsing or eliminating steps entirely.
Pipeline Overview
Click each component to learn more about the traditional voice pipeline.
Caller Audio
VAD
~20ms
STT
~200ms
LLM
~300ms
TTS
~150ms
Response
Data Format
PCM
chunks
text
text
audio
Caller Audio
Raw audio stream from a phone call or WebRTC connection. Typically 8kHz (PSTN) or 16-48kHz (WebRTC). This is the entry point for the entire pipeline.
Beyond the Traditional Pipeline
The STT → LLM → TTS pipeline above is well-understood and widely deployed, but it has inherent limitations — each hop adds latency, and converting between audio and text loses information like tone, emotion, and pacing. A new generation of architectures is emerging that rethinks this from the ground up.
Speech-Native Models
Models that process audio directly without converting to text in between — audio in, audio out. Open-weight examples like Kyutai's Moshi and Alibaba's Qwen-Omni are trained with audio tokens as a first-class part of their vocabulary alongside text. Vocal nuance (tone, hesitation, laughter) that text transcription throws away is preserved, and latency drops by eliminating separate STT and TTS steps.
Compare open speech-to-speech models →
Audio In
→
Speech-Native Model
→
Audio Out
Streaming Fusion
Hybrid approaches that keep the pipeline structure but tightly couple the components. The STT streams partial transcripts to the LLM, which starts generating while the caller is still speaking, and TTS begins synthesizing the first words before the LLM finishes its response. LiveKit, Pipecat, and Vocode use this approach.
Audio In
⇄
STT
⇄
LLM
⇄
TTS
⇄
Audio Out
The traditional pipeline isn't going away — it's proven, debuggable, and lets you swap components independently. But as speech-native models mature, expect the lines between STT, LLM, and TTS to keep blurring. The best architecture depends on your constraints: latency requirements, hardware budget, and how much control you need over each stage.
Latency Budget
Latency is one of the most important factors in whether a voice agent feels natural or robotic. For a conversation to feel truly indistinguishable from talking to another person, you should shoot for a total round trip under 500ms. Above this threshold, callers begin to notice. Above 1 second, the conversation begins to break down. Managing your latency budget across every component of the pipeline is critical to the caller experience.
VAD Processing20ms
STT (Streaming)200ms
LLM First Token300ms
TTS First Audio150ms
Network / Jitter80ms
Total Round Trip~750ms
Target: <500ms for natural feel. >1500ms feels like talking to someone overseas. >2000ms breaks conversation flow entirely.
Turn-Taking & Call Control
One of the trickier problems in voice agents is knowing when to talk, when to listen, and how to keep the conversation on track. Getting this wrong makes even a fast, accurate agent feel unnatural.
Endpointing
Detecting when the caller has finished speaking. Too aggressive = interruptions. Too conservative = awkward pauses.
Barge-in
Allowing callers to interrupt the agent mid-speech. Essential for natural conversation but challenging to implement correctly.
Backchanneling
Injecting 'mm-hmm', 'I see', 'right' while the caller speaks. Makes the agent feel alive and attentive.
Silence Handling
What to do when the caller goes silent? Prompt them? Wait? How long? Each scenario needs different treatment.
Call Control
Keeping the caller on task — redirecting tangents, summarizing to confirm understanding, and guiding the conversation toward resolution without feeling pushy.
Check Your Understanding
Question 1 / 2
What's the #1 factor that determines whether a voice agent feels 'natural'?