Loading…
The LLM Brain
The large language model (LLM) — also referred to as generative AI — is the decision-making center. It takes the text from the speech-to-text engine, generates a response, and sends it to the text-to-speech engine to be spoken back. It can also take actions via tool/function calling.
llama.cpp
Run language models entirely on your machine. No API keys, no data leaving your network. The quality and speed of local models will depend on how powerful your machine is — more RAM and a dedicated GPU allow you to run larger, more capable models.
bash
# llama.cpp is pre-built at /opt/llama.cpp # Chat with a local LLM in the terminal /opt/llama.cpp/build/bin/llama-cli \ -m /opt/llama.cpp/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \ -cnv -p "You are a helpful assistant." # Try a smaller model /opt/llama.cpp/build/bin/llama-cli \ -m /opt/llama.cpp/models/qwen2.5-0.5b-instruct-q4_k_m.gguf \ -cnv -p "You are a helpful assistant." # Run with built-in web chat UI (like Ollama) /opt/llama.cpp/build/bin/llama-server \ -m /opt/llama.cpp/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \ --jinja --host 0.0.0.0 --port 8080 -c 0 # Open http://localhost:8080 in your browser for a full chat interface # Run as OpenAI-compatible API server (no chat UI) /opt/llama.cpp/build/bin/llama-server \ -m /opt/llama.cpp/models/Llama-3.2-1B-Instruct-Q4_K_M.gguf \ --host 0.0.0.0 --port 8080 -c 4096 # Any framework that works with OpenAI API works with this! # Use with Open WebUI, LibreChat, or your own applications
Cloud LLM Providers
These companies are not sponsors or affiliated with this training or Call Center Village. They're listed for educational awareness only.
That being said, if any of you are reading this — Call Center Village is always looking for sponsors!
That being said, if any of you are reading this — Call Center Village is always looking for sponsors!
Claude model family with strong reasoning. Good for complex decision trees and tool use.
Ultra-fast cloud inference for open models. Extremely low time-to-first-token.
Unified API gateway for multiple LLM providers. Access hundreds of models through a single endpoint.
GPT model family with native function calling. Fast and affordable options for voice agents.
Have a suggestion for a cloud LLM provider to include here? Email us at [email protected]
System Prompts
A system prompt is the set of instructions you give to an LLM before it interacts with a user. It defines the model's personality, rules, capabilities, and boundaries — essentially telling it who it is and how to behave. Every LLM-powered application uses one, whether it's a chatbot, a coding assistant, or a voice agent. For voice agents specifically, system prompts need extra considerations because the output is spoken aloud, not read on a screen.
Earlier, we used -p "You are a helpful assistant." when chatting with llama.cpp — that's a system prompt too, just an extremely simple one. For a general-purpose chat, that's fine. But for a voice agent handling real calls, you need much more detail: rules about how to speak, what tools are available, guardrails for safety, and instructions for when things go wrong.
Example: Voice Agent System Prompt
Voice agent system prompts need special considerations compared to text chatbots. Hover over each section to learn why it matters.
ROLE DEFINITION:
You are a helpful customer service agent for Acme Corp.
VOICE-SPECIFIC RULES:
- Keep responses SHORT (1-3 sentences). Long responses feel like lectures.
- Use conversational language, not formal writing.
- Never use markdown, bullet points, or formatting — the caller can't see it.
- Spell out numbers and abbreviations: say "twenty three dollars" not "$23".
- Include filler words naturally: "Sure thing!", "Let me check on that."
- If you need to do a long task, say "One moment please" to fill silence.
- Handle mishearing gracefully: "I want to make sure I got that right..."
TOOLS AVAILABLE:
- lookup_account(phone_number) → returns account details
Format: 10-digit number, e.g. 5551234567
- check_balance(account_id) → returns current balance
- transfer_to_human(reason) → connects to live agent
Use when: customer requests a human, or issue is beyond your scope
GUARDRAILS:
- Never guess or make up information. If unsure, say so.
- Verify caller identity before sharing any account details. This step is important.
- Never process refunds over $500 without transferring to a human.
- If the caller becomes abusive, calmly offer to transfer to a supervisor.
ERROR HANDLING:
- If a tool call fails, apologize and offer to try again or transfer.
- Never invent account details or balances.
- If you cannot verify the caller, explain what they need and offer a callback.
TURN-TAKING:
- End your turns with a clear question or pause point.
- If the caller seems confused, ask a simpler question.
- If you don't understand, ask them to repeat — don't guess.
- After complex steps, confirm: "Does that make sense so far?"
A well-crafted system prompt is one of the most valuable parts of your voice agent — it's what sets your agent apart from everyone else's. Treat system prompts like code: save them, put them in version control, iterate on them, and test changes carefully. A great prompt is the difference between a demo and a product.
For a deeper dive into voice agent prompting, see the ElevenLabs Prompting Guide.
Streaming: The Key to Low Latency
The secret to fast voice agents is streaming everywhere. Don't wait for the LLM to finish generating before sending to TTS. Stream token-by-token.
With Streaming
0ms300ms600ms900ms1200ms1500ms
LLM
TTS
Caller hears
By streaming token-by-token, the TTS can start generating audio before the LLM has finished its response. This dramatically reduces the time-to-first-token — the caller starts hearing a response in under 500ms, even though the full phrase takes 1-2 seconds to finish saying. Without streaming, the caller would have to wait for the entire LLM response before hearing anything.
Without Streaming
0s2s4s6s8s10s12s13s
LLM
TTS
Caller hears
Give It a Try
These examples chain an LLM response directly into TTS. Notice how you have to wait for the full LLM generation to complete before you hear anything:
bash
# LLM → Piper TTS (non-streaming, fast but lower quality) time /opt/llama.cpp/build/bin/llama-simple \ -m /opt/llama.cpp/models/llama-3.2-3b-instruct-q4_k_m.gguf \ -p "Tell me a funny joke not about a chicken" \ 2>/dev/null \ | tail -n +2 \ | piper --model /opt/piper/models/en_US-lessac-medium.onnx --output_raw \ | aplay -r 22050 -f S16_LE # LLM → Kokoro TTS (non-streaming, slower but much higher quality) time /opt/llama.cpp/build/bin/llama-simple \ -m /opt/llama.cpp/models/llama-3.2-3b-instruct-q4_k_m.gguf \ -p "Tell me a funny joke not about a chicken" \ 2>/dev/null \ | tail -n +2 \ | kokoro \ | aplay -r 24000 -f S16_LE
On these laptops, you'll experience a 10-13 second delay of silence before hearing anything — imagine being on a phone call and waiting that long for a response.
Enter Voice Agent Frameworks!
Frameworks like LiveKit and Vocode handle token-level chunking, sentence boundary detection, and streaming orchestration — turning this painful 13-second delay into sub-second response times. We'll explore these frameworks in the next section.
Check Your Understanding
Question 1 / 3
Why is streaming token-by-token from the LLM to TTS critical for voice agents?