Speech-to-Speech Models

Everything so far chains three models together: STT turns audio into text, the LLM answers in text, and TTS turns it back into audio. A newer class of open-weight models collapses that loop into a single network that listens and speaks directly. Some of them listen while they talk, too. This page covers what exists in the open-weight world as of September 2026, what it takes to run each model locally, and why it matters to anyone defending a phone line.

What "Voice-to-Voice" Means

"Voice-to-voice" gets used for three quite different things. Only the first one replaces the pipeline.

End-to-end speech models

A single model takes audio tokens in and produces audio tokens out, usually with an internal text stream it "thinks" in. Tone, hesitation and laughter survive, and there is no STT → TTS handoff to wait on. This page is mostly about these.

Caller Audio

→

One Speech Model

→

Agent Audio

Speech-in, text-out models

Models like Ultravox understand audio directly but only answer in text, so you still need a TTS. They remove the STT hop, not the whole loop.

Caller Audio

→

Audio LLM

→

Text

→

TTS

Voice conversion

Tools like RVC, Seed-VC and OpenVoice don't understand or answer anything. They re-skin one person's speech so it sounds like another person, in real time. That is the classic live-deepfake vishing tool, covered in Voice Cloning → Local Tools.

Your Voice

→

VC Model

→

Target Voice

Full-Duplex Models

Full-duplex models listen and speak at the same time, like a person on a phone call. They can be interrupted mid-sentence, drop in "mm-hm" backchannels, and reply in roughly 200 ms. That is faster than most humans, and far below the ~750 ms budget of the cascaded pipeline in Agent Architecture.

Moshi · Kyutai · Sep 2024

Code MIT (Python) + Apache-2.0 (Rust) · Weights CC-BY 4.07B~16 GB VRAM (bf16), less quantized · Mac via MLX

The first open full-duplex speech model, and still the easiest one to run. It models the user's audio and its own audio as two parallel streams, plus an "inner monologue" text stream. Kyutai reports 160 ms theoretical latency and about 200 ms in practice. It ships two fixed voices, Moshiko (male) and Moshika (female), and speaks English only. The backbone is small, so expect chatty rather than smart. It is a great demo of how natural a machine can sound, not a capable agent.

Sources: GitHub · Paper (arXiv 2410.00037)

Moshika-RL · Kyutai · Jun 2026

Weights CC BY-NC 4.07BSame as Moshi

A reinforcement-learning fine-tune of Moshika aimed at smoother turn-taking, backchannels and interruptions. It is a good illustration that "conversational feel" is now something vendors train for directly.

Gated, non-commercial weights that the model card limits to research use, and it explicitly forbids impersonating people. Fine for a classroom demo; don't build on it.

Sources: Model card · Paper (arXiv 2606.11167)

PersonaPlex-7B · NVIDIA · Jan 2026

Code MIT · Weights NVIDIA Open Model License (gated)7B (Moshi-based)Tested on A100; CPU offload flag available

Fine-tuned from Moshiko to take a role prompt (text) and a voice prompt (audio), so you can make a full-duplex agent that plays a specific persona. "Friendly IT helpdesk" is exactly the kind of persona a vishing crew would want. It ships preset voices and is English only. You must accept NVIDIA's license on Hugging Face before downloading.

Sources: Model card · GitHub

MiniCPM-o 4.5 · OpenBMB · 2026

Apache-2.0 (verify on model card)9B~11 GB int4 · GGUF / llama.cpp / Ollama builds

An omni model (vision + audio) with full-duplex streaming. It is built from a Whisper encoder, a Qwen3-8B backbone and a CosyVoice2 speech decoder. Two things make it notable here: it runs on a single consumer GPU or Apple Silicon, and it can clone a voice from one reference clip during the conversation. English and Chinese.

Sources: Model card

NemotronLabs VoiceChat-11B · NVIDIA · Aug 2026

OpenMDW-1.1~11B (Nemotron Nano 9B v2 based)Data-center GPU (A100 / H100 class)

The first open full-duplex model with live tool calling, meaning it can look something up or take an action while it keeps talking. That closes one of the biggest gaps between speech-native models and pipelines. NVIDIA reports ~450 ms response latency, and the model is English only. It needs data-center hardware, so treat it as a preview of where things are going, not a lab tool.

Sources: Model card

Turn-Based "Omni" Models

These models take speech in and give speech out in one network, but they take turns like a walkie-talkie rather than overlapping. In exchange they tend to be smarter, more multilingual, and some support function calling.

Qwen2.5-Omni (3B / 7B) · Alibaba · Mar 2025

Code Apache-2.0 · 7B weights Apache-2.0 · 3B weights Qwen Research License3B, 7B3B and int4 builds fit consumer GPUs

A "Thinker-Talker" design: the Thinker is an LLM that understands audio, video and text, and the Talker streams speech from its hidden states. It has two fixed voices (Chelsie and Ethan). The 3B version is the most practical local omni model for a lab laptop with a mid-range GPU.

The 3B checkpoint uses the non-commercial Qwen Research License. Only the 7B weights are Apache-2.0.

Sources: GitHub · Paper (arXiv 2503.20215)

Qwen3-Omni-30B-A3B · Alibaba · Sep 2025

Apache-2.030B MoE (~3B active)~79 GB+ in BF16 · vLLM recommended

The most capable open omni model: it understands speech in 19 languages and speaks 10, has native function calling, and reports ~234 ms to first audio packet. Because the transcript never exists as a separate step, text-based guardrails that inspect an STT transcript have nothing to inspect. Keep that in mind for the Attack Surface section. Its successor, Qwen3.5-Omni (2026), is proprietary and API-only and does not meet this course's open-weight requirement.

Sources: GitHub · Paper (arXiv 2509.17765)

Step-Audio 2 mini · StepFun · Aug 2025

Apache-2.0~8BCUDA GPU, vLLM supported

End-to-end speech conversation with tool calling and paralinguistic understanding (emotion, speaking style). It covers English, Chinese, Japanese and more. Only the mini variants are open; the full Step-Audio 2 is API-only.

Sources: GitHub · Paper (arXiv 2507.16632)

Also worth knowing · various · 2024–2025

Mixed — check each

Fun-Audio-Chat-8B (Alibaba Tongyi, Apache-2.0, ~24 GB) supports speech function calling. Kimi-Audio-7B (Moonshot) is a strong universal audio model. GLM-4-Voice-9B (Zhipu) can be told to change emotion, speed and dialect. Mini-Omni2 (MIT, ~0.5B) is tiny enough to read end to end if you want to learn how these models work. Most of this group is Chinese/English-centric.

GLM-4-Voice's code is Apache-2.0, but its weights use a custom GLM model license. LLaMA-Omni 2 weights are academic-use only.

Sources: Fun-Audio-Chat · Kimi-Audio · GLM-4-Voice · LLaMA-Omni2 · Mini-Omni2

Speech-to-Speech Translation

Translation models keep who is speaking but change the language. For attackers, that removes accent and fluency as a tell: someone who doesn't speak the target's language can hold a live call in it, in their own voice.

Hibiki / Hibiki-Zero · Kyutai · Feb 2025 / Feb 2026

Code MIT + Apache-2.0 · Hibiki weights CC-BY 4.0 · Hibiki-Zero weights CC BY-NC-SA 4.01B–3BHibiki 1B runs on-device (MLX); Hibiki-Zero ~8–12 GB

Simultaneous speech translation that transfers the speaker's voice. Hibiki translates French to English; Hibiki-Zero adds Spanish, Portuguese and German into English.

Sources: Hibiki · Paper (arXiv 2502.03382) · Hibiki-Zero

SeamlessStreaming / SeamlessExpressive · Meta · Dec 2023

Code MIT · Streaming weights CC BY-NC 4.0 · Expressive under the Seamless License~2BCUDA GPU

Many-language streaming translation, with an Expressive variant that keeps tone and pacing. Notably, Meta runs Seamless output through a watermarking step, which makes it one of the few open speech models that watermarks at all. Meta's open watermarking research is AudioSeal.

Non-commercial weights. SeamlessExpressive is research-only under Meta's custom Seamless License and is released only through an access request form.

Sources: GitHub · Paper (arXiv 2312.05187)

Open "Voice Loop" Projects

Not every "speech-to-speech" repo is a single model. These projects are well-tuned cascades. They are worth knowing because they get close to speech-native latency while keeping a swappable LLM and a text transcript.

Kyutai Unmute

Kyutai streaming STT + any vLLM text model + Kyutai TTS. MIT, needs 16 GB+ VRAM. Wraps any LLM with a Moshi-like conversational layer.

HF speech-to-speech

Hugging Face's modular Apache-2.0 pipeline: Silero VAD → STT → any OpenAI-compatible LLM → TTS (e.g. Kokoro). Easy to point at your local llama.cpp server.

Sesame CSM-1B

Apache-2.0 conversational speech model: it uses previous turns of audio as context, so replies sound like they belong in the conversation. It cannot generate text, so it still needs an LLM in front.

Model Comparison

A quick reference. Latency figures are the vendors' own claims on their own hardware, and VRAM figures are approximate.

Model	Type	Weights License	Size	Duplex	Custom Voice	Languages	Local VRAM	Latency (claimed)
Moshi	End-to-end	CC-BY 4.0	7B	Yes	No (2 presets)	EN	~16 GB	~200 ms
PersonaPlex-7B	End-to-end	NVIDIA Open Model (gated)	7B	Yes	Voice prompt	EN	A100-tested	~170 ms
MiniCPM-o 4.5	End-to-end omni	Apache-2.0	9B	Yes	Clone from clip	EN, ZH	~11 GB (int4)	~0.6 s
NemotronLabs VoiceChat	End-to-end + tools	OpenMDW-1.1	~11B	Yes	—	EN	A100 / H100 class	~450 ms
Qwen2.5-Omni	Turn-based omni	7B Apache-2.0; 3B research-only	3B / 7B	No	No (2 presets)	Multilingual	Consumer GPU (3B)	—
Qwen3-Omni	Turn-based omni + tools	Apache-2.0	30B MoE	No	No (presets)	10 spoken	~79 GB	~234 ms
Step-Audio 2 mini	Turn-based + tools	Apache-2.0	~8B	No	Timbre switching	EN, ZH, JA +	Single GPU	—
Hibiki	Translation	CC-BY 4.0	1B / 2B	Streaming	Keeps speaker voice	FR → EN	On-device (1B)	—

Speech-to-Speech vs. the Pipeline

Neither approach wins everywhere. For training scenarios where you need scripted, gradable behavior, the cascade is usually still the right choice. Speech-native models shine when the point is realism.

Factor	Cascade (STT → LLM → TTS)	Speech-to-Speech Model
Latency	~500–1500 ms; needs VAD and endpointing heuristics	~170–450 ms (full-duplex); natural interruptions
Naturalness	Loses tone and emotion at the text step	Keeps prosody, laughter, hesitation, backchannels
Reasoning	Use any LLM, any size	Tied to a 3–9B backbone; weaker at complex tasks
Prompt control	Precise system prompts and scripts	Role prompts work, but drift is more common
Transcript / logging	Free: STT output is a text log	Must be reconstructed (the model's inner text, or a separate STT)
Guardrails	Filter text between each stage	No text boundary to filter in real time
Tool use	Any framework (LiveKit, Pipecat)	Only a few models (Qwen3-Omni, Step-Audio 2, VoiceChat)
Voice choice	Any TTS, including cloned voices	Mostly fixed presets; a few support voice prompts
Best for	Scripted scenarios, grading, production	Realism demos, showing how convincing agents have become

Security Implications

Speech-native models change what defenders can rely on, and what they can inspect.

The "robot pause" tell is gone

Cascaded bots give themselves away with a half-second gap, talking over you, or ignoring an interruption. Full-duplex models take ~200 ms turns and yield when interrupted. Stop training people to listen for lag.

No transcript to inspect

Many voice-agent guardrails work on the STT transcript. An end-to-end model never produces one as a separate artifact, so prompt-injection filters and content policies need to work on audio or on the model's internal text stream.

Almost nothing is watermarked

Meta's Seamless is the exception, not the rule. Kyutai chose not to watermark its TTS because watermarks on open models "can easily be deactivated", and in its tests they were stripped just by re-encoding the audio. Instead it restricts cloning to pre-computed voice embeddings.

Cloning barriers are tiny

MiniCPM-o clones from a single reference clip inside a live conversation. Voice-conversion tools like Seed-VC need 1–30 seconds of audio. A voicemail greeting is enough source material.

Translation removes the accent tell

Voice-preserving translation like Hibiki lets a caller speak a language they don't know, in their own voice, in near real time.

License traps: "downloadable" is not the same as "usable." Moshika-RL, Hibiki-Zero, the Qwen2.5-Omni 3B and Seamless are non-commercial. LLaMA-Omni 2 is academic-only. GLM-4-Voice and PersonaPlex use custom vendor licenses. Seed-VC is GPL-3.0. Read the model card before building on any of them.

If you build a demo agent with any of these, consider stamping its output with AudioSeal (MIT, runs locally), and only use voices from people who have consented.

Try It: Moshi Locally

Moshi is the simplest open full-duplex model to run: one pip package, and a built-in web UI where you can talk to it. It is not pre-installed on the lab machines. It needs a CUDA GPU with ~16 GB VRAM, or an Apple Silicon Mac using the MLX build.

Privacy Notice

The first run downloads the model weights (~15 GB) from Hugging Face. After that, inference is fully local and no audio leaves your machine. Set HF_HUB_OFFLINE=1 once the weights are cached to prevent any further network calls.

bash

# === NVIDIA GPU (PyTorch) ===
sudo mkdir -p /opt/moshi && sudo chown $USER /opt/moshi
cd /opt/moshi
uv venv --python 3.12
uv pip install -U moshi

# Start the server with the female voice (Moshika)
# then open http://localhost:8998 and allow microphone access
.venv/bin/python -m moshi.server --hf-repo kyutai/moshika-pytorch-bf16

# Or the male voice (Moshiko)
.venv/bin/python -m moshi.server --hf-repo kyutai/moshiko-pytorch-bf16

# === Apple Silicon (MLX, 4-bit quantized) ===
uv pip install moshi_mlx
.venv/bin/python -m moshi_mlx.local_web -q 4

Things to try: interrupt it mid-sentence, say nothing and wait, or talk over it. Compare how it feels with the llama.cpp + Piper demo in The LLM Brain. Then ask it something factual. You'll quickly see the trade between fluency and intelligence.

Check Your Understanding

Question 1 / 5

Why can speech-to-speech models weaken text-based guardrails?

Loading…