AI Voice Cloning

Modern neural networks clone a voice from minutes — or seconds — of audio.

Before There Was AI

Before AI-based TTS, rule-based synthesizers used formant models and pre-recorded phoneme snippets to generate speech. They sound robotic and can't clone voices, but they're lightweight, fast, and require no GPU or training data. These tools are useful as a baseline to compare against AI-generated speech.

eSpeak NG

GitHub ↗

Formant-based synthesizer supporting 100+ languages. Robotic but extremely fast and runs anywhere.

bash

# Preinstalled on Call Center Village laptops
# sudo apt install espeak-ng
cd ~/callcentervillage/voice-cloning

# Generate speech to audio file
espeak-ng "Hello, this is your bank calling" -w output.wav && play output.wav

# Change voice/language (Spanish)
espeak-ng -v es "Hola, le llama su banco" -w output.wav && play output.wav

# List available voices
espeak-ng --voices

Festival

Project Page ↗

Unit-selection synthesizer from University of Edinburgh. Uses pre-recorded speech snippets stitched together — more natural than formant synthesis.

bash

# Preinstalled on Call Center Village laptops
# sudo apt install festival
cd ~/callcentervillage/voice-cloning

# Save speech to audio file
echo "Hello, this is your bank calling" | text2wave > output.wav && play output.wav

# Play speech interactively
# Requires audio device
echo "Hello, this is your bank calling" | festival --tts

Compare the output of these tools to the AI-generated TTS example later on this page — the difference in quality is what makes AI voice cloning such a powerful (and dangerous) technology.

AI-Powered TTS and Voice Conversion

Unlike the rule-based tools above, these use neural networks to generate human-sounding speech — and can clone specific voices.

How TTS Cloning Works

Text-to-speech cloning takes typed text and generates audio that sounds like a specific person. You provide the words, and the AI handles everything else — pronunciation, rhythm, and vocal style. The output is entirely AI-generated; there is no original human recording being modified.

Click a step to learn more:

Text

Phonemes

Voice Model

Mel Spec

Vocoder

Audio

Text Input

Raw text is fed into the system. This can be anything — a script, a sentence, or even a single word.

Example

"Hello, this is your bank calling about your account."

TTS and Voice Conversion Tool Reference

Coqui TTS ↗Bark ↗Tortoise ↗Piper ↗StyleTTS2 ↗MetaVoice ↗Qwen3-TTS ↗Spark TTS ↗F5-TTS ↗Chatterbox ↗RVC ↗FreeVC ↗DDSP-SVC ↗OpenVoice ↗

Quality vs. Effort

Method	Audio	Compute	Real-Time
eSpeak NG / Festival	None	None	Yes
TTS (zero-shot)	3-10s	Medium	Sometimes
TTS (fine-tuned)	5-30min	High	No
VC (zero-shot)	3-10s	Medium	Yes
VC (fine-tuned / RVC)	10-30min	High	Yes*
Commercial API	10-30s	None	Yes

Speaker Embeddings

Remember the speaker embedding step from voice conversion above? A speaker embedding is typically a 256 to 512 dimension vector that captures what makes a voice unique — a mathematical fingerprint. Here's an example of how they might actually work under the hood.

How Speaker Embeddings Work

Pitch RangeFormantsBreathinessVibratoPaceNasality...256+ dims

What does this actually look like?

A dimension (dim) is a single measurement of a voice characteristic. A vector is just a list of these measurements — like a row of numbers that together describe a voice.

Each dimension is a number (usually between -1 and 1) representing how much of that characteristic the voice has. For example:

dim 1 (pitch range):  0.72  ← wide pitch variation
dim 2 (breathiness):  -0.15 ← not very breathy
dim 3 (nasality):    0.41  ← somewhat nasal
dim 4 (pace):        -0.63 ← slower speaker
...256 more dimensions

The full vector — all 256+ numbers together — is the voice's unique fingerprint. In code, it's just an array of numbers:

// Speaker embedding for "Patrick's voice"
speaker_vector = [
  0.72, -0.15, 0.41, -0.63, 0.28, 0.89, -0.44, 0.17,
  0.55, -0.31, 0.08, 0.93, -0.72, 0.36, -0.19, 0.61,
  ...  // additional dimensions
  0.47, -0.05, 0.33, -0.28, 0.14, 0.82, -0.51, 0.69
]

Two voices with similar vectors will sound alike. The AI compares and manipulates these vectors to clone a voice.

The number of dimensions (256, 512, etc.) is a design choice by the model's creators — not a fixed rule. More dimensions can capture finer details but cost more compute. 256 is common because it's proven to be "enough" for most voice cloning tasks. Also, the individual dimensions don't map neatly to concepts like "breathiness" or "pitch" — the model learns its own abstract representations during training. The labels above are simplified for illustration.

Vector Databases & Voice Lookup

These dimensions can be stored and searched in vector databases, which find similar voices by comparing the mathematical distance between embeddings. This is how systems can take an unknown voice and quickly match it against thousands of known voiceprints.

To learn more about how this works, see nearest neighbor search and vector databases.

Want to try vector search yourself? pgvector is a PostgreSQL extension that adds vector similarity search — a great way to get started storing and querying speaker embeddings with a database you may already know.

Check Your Understanding

Question 1 / 4

What is the main limitation of non-AI TTS tools like eSpeak NG and Festival?

Loading…