Local AI Tools

Run everything locally — no cloud, no API keys, no data leaving your machine. But before diving into the tools, it helps to understand the different approaches to voice cloning and how they've evolved — each requires different amounts of data, compute, and effort.

Voice Cloning Approaches

AI voice cloning has evolved rapidly. Understanding the three main approaches — and how much data each one needs — is key to understanding both the technology and the threat landscape.

Fine-Tuning

The original approach. You train (or fine-tune) a model on hours of labeled audio from the target speaker. This produces the highest quality clones but requires significant time, compute, and data. Tools like RVC use this approach — you provide 10-20+ minutes of clean audio and the model learns the speaker's voice characteristics over many training iterations.

Fine-tuning audio models is a broad topic beyond the scope of this training. To learn more, see the Hugging Face Audio Course for a comprehensive introduction to training and fine-tuning audio models.

One-Shot Cloning

A middle ground. You provide a single audio sample along with its exact text transcription. The model uses both the audio and the aligned text to understand the speaker's voice. Earlier versions of Coqui TTS (v1) and Spark TTS use this approach — you provide both the wav file and a matching transcript.

bash

# One-shot cloning with Spark TTS
# Requires both the audio AND its exact transcription
spark-tts --text "My fellow Americans." --device 0 \
    --save_dir ~/callcentervillage/voice-cloning \
    --prompt_speech_path /opt/spark-tts/pretrained_models/potus/reference.wav \
    --prompt_text "$(cat /opt/spark-tts/pretrained_models/potus/reference.txt)"

# Spark TTS will output the save path, e.g.:
# Audio saved at: ~/callcentervillage/voice-cloning/<YYYYMMDDHHMMSS>.wav
# Play it back with:
# play ~/callcentervillage/voice-cloning/<YYYYMMDDHHMMSS>.wav

# If the text doesn't match the audio exactly, quality suffers

Zero-Shot Cloning

The current state of the art — and the most dangerous from a security perspective. Just provide a few seconds of audio. No transcript needed, no training required. The model extracts the speaker's voice characteristics directly from the audio and can immediately speak any new text in that voice. This is what Coqui XTTS v2 and most modern commercial services use.

This command can take several minutes to run on CPU. Be patient — the model is doing a lot of work to clone the voice.

bash

# Zero-shot cloning with Coqui XTTS v2
# Just provide audio — no transcript, no training
cd ~/callcentervillage/voice-cloning

coqui-tts --model_name tts_models/multilingual/multi-dataset/xtts_v2 \
    --speaker_wav /opt/coqui-tts/celebrity-voice.wav \
    --language_idx en \
    --text "I'm entering my cloned era." \
    --out_path output.wav && play output.wav

Source audio: YouTube

Reference Tools

Hands-on tools for voice cloning, conversion, and synthesis — all running locally on your machine.

Coqui TTS

Open-source TTS with zero-shot cloning via XTTS v2 model. 16+ languages.

Coqui AI shut down in 2024. This tool has significant dependency issues and may require manual troubleshooting to install. It's pre-configured on these laptops, but setting it up elsewhere can be painful.

PROS

+ Zero-shot

+ Multi-language

+ Fine-tunable

CONS

- Coqui shut down

- GPU recommended

bash

# Preinstalled on Call Center Village laptops
# Wrapper at /usr/local/bin/coqui-tts → /opt/coqui-tts/.venv/bin/tts
cd ~/callcentervillage/voice-cloning

coqui-tts --model_name tts_models/multilingual/multi-dataset/xtts_v2 \
    --speaker_wav input.wav --language_idx en \
    --text "Somebody once told me, the world is gonna roll me." \
    --out_path output.wav && play output.wav

Supporting Tools

Voice cloning doesn't happen in isolation — you need to understand what was said before you can generate a convincing response. A speech-to-text tool like whisper.cpp lets you transcribe recordings, voicemails, or intercepted audio into text you can work with. A local LLM like llama.cpp takes that transcript and generates contextually appropriate responses — matching tone, vocabulary, and conversation flow — that you can then feed into a TTS or voice conversion tool.

Together, these tools close the loop: listen, understand, respond, and speak — all running locally with zero cloud dependency and no API logs. The Call Center Village laptops use whisper.cpp and llama.cpp directly, but there are user-friendly alternatives worth knowing about — Ollama, LM Studio, Open WebUI, LibreChat, Jan all make it easier to run local models with polished UIs.

See Appendix → Additional Resources for details on each.

whisper.cpp

Local speech-to-text — transcribe audio to text entirely on your machine.

Powered by OpenAI's Whisper model, compiled to run efficiently on CPU. Transcribe audio files, extract text from recordings for re-synthesis, or analyze call recordings — all offline.

bash

cd ~/callcentervillage/voice-cloning

# Transcribe an audio file (using the tiny model for speed on i3)
/opt/whisper.cpp/build/bin/whisper-cli \
  -m /opt/whisper.cpp/models/ggml-tiny.en.bin \
  -f input.wav

# Transcribe with timestamps
/opt/whisper.cpp/build/bin/whisper-cli \
  -m /opt/whisper.cpp/models/ggml-tiny.en.bin \
  -f input.wav -otxt

# Output as SRT subtitles
/opt/whisper.cpp/build/bin/whisper-cli \
  -m /opt/whisper.cpp/models/ggml-tiny.en.bin \
  -f input.wav -osrt

# Use the small model for better accuracy (slower)
/opt/whisper.cpp/build/bin/whisper-cli \
  -m /opt/whisper.cpp/models/ggml-small.en.bin \
  -f input.wav

The tiny model is fastest and works well on low-powered hardware. Use small for better accuracy if you can wait a bit longer. Models ending in .en are English-only and slightly more accurate for English.

Model	Size	Memory
tiny / tiny.en	75 MB	~390 MB
base / base.en	142 MB	~500 MB
small / small.en	466 MB	~1.0 GB
medium / medium.en	1.5 GB	~2.6 GB
large-v3	3.1 GB	~4.7 GB

For these laptops, tiny or base are recommended. See all available models at huggingface.co/ggerganov/whisper.cpp.

llama.cpp

Run large language models locally — generate scripts and dialogue without the cloud.

Generate realistic call scripts, social engineering dialogue, or conversational responses — all without sending data to the cloud. Perfect for generating text that a cloned voice can speak.

bash

cd ~/callcentervillage/voice-cloning

# Generate a simple response (using a small quantized model for i3)
/opt/llama.cpp/build/bin/llama-cli \
    -m /opt/llama.cpp/models/tinyllama-1.1b-chat.Q4_K_M.gguf \
    -p "Write a short phone script where a bank employee asks a customer to verify their identity." \
    -n 150

# Interactive chat mode
/opt/llama.cpp/build/bin/llama-cli \
    -m /opt/llama.cpp/models/tinyllama-1.1b-chat.Q4_K_M.gguf \
    --interactive \
    -p "You are a call center agent. Respond naturally to the customer."

# Generate text and save to file (for feeding into TTS)
/opt/llama.cpp/build/bin/llama-cli \
    -m /opt/llama.cpp/models/tinyllama-1.1b-chat.Q4_K_M.gguf \
    -p "Write a convincing voicemail message from a bank about suspicious activity." \
    -n 100 > script.txt

Smaller quantized models (Q4_K_M) run best on limited hardware. The output won't match GPT-4, but it's enough to generate realistic scripts entirely offline.

Putting It All Together

These tools chain together to build fully local voice cloning pipelines — no internet connection required. Here are two realistic scenarios:

Scenario A: Clone a voice and put words in their mouth

You have a voice sample of someone. Generate a script and make it sound like they said it.

bash

cd ~/callcentervillage/voice-cloning

# 1. Record or obtain a voice sample of the target
# (You can also use Audacity to record)
rec -r 44100 -c 1 target_voice.wav trim 0 20

# 2. Generate a script with llama.cpp
/opt/llama.cpp/build/bin/llama-cli \
    -m /opt/llama.cpp/models/tinyllama-1.1b-chat.Q4_K_M.gguf \
    -p "Write a short voicemail about a package delivery." \
    -n 80 > script.txt

# 3. Clone the voice with the generated script
tts --model_name tts_models/multilingual/multi-dataset/xtts_v2 \
    --speaker_wav target_voice.wav \
    --language_idx en \
    --text "$(cat script.txt)" \
    --out_path cloned_output.wav

Scenario B: Intercept, understand, and respond as someone

You have a recording of a conversation. Transcribe it, generate a contextual follow-up, and deliver it in the original speaker's voice.

bash

cd ~/callcentervillage/voice-cloning

# 1. Transcribe the recording to understand what was said
/opt/whisper.cpp/build/bin/whisper-cli \
    -m /opt/whisper.cpp/models/ggml-tiny.en.bin \
    -f recorded_call.wav \
    -otxt

# 2. Feed the transcription to llama.cpp for a contextual response
/opt/llama.cpp/build/bin/llama-cli \
    -m /opt/llama.cpp/models/tinyllama-1.1b-chat.Q4_K_M.gguf \
    -p "The caller said: $(cat recorded_call.wav.txt). Write a convincing follow-up response as if you are the same person calling back." \
    -n 100 > response.txt

# 3. Synthesize the response in the original speaker's voice
tts --model_name tts_models/multilingual/multi-dataset/xtts_v2 \
    --speaker_wav recorded_call.wav \
    --language_idx en \
    --text "$(cat response.txt)" \
    --out_path cloned_response.wav

Check Your Understanding

Question 1 / 5

What makes Piper TTS different from Coqui TTS?

Loading…