Voice Agent¶

A full voice conversation on device, driven by one object. The agent listens through the microphone, decides when the user has finished, gets a reply from a language model, speaks it, and lets the user interrupt at any point. Every stage runs locally — no audio, transcript or reply leaves the phone.

You choose the language model: an on-device pack for a fully offline assistant, or your own OpenAI-compatible endpoint. Everything else stays the same.

Main features

Full duplex: the user can interrupt mid-reply and the agent stops speaking.
Fully on device: microphone, transcription, reply and speech all run locally.
Bring your own LLM: on-device by default, or route replies to your provider without changing the rest.
Typed streams: live captions, final transcripts, reply text and speech level each on their own channel — a chat UI never parses a blob.
Tool calling: the same [Tool] closures as the LLM page, run between turns and surfaced as events.
Streaming speech: the first words play before the reply is complete.
Echo cancellation: the agent does not hear itself, so barge-in works on a speaker as well as headphones.

The voice agent pipeline, running entirely on device

In this page ¶

Here we will cover the following topics:

Supported models: which packs the agent accepts for each stage, and the LLM choice.
Quick start: an offline assistant, or one on your own LLM — Swift and Flutter side by side.
Follow the conversation: states, streams and events, and which to use for what.
Configure the agent: the knobs people change, with defaults, and which apply live.
Usage Guides: offline assistant, chat UI with captions, barge-in, turn-taking, tools, voices, custom nodes.
Troubleshooting: symptom → cause → fix.
Load Progress / Prefetch: first-run download of four packs, and warming them on a splash screen.

Supported models ¶

The agent is four stages; each accepts the packs from its own page.

Stage	Packs	Notes
Voice detection	`TheStageAI/silero-vad`	Always on device.
Speech to text	`TheStageAI/thewhisper-large-v3-turbo`, `TheStageAI/Qwen3-ASR-0.6B`	TheWhisper for live captions; see ASR (Speech-to-Text).
Language model	`TheStageAI/Qwen3-0.6B`, `TheStageAI/LFM2.5-230M` / `350M`, `TheStageAI/gemma-3-1b-it` — or any OpenAI-compatible endpoint	Local packs run tools; Gemma3 does not. See LLM (Language Model).
Text to speech	`TheStageAI/neutts-nano-multilingual`, `TheStageAI/Qwen3-TTS-12Hz-0.6B-Base`	Voices and languages per TTS (Text-to-Speech).
Turn detection (optional)	`TheStageAI/smart-turn-v3`	Neural end-of-turn; tolerates mid-sentence pauses.
Speaker ID (optional)	`TheStageAI/redimnet2`	Respond only to an enrolled voice; see Speaker embedding.

Which LLM?

You need…	Pick	Why
Works with no network; private by design	On-device (`TSLocalLLMProvider`)	Qwen3-0.6B or LFM2.5-350M; tools included.
Best answer quality, network is fine	Your endpoint (`TSOpenAICompatibleProvider`)	Any OpenAI-compatible chat API. Tools are the provider’s.
Lowest latency to first word	On-device, LFM2.5-230M	No round trip; first token in tens of milliseconds.

Quick start ¶

Two ways to run the agent. They differ only in where the reply comes from.

You want	Network	LLM
A fully offline assistant	none after first download	`TSLocalLLMProvider` / `'llm_provider': 'local'`
Your own model behind an API	required	`TSOpenAICompatibleProvider` / `'llm_provider': 'openai_compatible'`

Fully offline ¶

Swift

import TheStageSDK

try await TheStageAI.shared.initialize(api_token: "your-api-token")

// 1. Load the language model under a handle
try await TheStageAI.shared.start_model(
    model_name: "llm",
    engines_path: "TheStageAI/Qwen3-0.6B"
)

// 2. Point the agent at that handle
let llm = TSLocalLLMProvider(
    model_path: "llm",
    // weather, time, search — or []
    tools: DefaultTools.live,
    system_prompt: DefaultTools.voice_system_prompt
)
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)
config.tts_voice = "dave"

// 3. Subscribe, then start
let agent = TSVoiceAgent(config: config)
Task {
    for await text in agent.transcripts.recv() { chat.addUser(text) }
}
Task {
    for await delta in agent.llm_deltas.recv() { chat.appendAssistant(delta) }
}

// listening from here
try await agent.start()

Flutter

import 'package:thestage_apple_sdk/thestage_apple_sdk.dart';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');

// 1. Load the language model under a handle
await TheStageFlutterSDK.start_model(
  model_name: 'llm',
  engines_path: 'TheStageAI/Qwen3-0.6B',
);

// 2 + 3. Subscribe, then start with the same handle
final agent = TSVoiceAgent();
agent.transcripts.listen(chat.addUser);
agent.llm_deltas.listen(chat.appendAssistant);

await agent.start(config: {
  'vad': 'TheStageAI/silero-vad',
  'stt': 'TheStageAI/thewhisper-large-v3-turbo',
  'tts': 'TheStageAI/neutts-nano-multilingual',
  'tts_voice': 'dave',
  'llm_provider': 'local',
  // the start_model handle
  'llm_model': 'llm',
  // none | voice | web | phone
  'llm_tools': 'voice',
// listening from here
});

Your own LLM endpoint ¶

Swift

let llm = TSOpenAICompatibleProvider(
    endpoint: "https://api.openai.com/v1/chat/completions",
    api_key: "sk-YOUR-KEY",
    model: "gpt-4o-mini"
)
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)
config.system_prompt = "You are a concise voice assistant."

let agent = TSVoiceAgent(config: config)
try await agent.start()

Flutter

final agent = TSVoiceAgent();
await agent.start(config: {
  'vad': 'TheStageAI/silero-vad',
  'stt': 'TheStageAI/thewhisper-large-v3-turbo',
  'tts': 'TheStageAI/neutts-nano-multilingual',
  'llm_provider': 'openai_compatible',
  'llm_endpoint': 'https://api.openai.com/v1/chat/completions',
  'llm_api_key': 'sk-YOUR-KEY',
  'llm_model': 'gpt-4o-mini',
  'system_prompt': 'You are a concise voice assistant.',
});

Attention

Subscribe before start(). Streams attached afterwards miss the first turn. And one running agent per microphone — call stop() before starting another.

Important API ¶

Purpose	Swift	Flutter
Build	`TSAgentConfig(vad:stt:tts:llm:)` → `TSVoiceAgent(config:)`	`TSVoiceAgent()` + a config map on `start`
Run	`start()` / `stop()`	`start(config:)` / `stop()`
Mic on / off	`begin_listening()` / `pause_listening()`	`begin_listening()`
Text in, voice out	`send_request(_:)` (LLM → TTS), `say(_:)` (TTS only)	`send_request(text)`, `say(text)`
Cut playback	`interrupt()`	`interrupt()`
Voice	`set_voice(voice_id:voice_dir:language:)`	`set_voice(voice_id:, voice_dir:, language:)`
Memory	`history()` / `clear_history()` / `set_system_prompt(_:)`	`clear_history()` / `set_system_prompt(prompt)`
Live re-tune	`update_interrupt_config(…)` / `update_turn_config(…)`	`update_interrupt_config(…)`

Follow the conversation ¶

Typed streams carry content; events carry what happened

The agent gives you two kinds of output. Streams carry content you render — what the user said, what the assistant is saying. Events tell you what happened — a state change, a tool call, playback ending, an error. A chat UI needs both.

Swift

// "llm" is the handle you gave start_model in Quick start
let llm = TSLocalLLMProvider(
    model_path: "llm",
    tools: DefaultTools.live,
    system_prompt: DefaultTools.voice_system_prompt
)
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)
let agent = TSVoiceAgent(config: config)

// Content
Task { for await p in agent.partial_transcripts.recv() { caption.text = p } }
Task { for await t in agent.transcripts.recv()         { chat.addUser(t) } }
Task {
    for await d in agent.llm_deltas.recv() { chat.appendAssistant(d) }
}
Task {
    for await level in agent.tts_levels.recv() { avatar.mouth = level }
}

// What happened
Task {
    for await event in agent.events {
        switch event.kind {
        case .state_changed:
            statusLabel.text = event.data["state"] as? String
        case .response_done:
            if event.data["interrupted"] as? Bool == true {
                chat.markInterrupted()
            }
        case .tool_started:
            chip.show(event.data["name"] as? String ?? "")
        case .error:
            banner.show(event.data["message"] as? String ?? "")
        default: break
        }
    }
}

Flutter

// agent as in Quick start; agentConfig is its model map
final agent = TSVoiceAgent();

// Content
agent.transcripts.listen(chat.addUser);
agent.llm_deltas.listen(chat.appendAssistant);
agent.tts_levels.listen((level) => avatar.mouth = level);

// What happened (live captions arrive here on Flutter)
agent.events.listen((e) {
  final data = (e['data'] as Map?)?.cast<String, dynamic>() ?? {};
  switch (e['kind']) {
    case 'user_request_partial': caption.value = data['text'] as String? ?? '';
    case 'state_changed':        status.value  = data['state'] as String? ?? '';
    case 'response_done':
      if (data['interrupted'] == true) chat.markInterrupted();
    case 'tool_started':         chip.show(data['name'] as String? ?? '');
    case 'error':                banner.show(data['message'] as String? ?? '');
  }
});

Agent states ¶

The agent is always in exactly one state; state_changed fires on every move.

State	Means	Typical UI
`idle`	Not listening	“Tap to talk”
`loading`	Packs are loading	Progress
`listening`	Mic open, capturing speech	Mic active, live caption
`thinking`	Turn committed, reply not started	Spinner
`tool_calling`	Running a tool	“Looking that up…”
`speaking`	Playing the reply	Waveform, tap to interrupt
`sleeping`	Waiting for a wake word	Dimmed mic

Note

speaking ends when the speaker has drained, not when synthesis finishes. A “done” cue driven by playback_ended is accurate; one driven by the last text delta fires early.

Streams ¶

Carries	Swift	Flutter
Live user text while speaking	`partial_transcripts`	`events` → `user_request_partial`
Final user text	`transcripts`	`transcripts`
Assistant text, token by token	`llm_deltas`	`llm_deltas`
Speech level, for an avatar	`tts_levels`	`tts_levels`
Mic level, for a meter	`vad_probabilities`	`vad_probabilities`
Synthesised audio (PCM)	`tts_playback`	—

Events ¶

All arrive on events with kind and a data payload.

Kind	Payload	Fires when
`state_changed`	`state`	Any move between the states above.
`user_request_partial`	`text`	Live transcript while the user speaks.
`user_request`	`text`, `source`	The user’s turn is final.
`response_delta`	`delta`	Each chunk of the reply (same as `llm_deltas`).
`response_done`	`text`, `reason`, `interrupted`	Reply finished — check `interrupted`.
`tool_started` / `tool_ended`	`name`, `arguments` / `name`, `content`	A tool call begins / returns.
`playback_started` / `playback_ended`	— / `reason`	First audio reaches the speaker / speaker drained or cut by barge-in.
`wake_word`	—	The wake word was heard (wake-word mode).
`error`	`message`, `code`, `recoverable`	A recoverable or terminal failure.

Warning

response_done and playback_ended fire on a clean finish and on a barge-in. interrupted / reason tell them apart. Logging a turn as complete without checking is the most common source of wrong analytics.

Configure the agent ¶

Everything lives on TSAgentConfig (Swift) or the start map (Flutter). The defaults ship a working assistant; these are the fields people change, grouped by what they affect. Unless marked live, a change needs stop() then start().

Swift

var config = TSAgentConfig(
    vad: vadRepo,
    stt: sttRepo,
    tts: ttsRepo,
    llm: llm
)

// Reply
config.system_prompt = "You are Acme's assistant. Be brief."
config.max_tokens = 256
config.chat_memory = AgentMessageSlidingWindowMemory(max_turns: 10)

// Voice
// or tts_voice_dir / tts_language
config.tts_voice = "dave"

// Turn taking
// hush that ends a turn (VAD mode)
config.silence_timeout_ms = 608
// neural end-of-turn instead
config.turn_end_mode = .dnn
config.turn_detector = "TheStageAI/smart-turn-v3"

// Barge-in
// iOS default; .none on Mac speakers
config.interrupt_mode = .vad
config.interrupt_min_speech_ms = 600

// Audio
config.audio.sample_rate_out = 48_000
config.audio.aec_method = .VPIO

// Start muted, open the mic later
config.auto_listen = false

Flutter

// agent as in Quick start; agentConfig is its model map
final agent = TSVoiceAgent();

await agent.start(config: {
  'vad': vadRepo, 'stt': sttRepo, 'tts': ttsRepo,
  'llm_provider': 'local', 'llm_model': 'llm',

  // Reply
  'system_prompt': "You are Acme's assistant. Be brief.",
  'max_tokens': 256,
  'chat_memory_max_turns': 10,

  // Voice
  // or tts_voice_dir / tts_language
  'tts_voice': 'dave',

  // Turn taking
  'silence_timeout_ms': 608,
  'turn_end_mode': 'dnn',
  'turn_detector': 'TheStageAI/smart-turn-v3',

  // Barge-in
  'interrupt_mode': 'vad',
  'interrupt_min_speech_ms': 600,

  // Audio
  'sample_rate_out': 48000,
  'aec_method': 'vpio',

  'auto_listen': false,
});

Field	Default	Live?	Change it when
`system_prompt`	voice default	yes (`set_system_prompt`)	Persona, brand, scope. Keep it short — paid on every turn.
`max_tokens`	256	no	Replies get cut off — raise. Spoken replies rarely need more.
`chat_memory` / `chat_memory_max_turns`	sliding window	no	Long sessions on small packs run out of context — lower.
`tts_voice` / `tts_voice_dir` / `tts_language`	pack default	yes (`set_voice`)	See TTS (Text-to-Speech) for voices and packs.
`silence_timeout_ms`	608	no	Cuts users off mid-thought — raise to 1 000–1 500. Feels slow — lower.
`turn_end_mode` / `turn_detector`	`.vad`	no	Users pause mid-sentence — `.dnn` with `smart-turn-v3`.
`turn_eot_threshold`	0.85	yes (`update_turn_config`)	DNN mode: lower to commit sooner, raise to wait longer.
`interrupt_mode`	iOS `.vad` · macOS `.none`	yes (`update_interrupt_config`)	`.none` to disable barge-in; `.vad_speaker_id` to accept only the enrolled voice; `.vad_wake_word` to require the wake word.
`interrupt_min_speech_ms`	600	yes	Snappier barge-in — lower (more false triggers); twitchy — raise.
`vad_threshold`	0.5	no	Starts on background noise — raise; misses quiet users — lower.
`auto_listen`	true	no	Start muted and open the mic with `begin_listening()` after a welcome message.
`audio.*`	see Audio Nodes	no	Speaker rate, echo cancellation, custom audio source.

Troubleshooting ¶

Symptom	Cause	Fix
Agent interrupts itself	No working echo cancellation on this route.	iOS `.VPIO`; Mac speakers `interrupt_mode = .none` or headphones; raise `interrupt_min_speech_ms`.
Cuts the user off	Silence timer too short for this speaker.	Raise `silence_timeout_ms` or switch to `turn_end_mode = .dnn`.
Replies too slowly	Waiting too long for end of turn.	Lower `silence_timeout_ms` / `turn_eot_threshold`.
Never leaves `loading`	First-run download, or a pack failed.	Prefetch on a splash screen; watch `on_load_progress`; check the `error` event.
Silence forever	Mic permission, or `auto_listen = false` without `begin_listening()`.	Check permission; call `begin_listening()`.
Offline LLM never answers	`llm_model` ≠ the `start_model` handle, or the model was not loaded before `start()`.	Use the same string; await `start_model` first.
Playback too fast / slow	`sample_rate_out` ≠ the real speaker rate.	Set it to your session’s rate.
Barge-in never triggers	`interrupt_mode = .none`, or lockouts too long.	`.vad`; lower `interrupt_min_speech_ms`.
Turn logged as complete when it was cut	Ignoring `interrupted` on `response_done`.	Read the payload.

Load Progress / Prefetch ¶

First start() downloads and prepares up to four packs. Show progress, and warm them on a splash screen so the first conversation starts instantly.

Swift

// "llm" is the handle you gave start_model in Quick start
let llm = TSLocalLLMProvider(
    model_path: "llm",
    tools: DefaultTools.live,
    system_prompt: DefaultTools.voice_system_prompt
)
var config = TSAgentConfig(
    vad: "TheStageAI/silero-vad",
    stt: "TheStageAI/thewhisper-large-v3-turbo",
    tts: "TheStageAI/neutts-nano-multilingual",
    llm: llm
)

// Progress for every pack the agent loads
config.on_load_progress = { p in
    print("[\(p.model)] \(p.phase) \(Int(p.fraction * 100))%")
}

// Warm on a splash screen
for repo in ["TheStageAI/silero-vad",
             "TheStageAI/thewhisper-large-v3-turbo",
             "TheStageAI/neutts-nano-multilingual",
             "TheStageAI/Qwen3-0.6B"] {
    _ = try await TheStageAI.shared.prefetch_engines(repo_id: repo)
}

Flutter

TheStageFlutterSDK.on_progress.listen((e) =>
  print('[${e['model_name']}] ${e['phase']} '
        '${((e['progress'] ?? 0) * 100).round()}%'));

for (final repo in ['TheStageAI/silero-vad',
                    'TheStageAI/thewhisper-large-v3-turbo',
                    'TheStageAI/neutts-nano-multilingual',
                    'TheStageAI/Qwen3-0.6B']) {
  await TheStageFlutterSDK.prefetch_engines(repo_id: repo);
}

Teardown: await agent.stop(). If you start_model’d a local LLM, also stop_model that handle. Full contract: TheStage Apple SDK (Load Progress).

Voice Agent¶

In this page ¶

Supported models ¶

Quick start ¶

Fully offline ¶

Your own LLM endpoint ¶

Important API ¶

Follow the conversation ¶

Agent states ¶

Streams ¶

Events ¶

Configure the agent ¶

Usage Guides ¶

A chat screen with live captions ¶

The agent interrupts itself ¶

It cuts users off — or waits too long ¶

Let the assistant do things ¶

Change the voice from a settings screen ¶

Greet the user before listening ¶

Add your own stage to the pipeline ¶

Troubleshooting ¶

Load Progress / Prefetch ¶