Apple SDK: dynamic language-switching TTS

Overview

Build dynamic language-switching TTS with the TheStage Apple SDK: one on-device Qwen3TTSPipeline, a tagged mixed-language script, and a hot-swap of clone voice packs between spans — no second model and no engine reload.

The full runnable app is examples/tutor_tts in AppleSDK 1.3.0+. Clone that tree if you want to follow along in Xcode while you read.

API reference: TTS (Text-to-Speech).

What you use from the SDK

These types ship in TheStageSDK — you call them; you do not reimplement them:

SDK piece

Role in this tutorial

TheStageAI.shared

initialize(apiToken:) once; downloads / caches engines

Qwen3TTSPipeline

On-device Qwen3-TTS (NPU). One instance for the whole lesson

set_voice(voice_dir:voice_id:language:)

Hot-swap clone pack + language without reloading engines

infer_stream(text:config:stream_config:)

Stream 24 kHz PCM per language span

TTSGenerationConfig, recommended_stream_config

Sampling / chunking knobs for stable tutor audio

AudioStreamPlayer

Gapless enqueue of PCM across spans (start / enqueue / drain / stop)

Related docs: TTS (Text-to-Speech), Streaming.

What you implement in the app

Everything below is product / demo code (as in tutor_tts) — not part of the SDK surface:

You build

Why

Tagged phrase book

Editable <en>…</en><es>…</es> script (what is spoken)

Language tag parser

Ordered (tag, qwen_language, text) spans from that script

Bundled voice packs

One VoicePacks/tutor_<lang>/voice.json per language (who speaks). Encode with prepare_voice_packs or download HF tutor packs

Stream loop

For each span: set_voice → infer_stream → player.enqueue (+ optional inter-span silence)

Where it is useful

Anywhere the product must sound native in more than one language in a single turn, while keeping one speaker identity family and staying on device:

  • Language tutors — English coaching lines + target-language drills (the tutor_tts demo)

  • Travel / phrasebook apps — short bilingual cues (“say this at the café…”) with authentic pronunciation

  • Accessibility & education — bilingual instructions, captions read aloud, dual-language flashcards

  • Voice agents — an assistant that answers in the user’s language, then quotes a foreign phrase or product name without switching clouds

  • Localization QA — hear the same clone pack across en / es / fr / … before you ship copy

Why NPU matters here. Qwen3-TTS in the Apple SDK targets the Neural Engine (NPU) by default (device: "npu"). That keeps streaming synthesis efficient and battery-friendly on phones — important when a lesson plays many short spans back-to-back. You are not round- tripping audio to a cloud TTS for every language change.

Supported devices / environment

Requirement

Detail

OS

iOS / iPadOS (tutor_tts is an iPhone demo). SwiftPM APIs also run on Apple Silicon macOS; this example ships as an iOS app.

Hardware

Physical Apple Silicon device. Simulator is not supported (MLX / Metal). Prefer a recent iPhone for NPU TTS.

SDK

Apple SDK 1.3.0+ (Qwen3TTSPipeline, AudioStreamPlayer, set_voice)

Model

TheStageAI/Qwen3-TTS-12Hz-0.6B-Base (fleet pin via ModelRevisionMap)

Auth

API token from app.thestage.ai

Tutorial structure

Goals — ship the same loop as tutor_tts:

  1. A tagged script the UI (or an LLM) can edit

  2. A parser that turns tags into (language, text) spans

  3. Bundled voice packs (VoicePacks/tutor_<lang>/)

  4. A stream loop: set_voice → infer_stream → player

Demo (device capture — English glue + Spanish spans, pack swap per tag):

Mixed-language streaming TTS on device (~18 s)

Tagged script. Speakable text wrapped in paired language tags. Untagged glue falls back to en:

<en>Let's practice a natural Spanish greeting. First, how are you:</en>
<es>Hola, ¿cómo estás hoy? Espero que todo vaya muy bien.</es>
<en>And a warm thank you:</en>
<es>Muchas gracias por tu ayuda. Te lo agradezco de verdad.</es>

Map used by the example: en→english, es→spanish, fr→french, de→german, pt→portuguese (plus zh / ja / ko). Keep target-language spans about one full sentence so ICL does not rush the clone.

Phrases ≠ voice packs

Piece

Role

In tutor_tts

Tagged script

What the model says

Sources/Phrases.swift

Voice pack

Who it sounds like (clone ref_text / codes / embedding)

VoicePacks/tutor_<lang>/voice.json

Voice-pack ref_text is only a short clone reference — not the café / greeting demo lines.

Pipeline

tagged script
     │
     ▼
LangTagParser  →  [ (tag=en, text=…), (tag=es, text=…), … ]
     │
     ▼
for each span:
     set_voice(voice_dir: VoicePacks/tutor_<tag>, language: …)
     infer_stream(span.text)  →  AudioStreamPlayer.enqueue(pcm)
     │
     ▼
player.drain()

Component-by-component build

The sections below mirror the files under examples/tutor_tts/Sources.

Tagged script (phrase book)

Stock drills live in Phrases.swift. The UI can also edit the tagged string before play. Example span (Spanish greetings):

<en>Let's practice a natural Spanish greeting. First, how are you:</en>
<es>Hola, ¿cómo estás hoy? Espero que todo vaya muy bien.</es>
<en>And a warm thank you:</en>
<es>Muchas gracias por tu ayuda. Te lo agradezco de verdad.</es>

Language tag parser

LangParser.swift turns the script into ordered LangSegment values (short tag, Qwen language name, plain text). Pair-matched tags; untagged glue uses the default (en):

import Foundation

struct LangSegment {
    let tag: String           // "en", "es", …
    let qwen_language: String // "english", "spanish", …
    let text: String          // plain speakable text
}

let segments = try LangTagParser.parse(script)

Voice packs on disk

Bundle one prepared pack per language next to the app:

VoicePacks/
  tutor_en/voice.json
  tutor_es/voice.json
  tutor_fr/voice.json
  tutor_de/voice.json
  tutor_pt/voice.json

Ready-made packs: TheStageAI/Qwen3-TTS-Tutor-VoicePacks.

Make your own with the public encode helpers in examples/tools/prepare_voice_packs (PyPI + Hugging Face only — not TheStage Models / Qlip.SDK/scripts):

cd examples/tools/prepare_voice_packs
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements-qwen3.txt   # Qwen3 packs

# One pack per language for the tutor loop
python prepare_qwen3_voice_pack.py \
  --ref-audio ./tutor_en.wav \
  --ref-text "We still have time to practice clear speech." \
  --language english \
  --name tutor_en \
  --out-dir ./VoicePacks/tutor_en

(NeuTTS packs use requirements-neutts.txt + prepare_neutts_voice_pack.py — same folder. Full flags: TTS (Text-to-Speech) → How do I produce a voice_dir?).

Reference clip recommendations (this is the clone identity — not the tagged demo script):

Knob

Guidance

Length (Qwen3)

~2–4 s of continuous speech. Shorter under-constrains the clone; much longer adds little and can blur the speaker.

Length (NeuTTS)

~3–10 s at 16 kHz mono (NeuCodec native).

Content

One clear sentence in the target language of that pack. --ref-text must match the WAV verbatim.

Quality

Clean close-mic, little reverb/noise, no music, no overlapping speakers. Prefer consistent loudness (avoid clipped peaks).

Consistency across langs

Same speaker / same mic setup for tutor_en, tutor_es, … so hot-swaps keep one “tutor family”.

What not to use

The long café / greeting demo lines. Pack ref_text is only a short identity prompt for the codec.

Clone reference samples (pack identity audio — not the demo script):

English pack ref
French pack ref

Load Qwen3-TTS on NPU

Seed with any pack; every span calls set_voice again:

import TheStageSDK

try await TheStageAI.shared.initialize(apiToken: token)

let en_pack = Bundle.main.resourceURL!
    .appendingPathComponent("VoicePacks/tutor_en").path

let tts = try await Qwen3TTSPipeline(
    engines_path: "TheStageAI/Qwen3-TTS-12Hz-0.6B-Base",
    voice_id: "b_ref",
    voice_dir: en_pack,
    language: "english",
    device: "npu"   // Neural Engine — power-efficient on device
)

Stream loop with per-span set_voice

This is the core of dynamic language switching. infer_stream already defaults splitter to NLSentenceSplitter() — you only pass one if you need a custom split policy:

let player = AudioStreamPlayer(sample_rate: Double(tts.sample_rate))
player.start()

let gap = [Float](
    repeating: 0,
    count: Int(tts.sample_rate) * 250 / 1000  // 250 ms between spans
)

for (i, seg) in segments.enumerated() {
    let pack = voice_pack_path(tag: seg.tag)  // VoicePacks/tutor_<tag>
    try tts.set_voice(
        voice_dir: pack,
        voice_id: "b_ref",
        language: seg.qwen_language
    )

    var cfg = TTSGenerationConfig()
    cfg.seed = 42

    let stream = tts.infer_stream(
        text: seg.text,
        config: cfg,
        stream_config: Qwen3TTSPipeline.recommended_stream_config
    )
    for await chunk in stream {
        guard let pcm = chunk.audio, !pcm.isEmpty else { continue }
        player.enqueue(pcm)
    }

    if i + 1 < segments.count {
        player.enqueue(gap)
    }
}

await player.drain()
player.stop()

Why this works

  • set_voice does not reload Core ML / MLX engines

  • Each span gets the right clone pack and Qwen language=

  • AudioStreamPlayer keeps playback gapless across spans

Run the example app

From an AppleSDK checkout (tag 1.3.0+):

cd examples/tutor_tts
cp Secrets.xcconfig.example Secrets.xcconfig
# set TS_API_TOKEN=… from https://app.thestage.ai

xcodegen generate
open TutorTTS.xcodeproj
# Xcode → Team → Run (Release) on a physical iPhone

Or ./build_and_run.sh. Simulator is not supported.

In the app: Load model once → pick / edit a tagged phrase → Play stream.

Results

When it works you should hear:

  • Clear English coaching lines and target-language sentences in one continuous stream

  • Stable clone character across languages (same tutor “family”, pack per tag)

  • Short pauses between spans (~250 ms in the example)

  • NPU-backed synthesis without a cloud TTS hop for language changes

Further reading