Streaming¶

Real-time streaming inference for TTS and ASR pipelines (and, coming soon, the on-device LLM).

Overview ¶

TTS streaming yields audio chunks as sentences are synthesized — much lower time-to-first-audio than batch mode. Streaming ASR is the inverse direction: push microphone audio in, read transcripts out. LLM streaming (decoded token text chunk by chunk) ships with the on-device chat LLM — coming soon; a remote LLM response works as a push source today.

Streaming and batch inference use the same model bundle — no extra setup. Just use infer_stream instead of infer, or pull a push-based TTSStreamer via TheStageAI.open_tts_streamer(...) (see Engine Requirements below).

Usage Guides ¶

TTS Streaming ¶

Kotlin — Simple Consumer

// build.gradle.kts: implementation(files("libs/TheStageCore.aar"))
import ai.thestage.qlip.TheStageAI

// Inside a coroutine.
TheStageAI.registerContext(context)
TheStageAI.initialize(api_token = "your_api_token")

TheStageAI.start_model(
    model_name = "tts",
    engines_path = "TheStageAI/neutts-multilingual",
    config = mapOf("voice_id" to "dave"),
    revision = "android"
)

TheStageAI.infer_stream(
    model_name = "tts",
    input_json = mapOf("text" to "A long paragraph of text to speak aloud.")
).collect { chunk ->
    val audio = chunk.audio ?: return@collect
    play_audio(audio, chunk.sample_rate ?: 24000)

    if (chunk.is_final) {
        println("Decode tokens: ${chunk.generated_tokens ?: 0}")
        println("Tok/s: ${chunk.tokens_per_second ?: 0.0}")
        println("First chunk: ${chunk.time_to_first_token ?: 0.0}s")
        println("Total wall-clock: ${chunk.total_seconds ?: 0.0}s")
    }
}

Kotlin — Producer / Consumer with AudioStreamPlayer

Two concurrent coroutines: one drives TTS inference, the other plays audio as chunks arrive. This is the recommended pattern for low-latency playback.

import ai.thestage.qlip.audio.AudioStreamPlayer
import ai.thestage.qlip.audio.AudioStreamConfig

TheStageAI.start_model(
    model_name = "tts",
    engines_path = "TheStageAI/neutts-multilingual",
    config = mapOf("voice_id" to "paul")
)

val player = AudioStreamPlayer(
    AudioStreamConfig(
        sample_rate = 24000,
        channels = 1,
        queue_capacity = 256
    )
)
player.start()

TheStageAI.infer_stream(
    model_name = "tts",
    input_json = mapOf("text" to "Hello! This is a streaming demo with real-time audio.")
).collect { chunk ->
    val audio = chunk.audio
    if (audio != null && audio.isNotEmpty()) player.enqueue(audio)
}

player.drain()
player.stop()

See AudioStreamConfig Options in the API Reference below for the player’s configuration knobs.

Kotlin — Push-Based (LLM → TTS)

Feed text incrementally as it arrives from another model. The streamer splits sentences internally and produces audio as complete sentences are ready. Use TheStageAI.open_tts_streamer(model_name) to pull a fresh streamer per turn — there is no need (or way) to construct TTSStreamer directly from outside the SDK module.

TheStageAI.start_model(
    model_name = "tts",
    engines_path = "TheStageAI/neutts-multilingual",
    config = mapOf("voice_id" to "dave")
)

val player = AudioStreamPlayer(sample_rate = 24000)
player.start()

val streamer = TheStageAI.open_tts_streamer(model_name = "tts")

// Audio consumer coroutine
val consumer = launch {
    streamer.output.collect { chunk ->
        val pcm = chunk.audio
        if (pcm != null && pcm.isNotEmpty()) player.enqueue(pcm)
    }
    player.drain()
    player.stop()
}

// Producer: push text deltas as they arrive from your LLM —
// e.g. a remote chat-completions stream (this is what the voice
// agent's openai_compatible provider does).
llmDeltas.collect { delta ->
    if (delta.isNotEmpty()) streamer.send(delta)
}
streamer.finish()   // flush tail + close output cleanly

streamer.finish() flushes any partial sentence buffered inside the splitter before closing the output flow. Use streamer.cancel() instead if you want to abort an in-flight turn (e.g. on barge-in) — that drops the buffer and closes immediately.

This is exactly what TheStageVoiceAgent does internally: its TTS node opens one push-based TTSStreamer per turn, pumps LLM deltas straight in, and cancel()s on barge-in.

Flutter — TTS Streaming ¶

Simple Usage

final stream = TheStageFlutterSDK.infer_stream(
  model_name: 'tts',
  input_json: {'text': 'A long paragraph of text to speak aloud.'},
);

final player = TheStageAudioPlayer(sampleRate: 24000);
await player.start();

await for (final chunk in stream) {
  final audio = chunk['audio'] as Float32List?;
  if (audio != null && audio.isNotEmpty) {
    player.enqueue(audio);
  }
  if (chunk['is_final'] == true) break;
}

await player.drain();
await player.stop();

Push-Based (LLM → TTS)

final ttsStream = TheStageFlutterSDK.infer_stream(
  model_name: 'tts',
  input_json: {'text': ''},
  stream_id: 'voice_agent_tts',
);

final player = TheStageAudioPlayer(sampleRate: 24000);
await player.start();

ttsStream.listen((chunk) {
  final audio = chunk['audio'] as Float32List?;
  if (audio != null) player.enqueue(audio);
});

// Push text deltas as they arrive from your LLM — e.g. a remote
// chat-completions stream.
await for (final delta in llmDeltas) {
  await TheStageFlutterSDK.send(
    stream_id: 'voice_agent_tts',
    text: delta,
  );
}
await TheStageFlutterSDK.finish_stream(stream_id: 'voice_agent_tts');

await player.drain();
await player.stop();

Streaming ASR (Speech-to-Text)¶

The inverse direction — push microphone audio in, read transcripts out — is available on Android through the Voice Agent (Voice Agent), which runs streaming ASR internally. Its ASR node re-decodes the growing turn buffer on a single serial worker and commits stable text via LocalAgreement, so partials grow monotonically and never flicker, while the authoritative end-of-turn transcript always covers the complete utterance (including the last word).

For a standalone, reusable streamer outside the voice agent, open one on a loaded Whisper model from Kotlin (mirrors open_tts_streamer):

val asr = TheStageAI.open_asr_streamer(
    model_name = "stt", language = "en",
)
scope.launch { asr.partials.collect { caption -> /* live text */ } }
for (chunk in micStream) asr.send(chunk)   // 16 kHz mono FloatArray
val finalText = asr.finish()               // authoritative transcript

It re-decodes the growing buffer on a single serial worker and commits stable text via the same LocalAgreement-2 as the voice agent, so partials grow monotonically. This is a native-Kotlin API — there is no plugin/Dart channel route for it; from Flutter, drive TheStageVoiceAgent for live speech-to-text or use the batch infer for one-shot transcription (see ASR (Whisper)). Convert mic input to 16 kHz mono Float first — it is not resampled.

Tuning the TTS Streamer ¶

NeuTTSStreamConfig exposes the codec-side chunking knobs that decide time-to-first-audio and how seams between sentences sound. Defaults match what the SDK ships with — only override these when you need to trade latency against smoothness.

Kotlin:

import ai.thestage.qlip.models.neutts.NeuTTSStreamConfig

val streamer = tts.open_streamer(
    stream_config = NeuTTSStreamConfig(
        frames_per_chunk = 25,
        first_frames_per_chunk = 12,   // smaller first chunk → faster first audio
        lookforward = 5,
        lookback = 50,
        overlap_frames = 1
    )
)

// Or via the singleton:
val s2 = TheStageAI.open_tts_streamer(
    model_name = "tts",
    stream_config = NeuTTSStreamConfig(first_frames_per_chunk = 12)
)

infer_stream(text, stream_config) accepts the same struct when you already have the full text up front.

Flutter — pass a nested stream_config map inside input_json:

final stream = TheStageFlutterSDK.infer_stream(
  model_name: 'tts',
  input_json: {
    'text': 'Hello, world.',
    'stream_config': {
      'first_frames_per_chunk': 12,
      'frames_per_chunk': 25,
      'lookforward': 5,
      'lookback': 50,
      'overlap_frames': 1,
    },
  },
);

Cheat-sheet — the knobs above at a glance:

Knob	Effect	When to change
`first_frames_per_chunk`	Codec frames in the first emitted chunk	Shrink it when time-to-first-audio matters more than a slightly shorter opening chunk
`frames_per_chunk`	Frames per emitted chunk after the first — larger means fewer, longer chunks	Raise for smoother, less chatty output; lower for finer-grained delivery
`lookforward`	Future frames decoded with each chunk to stabilise the seam	Raise if chunk seams sound unstable
`lookback`	Past frames re-decoded for context when bridging chunks	Raise if chunk boundaries are audible
`overlap_frames`	Frames of crossfade between consecutive chunks	Raise if you hear clicks at chunk boundaries

See TTS (NeuTTS) (Streaming Hyperparameters) for the full field reference.

API Reference ¶

AudioStreamConfig Options ¶

Parameter	Type	Default	Description
`sample_rate`	`Int`	24000	Audio sample rate in Hz
`channels`	`Int`	1	Number of audio channels
`queue_capacity`	`Int`	256	Bounded PCM queue depth feeding the writer thread
`voice_processing`	`Boolean`	false	Route playback as voice communication so the platform `AcousticEchoCanceler` references it (required for AEC); media routing otherwise

The player is backed by an Android AudioTrack in MODE_STREAM; enqueue is non-blocking and drain() (suspend) waits until everything queued and buffered has actually played out.

Flutter API Reference ¶

Lifecycle

await TheStageFlutterSDK.initialize(api_token: 'your_token');

await TheStageFlutterSDK.start_model(
  model_name: 'tts',
  engines_path: 'TheStageAI/neutts-multilingual',
  model_type: 'neutts-multilingual',
  revision: 'android',
  config: {'voice_id': 'dave'},
);

await TheStageFlutterSDK.stop_model(model_name: 'tts');

Streaming

final stream = TheStageFlutterSDK.infer_stream(
  model_name: 'tts',
  input_json: {'text': 'Hello world.'},
);

await TheStageFlutterSDK.send(stream_id: id, text: 'more text');
await TheStageFlutterSDK.finish_stream(stream_id: id);
await TheStageFlutterSDK.stop_stream(stream_id: id);

Audio Player

final player = TheStageAudioPlayer(sampleRate: 24000);
await player.start();
player.enqueue(audioData);
await player.pause();
await player.resume();
await player.drain();
await player.stop();

Stream Chunk Format ¶

For the SDK-wide audio format contract (sample rates, mono Float, frame sizes for VAD vs ASR vs TTS) see TheStage Android SDK (Audio I/O Contract).

TTS Chunks

Field	Type	Description
`audio`	`FloatArray` / `Float32List`	PCM audio samples, 24 kHz mono, normalized to `[-1.0, 1.0]`
`sample_rate`	`Int`	Always 24000
`index`	`Int`	Sequential chunk number
`is_final`	`Boolean`	`true` on the sentinel (empty) last chunk
`time_to_first_token`	`Double?`	Seconds to first audio chunk
`generated_tokens`	`Int?`	Decode step count (excludes prefill)
`tokens_per_second`	`Double?`	Decode speed: `steps / sum_of_step_durations` (measured inside decoder)
`total_seconds`	`Double?`	Wall-clock time from stream start to last chunk (final only)

LLM Chunks (on-device chat LLM — coming soon)

Field	Type	Description
`delta` (Kotlin & Flutter)	`String?`	Decoded token text (null on the final sentinel)
`index`	`Int`	Position in sequence
`is_final`	`Boolean`	`true` for the sentinel chunk
`time_to_first_token`	`Double?`	Seconds to first token (final only)
`prompt_tokens`	`Int?`	Input token count (final only)
`generated_tokens`	`Int?`	Output token count (final only)
`tokens_per_second`	`Double?`	Generation speed (final only)
`total_seconds`	`Double?`	Wall-clock time (final only)

Cancellation ¶

Cancel any active stream at any time:

// Kotlin — cancel the collecting coroutine, or call streamer.cancel()

// Flutter
await TheStageFlutterSDK.stop_stream(stream_id: 'my_stream');

The stream will emit a final event with kind: 'cancelled' and close.

Engine Requirements ¶

Streaming and batch inference use the same model bundle — no extra setup. Just use infer_stream instead of infer, or pull a push-based TTSStreamer via TheStageAI.open_tts_streamer(...).

Architecture ¶

TTSStreamer — Single Token Stream with Sentinels:

Producer Task                          Consumer Task
─────────────                          ─────────────
sentence_stream                        token_stream
    │                                      │
    ▼                                      ▼
┌──────────┐                         ┌───────────┐
│preprocess│                         │is sentinel?│
└────┬─────┘                         └─┬───────┬─┘
     │                                 no      yes
     ▼                                 │        │
┌──────────────┐                       ▼        ▼
│decoder       │                  accumulate   flush
│  .prefill()  │                  codes        + fade-out
│  .decode_step│                       │        + reset
│  (loop)      │                       ▼
└────┬─────────┘                  ┌─────────┐
     │                            │codec.infer│
     ▼                            │           │
yield tokens                      └────┬────┘
     │                                 │
     ▼                                 ▼
yield sentinel                    OLA + emit

The producer runs ahead — while the consumer decodes audio for the current sentence, the producer is already preprocessing and generating tokens for the next one. This eliminates inter-sentence pauses.

Troubleshooting ¶

Symptom	Cause / Fix
Push-based audio stalls / stays buffered after `send`	Nothing is draining the streamer. Run two concurrent coroutines — start the consumer collecting `streamer.output` (feeding `AudioStreamPlayer`) before or while the producer pushes text, as in the Push-Based (LLM → TTS) example.
Tail of the pushed text is never spoken	The producer never signalled end of input. Call `streamer.finish()` (Flutter: `finish_stream`) — it flushes the partial sentence buffered inside the splitter before closing the output flow.
Stream closes immediately, dropping queued speech	`streamer.cancel()` / `stop_stream` aborts the in-flight turn: it drops the buffer and closes immediately, emitting a final `kind: 'cancelled'` event. Use it for barge-in; use `finish()` / `finish_stream` for a clean end of turn.
Clicks or rough seams between audio chunks	Codec-side chunking trades latency against smoothness — raise `overlap_frames` (crossfade) or `lookback` in `NeuTTSStreamConfig`. See Tuning the TTS Streamer above.
First `infer_stream` run is much slower than later ones	First run downloads and extracts the model bundle; later runs hit the cache. Track the `downloading` / `extracting` phases via TheStage Android SDK (Load Progress).

Streaming