ASR (Whisper)

On-device speech recognition using Whisper. The ASR pipeline handles mel-spectrogram, encoder and decoder, with automatic VAD chunking and long-audio stitching.

Flutter consumers go through the singleton start_model + infer JSON path — there is no direct pipeline constructor on Dart. Both surfaces share the same on-disk cache and response shape.

Supported models

Model	HF repo	Notes
Whisper large-v3-turbo	TheStageAI/thewhisper-large-v3-turbo	model_name = stt; any input length — the SDK windows long audio

API surface

Singleton (TheStageAI)

TheStageAI is a Kotlin object (singleton); its methods are suspend.

TheStageAI.start_model(
    model_name = "stt",                            // any handle you choose
    engines_path = "TheStageAI/thewhisper-large-v3-turbo"
)

val json = TheStageAI.infer(
    model_name = "stt",
    input_json = mapOf(
        "audio" to audio_samples,   // FloatArray, 16 kHz mono
        "language" to "en"          // optional, default "en"
    )
)
val text = json[0]["transcription"] as String

Flutter reaches this exact JSON path through TheStageFlutterSDK — see Quick start for the Dart calls and Result object for the response keys.

Direct constructor

val engine = WhisperModel(
    engines_path = dir,                  // local (prefetched) dir
    device = "npu",                      // "npu" | "gpu" | "cpu"
    overlap_seconds = 0.0                // chunk overlap for long audio
)
val stt = TheStageASRPipeline.wrap(engine)

Per-module device placement (mel on CPU, encoder / decoder on NPU) is set in the bundle’s metadata.json; the top-level device is a coarse default. When downloading through the singleton, pass per-module overrides via start_model(devices = mapOf("encoder" to "npu", ...)). TheStageAI.initialize(...) must have succeeded before either call.

Quick start

Kotlin — via the singleton (recommended; auto-downloads the bundle):

// build.gradle.kts: implementation(files("libs/TheStageCore.aar"))
import ai.thestage.qlip.TheStageAI

// Inside a coroutine (initialize / start_model / infer are suspend).
TheStageAI.registerContext(context)
TheStageAI.initialize(api_token = "your-api-token")

TheStageAI.start_model(
    model_name = "stt",
    engines_path = "TheStageAI/thewhisper-large-v3-turbo"  // HF repo or local
)

// Transcribe audio (16 kHz mono FloatArray, see Audio I/O Contract).
val json = TheStageAI.infer(
    model_name = "stt",
    input_json = mapOf("audio" to audio_samples, "language" to "en")
)
println(json[0]["transcription"])  // "Hello, how are you today?"

The direct Kotlin classes (WhisperModel + TheStageASRPipeline) are also available for a local (already-downloaded) engines directory. They load from a local dir — they do not download, so prefetch_model first:

import ai.thestage.qlip.models.whisper.WhisperModel
import ai.thestage.qlip.models.asr.TheStageASRPipeline

val dir = TheStageAI.prefetch_model(
    repo_id = "TheStageAI/thewhisper-large-v3-turbo"
)

val engine = WhisperModel(engines_path = dir, device = "npu")
val stt    = TheStageASRPipeline.wrap(engine)

val result = stt.infer(audio = audio_samples, language = "en")
println(result.text)               // result.tokens_per_second, ...

Flutter — JSON path:

import 'package:thestage_android_sdk/thestage_android_sdk.dart';
import 'dart:typed_data';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');

await TheStageFlutterSDK.start_model(
  model_name: 'stt',
  engines_path: 'TheStageAI/thewhisper-large-v3-turbo',
);

// audio_samples: Float32List, 16 kHz mono, samples in [-1.0, 1.0].
final result = await TheStageFlutterSDK.infer(
  model_name: 'stt',
  input_json: {
    'audio': audio_samples,
    'language': 'en',
  },
);
print(result[0]['transcription']);

Audio contract

• 16 kHz mono FloatArray, samples normalized to [-1.0, 1.0].

• Long buffers are split internally into the bundle’s chunk_seconds windows. The shipped bundle sets its transcription window (see the bundle’s encoder_spec.json); the SDK default is 30 s, and other window sizes (10 / 15 / 30 s exports) work the same way.

• Overlap between windows is configurable via the overlap_seconds constructor argument (default 0). Useful on streaming captures to avoid losing words straddling a chunk boundary.

• Mismatched-rate input is not auto-resampled — convert your mic capture to 16 kHz mono Float before calling infer.

See TheStage Android SDK (Audio I/O Contract) for the shared format used across VAD / ASR / TTS.

Configuration

Input	Type	Description
audio	FloatArray	16 kHz mono PCM, samples in [-1.0, 1.0], any length.
language	String (default "en")	Whisper language code — see Language codes below.
config.max_new_tokens	Int?	Cap per-window decode.
config.return_tokens	Boolean (default false)	Direct Kotlin API only — include token IDs in ASRResult.tokens. Not honored on the singleton / JSON path.

Language codes

Common values for the language input (Whisper’s standard codes):

Code	Language	Code	Language
en	English	ja	Japanese
fr	French	ko	Korean
de	German	zh	Chinese
es	Spanish	ar	Arabic
pt	Portuguese	hi	Hindi
ru	Russian

Internal VAD chunking

The Whisper pipeline includes a Silero-VAD pre-pass that finds speech segments before transcribing — this is the “automatic VAD chunking” referenced above. It is active whenever the bundle ships a vad sub-engine; a bundle without one falls back to fixed-length window chunking. The pre-pass is driven purely by bundle contents.

Streaming API

Live, push-based transcription — feed mic audio as it arrives and read stable, monotonically-growing partials — is available on Android through the Voice Agent (Voice Agent), which runs streaming ASR internally: it re-decodes the growing turn buffer on a single serial worker and commits stable text via LocalAgreement, so captions never flicker or retract, and the authoritative end-of-turn transcript always covers the complete utterance (including the last word). There is no standalone streaming-ASR entry point on the direct pipeline; use the batch infer above for one-shot transcription and the Voice Agent for real-time captioning.

Result object

Field	Type	Description
ASRResult.text	String	Transcribed text.
ASRResult.token_count	Int	Total decoded tokens (sum across windows).
ASRResult.decode_seconds	Double	Decoder wall time.
ASRResult.tokens	IntArray?	Token IDs (only if return_tokens == true).

Note

The fields above (token_count, decode_seconds, tokens) and return_tokens are the direct Kotlin ``TheStageASRPipeline`` / ``ASRResult`` surface. The singleton / JSON path returns a different key set — see below.

JSON response keys: transcription (String), mel_seconds (Double), encoder_seconds (Double), total_seconds (Double), tokens_per_second (Double), generated_tokens (Int).

The Flutter TheStageFlutterSDK.infer call hits this exact JSON path, so the response keys above apply unchanged on Dart. audio crosses the platform channel as Float32List; do not promote to Float64List.

Lifecycle

1. TheStageAI.registerContext(context) then TheStageAI.initialize(api_token = ...) — must have succeeded before start_model or the direct constructor.

2. start_model(model_name = "stt", engines_path = ...) — first call downloads and caches the bundle. For the direct classes, run prefetch_model first — they load from a local dir only.

3. infer for one-shot transcription; the Voice Agent for live captioning (see Streaming API).

4. stop_model(model_name = "stt") when done.

Usage Guides

Microphone capture with AudioRecord

The pipeline expects 16 kHz mono FloatArray in [-1.0, 1.0] and does not resample (see Audio contract), so capture at 16 kHz directly. Standard Android AudioRecord with ENCODING_PCM_FLOAT yields normalized floats as-is (requires the RECORD_AUDIO permission):

import android.media.AudioFormat
import android.media.AudioRecord
import android.media.MediaRecorder

val sample_rate = 16000
val min_buf = AudioRecord.getMinBufferSize(
    sample_rate,
    AudioFormat.CHANNEL_IN_MONO,
    AudioFormat.ENCODING_PCM_FLOAT
)
val recorder = AudioRecord(
    MediaRecorder.AudioSource.MIC,
    sample_rate,
    AudioFormat.CHANNEL_IN_MONO,
    AudioFormat.ENCODING_PCM_FLOAT,
    min_buf
)

val chunk = FloatArray(sample_rate)          // 1 s per read
val captured = ArrayList<Float>()

recorder.startRecording()
while (recording) {                          // your stop condition
    val n = recorder.read(
        chunk, 0, chunk.size, AudioRecord.READ_BLOCKING
    )
    for (i in 0 until n) captured.add(chunk[i])
}
recorder.stop()
recorder.release()

val audio_samples = captured.toFloatArray()  // 16 kHz mono, [-1, 1]

Then transcribe with the same infer call as in Quick start:

val json = TheStageAI.infer(
    model_name = "stt",
    input_json = mapOf("audio" to audio_samples, "language" to "en")
)
println(json[0]["transcription"])

Int16 PCM → Float conversion

If your source produces 16-bit PCM (ShortArray, e.g. an ENCODING_PCM_16BIT capture or a WAV file), divide by 32768 to normalize into [-1.0, 1.0]:

// pcm: ShortArray of 16-bit samples, already 16 kHz mono
val audio_samples = FloatArray(pcm.size) { i ->
    (pcm[i] / 32768.0f).coerceIn(-1.0f, 1.0f)
}

Troubleshooting

Symptom	Cause / Fix
Audio fed at a rate other than 16 kHz	Mismatched-rate input is not auto-resampled — convert your mic capture to 16 kHz mono Float before calling infer.
No token IDs in the response on the singleton / JSON path	return_tokens is honored only on the direct Kotlin API — ASRResult.tokens is not exposed through the JSON path.
First infer very slow	HF download on first start_model — wait for ready; later runs use the cache.
Flutter audio glitches / NaNs	Float64List or wrong rate — pass audio as Float32List, 16 kHz mono.

Load Progress / Prefetch / Cleanup

Load progress

Download / extract / load progress is reported through the singleton and Flutter entry points via an optional on_load_progress handler that fires through four phases with a monotonic fraction in 0.0 .. 1.0:

TheStageAI.start_model(
    model_name = "stt",
    engines_path = "TheStageAI/thewhisper-large-v3-turbo",
    on_load_progress = { p ->
        // p.phase ∈ { DOWNLOADING, EXTRACTING, LOADING, READY }
        println("[${p.model}] ${p.phase} ${(p.fraction * 100).toInt()}%")
    }
)

The same on_load_progress parameter is accepted by TheStageAI.prefetch_model(...). For the full event contract see TheStage Android SDK (Load Progress).

Flutter:

TheStageFlutterSDK.on_progress.listen((event) {
  if (event['model_name'] != 'stt') return;
  final phase    = event['phase']    as String?;   // downloading | extracting | loading | ready
  final fraction = event['progress'] as double?;   // 0.0 ... 1.0, monotonic
  print('[stt] $phase ${(fraction ?? 0) * 100}%');
});

await TheStageFlutterSDK.start_model(
  model_name: 'stt',
  engines_path: 'TheStageAI/thewhisper-large-v3-turbo',
);

Prefetch

val engines_dir = TheStageAI.prefetch_model(
    repo_id = "TheStageAI/thewhisper-large-v3-turbo"
)

// Later — instant load, no network:
val engine = WhisperModel(engines_path = engines_dir)

Cleanup

When you used the singleton API:

TheStageAI.stop_model(model_name = "stt")

Flutter:

await TheStageFlutterSDK.stop_model(model_name: 'stt');

• ASR (Whisper)
  • Supported models
  • API surface
    • Singleton (TheStageAI)
    • Direct constructor
  • Quick start
  • Audio contract
  • Configuration
    • Language codes
    • Internal VAD chunking
  • Streaming API
  • Result object
  • Lifecycle
  • Usage Guides
    • Microphone capture with AudioRecord
    • Int16 PCM → Float conversion
  • Troubleshooting
  • Load Progress / Prefetch / Cleanup
    • Load progress
    • Prefetch
    • Cleanup