VLM (Vision-Language)¶

On-device image + text → text. TheStageVLM (LFM2.5-VL) runs the vision encoder and the language decoder entirely on-device — hand it an image and a prompt, get back a text answer. Batch and token-by-token streaming share the same response shape as the LLM path.

Supported models ¶

Model	HF Repo	Notes
LFM2.5-VL	`TheStageAI/LFM2.5-VL-450M`	`model_name` `lfm2-vl`; vision encoder + language decoder entirely on-device

API surface ¶

Purpose	Kotlin (singleton)	Flutter
Start	`TheStageAI.start_model(model_name = "lfm2-vl", engines_path = …)`	`TheStageFlutterSDK.start_model(model_name: 'lfm2-vl', …)`
One-shot	`TheStageAI.infer(model_name, input_json)` → `result[0]["text"]`	`TheStageFlutterSDK.infer(…)` → `result[0]['text']`
Streaming	see Streaming	`TheStageFlutterSDK.infer_stream(…)` → `text` deltas, final chunk has `is_final == true`

Quick start ¶

Kotlin — via the singleton:

import ai.thestage.qlip.TheStageAI

TheStageAI.registerContext(context)
TheStageAI.initialize(api_token = "your-api-token")

TheStageAI.start_model(
    model_name = "lfm2-vl",
    engines_path = "TheStageAI/LFM2.5-VL-450M",
)

val result = TheStageAI.infer(
    model_name = "lfm2-vl",
    input_json = mapOf(
        "image" to "/path/to/photo.jpg",
        "prompt" to "What is in this image?",
        "max_new_tokens" to 128,
    ),
)
println(result[0]["text"])

Flutter — JSON path:

await TheStageFlutterSDK.start_model(
  model_name: 'lfm2-vl',
  engines_path: 'TheStageAI/LFM2.5-VL-450M',
);

final result = await TheStageFlutterSDK.infer(
  model_name: 'lfm2-vl',
  input_json: {
    'image': '/path/to/photo.jpg',
    'prompt': 'What is in this image?',
    'max_new_tokens': 128,
  },
);
print(result[0]['text']);

Configuration ¶

Optional per-call generation knobs:

Key	Type	Default	Description
`preset`	`String`	`"medium"`	Image tiling: `"medium"` (single tile) or `"high"` (grid + thumbnail, more detail at higher cost).
`system_prompt`	`String?`	—	Optional ChatML system message.
`max_new_tokens`	`Int`	512	Cap on generated tokens.
`seed`, `temperature`, `top_k`, `min_p`, `repetition_penalty`	—	—	Per-call sampling overrides — omit to keep the bundle’s preset.

Inputs and outputs ¶

Key	Type	Default	Description
`image`	`String`	required	File path to the input image.
`prompt`	`String`	required	Question / instruction about the image.
output `text`	`String`	—	The generated answer.

The result also carries prompt_tokens / image_tokens / token counts and timing (prompt_ms, …) for diagnostics.

Streaming ¶

infer_stream streams text deltas exactly like the LLM path — each chunk before the terminal one carries a text delta; the final chunk has is_final == true. For the VLM the streaming delta key is text.

Kotlin — via the singleton:

TheStageAI.infer_stream(
    model_name = "lfm2-vl",
    input_json = mapOf(
        "image" to "/path/to/photo.jpg",
        "prompt" to "Describe this scene in detail.",
        "max_new_tokens" to 256,
    ),
).collect { chunk ->
    val delta = chunk.text
    if (delta != null) print(delta)
    if (chunk.is_final) return@collect
}

Flutter — JSON path:

final stream = TheStageFlutterSDK.infer_stream(
  model_name: 'lfm2-vl',
  input_json: {
    'image': '/path/to/photo.jpg',
    'prompt': 'Describe this scene in detail.',
    'max_new_tokens': 256,
  },
);

await for (final chunk in stream) {
  if (chunk['is_final'] == true) break;
  stdout.write(chunk['text']);
}

See Streaming for the shared chunk contract and Model Management for loading the VLM’s vision encoder + decoder components independently.

Lifecycle ¶

TheStageAI.registerContext(context) then TheStageAI.initialize(api_token = ...) — must have succeeded before start_model.
start_model(model_name = "lfm2-vl", engines_path = "TheStageAI/LFM2.5-VL-450M") — first call downloads and caches the bundle.
infer for one-shot answers; infer_stream for token-by-token text deltas (see Streaming).
stop_model(model_name = "lfm2-vl") when done.

Troubleshooting ¶

Symptom	Cause / Fix
First infer very slow	HF download on first `start_model` — wait for `ready`; later runs use the cache.
Model never sees the image	`image` is a file path (`String`) — pass a path to a readable image file on device storage.
Streamed answer looks truncated	Consume the stream until the chunk with `is_final == true` — every chunk before it carries a `text` delta.

Load Progress / Prefetch / Cleanup ¶

Load progress ¶

Download / extract / load progress is reported through the singleton and Flutter entry points via an optional on_load_progress handler that fires through four phases with a monotonic fraction in 0.0 .. 1.0:

TheStageAI.start_model(
    model_name = "lfm2-vl",
    engines_path = "TheStageAI/LFM2.5-VL-450M",
    on_load_progress = { p ->
        // p.phase ∈ { DOWNLOADING, EXTRACTING, LOADING, READY }
        println("[${p.model}] ${p.phase} ${(p.fraction * 100).toInt()}%")
    }
)

The same on_load_progress parameter is accepted by TheStageAI.prefetch_model(...). For the full event contract see TheStage Android SDK (Load Progress).

Flutter:

TheStageFlutterSDK.on_progress.listen((event) {
  if (event['model_name'] != 'lfm2-vl') return;
  final phase    = event['phase']    as String?;   // downloading | extracting | loading | ready
  final fraction = event['progress'] as double?;   // 0.0 ... 1.0, monotonic
  print('[lfm2-vl] $phase ${(fraction ?? 0) * 100}%');
});

await TheStageFlutterSDK.start_model(
  model_name: 'lfm2-vl',
  engines_path: 'TheStageAI/LFM2.5-VL-450M',
);

Prefetch ¶

val engines_dir = TheStageAI.prefetch_model(
    repo_id = "TheStageAI/LFM2.5-VL-450M"
)

// Warm cache — the next start_model loads without network.

Cleanup ¶

When you used the singleton API:

TheStageAI.stop_model(model_name = "lfm2-vl")

Flutter:

await TheStageFlutterSDK.stop_model(model_name: 'lfm2-vl');

VLM (Vision-Language)