VLM (Vision-Language Model)¶

On-device vision. Give the model a photo and a question, and it answers in text — a caption, an OCR transcription, a comparison of two frames, or fields extracted as JSON. The image never leaves the device.

Use it like the LLM: one call, one answer, or a streamed reply. The only new input is the image.

Main features

One pack, four jobs: captions, visual Q&A, OCR / document text, and structured field extraction from a single TSVLM.
Several images at once: compare, spot the difference, or add up receipts across frames.
Streaming replies: the same token deltas as the LLM page, so a caption fills in as it is written.
Deterministic OCR: temperature = 0 gives repeatable text.
Same knobs as LLM: LLMGenerationConfig carries over; nothing new to learn.
Nothing to preprocess: pass a CGImage or a file path. No resizing, no normalisation.

In this page ¶

Here we will cover the following topics:

Supported models: LFM2.5-VL-450M and what it does well.
Quick start: caption a photo — Swift and Flutter side by side.
Ask about an image: the call, how to pass images, and the generation knobs.
Result object: LLMResult and the latency breakdown.
Usage Guides: receipts to JSON, accessibility captions, before/after comparison, a slow first token, VLM inside a voice app.
Troubleshooting: symptom → cause → fix.
Load Progress / Prefetch / Cleanup: first-run download, warming the cache, releasing models.

Supported models ¶

Model	HF repo	Size	Device	Fleet pin
LFM2.5-VL-450M	`TheStageAI/LFM2.5-VL-450M`	450M	NPU	v1.2

Feature	LFM2.5-VL-450M
Captions / visual Q&A	yes
OCR / document text	yes
Structured field extraction	yes
Multiple images per call	yes, 2–4 recommended
Streaming	yes, Swift and Flutter

Image tokens and text share one context window. Dense pages and long prompts compete for it — crop to the region you care about.

Quick start ¶

Swift

import TheStageSDK

try await TheStageAI.shared.initialize(api_token: "your-api-token")

let vlm = try await TSVLM(
    engines_path: "TheStageAI/LFM2.5-VL-450M"
)

let image = try ImageIOHelpers.load_cgimage(path: "/path/to/photo.jpg")

var config = vlm.generation_defaults
config.max_new_tokens = 96
// repeatable captions
config.temperature = 0

let result = try vlm.infer(
    images: [image],
    prompt: "Describe this image in one sentence.",
    config: config
)
print(result.text)

Flutter

import 'package:thestage_apple_sdk/thestage_apple_sdk.dart';

await TheStageFlutterSDK.initialize(api_token: 'your-api-token');
await TheStageFlutterSDK.start_model(
  model_name: 'vlm',
  engines_path: 'TheStageAI/LFM2.5-VL-450M',
);

final rows = await TheStageFlutterSDK.infer(
  model_name: 'vlm',
  input_json: {
    // or 'image_base64'
    'image': '/absolute/path/to/photo.jpg',
    'prompt': 'Describe this image in one sentence.',
    'max_new_tokens': 96,
    // repeatable captions
    'temperature': 0,
  },
);
print(rows[0]['text']);

Important API ¶

Purpose	Swift	Flutter
Load a model	`try await TSVLM(engines_path:)`	`start_model(model_name: 'vlm', engines_path:)`
Load an image	`ImageIOHelpers.load_cgimage(path:)` / `(data:)`	`'image': path` or `'image_base64': …`
One answer	`try vlm.infer(images:prompt:config:)` → `LLMResult`	`infer(model_name: 'vlm', input_json:)` → `rows[0]`
Streamed answer	`try vlm.infer_stream(images:prompt:config:)` → `LLMStreamChunk`	`infer_stream(model_name: 'vlm', input_json:)` → `chunk['delta']`
Release	drop the object	`stop_model(model_name: 'vlm')`

Ask about an image ¶

How an image becomes text: vision encoder then decoder

One or more images plus a prompt. The prompt decides the job: “describe” gives a caption, “how many” gives an answer, “transcribe exactly” gives OCR, “extract as JSON” gives fields. Streaming works the same as the LLM — the first token just arrives after the image is encoded.

Swift

let vlm = try await TSVLM(engines_path: "TheStageAI/LFM2.5-VL-450M")

// From disk, from bytes, or straight from the camera
let a = try ImageIOHelpers.load_cgimage(path: "/tmp/before.jpg")
// Data from a download
let b = try ImageIOHelpers.load_cgimage(data: jpegData)
// UIImage from the camera
let c: CGImage = capturedImage.cgImage!

var config = vlm.generation_defaults
config.max_new_tokens = 192
config.temperature = 0

// Batch, two images
let result = try vlm.infer(
    images: [a, b],
    prompt: "The first image is BEFORE, the second is AFTER. List the differences.",
    config: config
)

// Streaming, one image
for await chunk in try vlm.infer_stream(
    images: [c], prompt: "Describe this photo.", config: config
) {
    if !chunk.is_final { caption.text += chunk.text }
}

Flutter

await TheStageFlutterSDK.start_model(
  model_name: 'vlm',
  engines_path: 'TheStageAI/LFM2.5-VL-450M',
);

// Batch, two images: numbered keys, contiguous from 0
final rows = await TheStageFlutterSDK.infer(
  model_name: 'vlm',
  input_json: {
    'image_0': '/tmp/before.jpg',
    // or 'image_base64_1'
    'image_1': '/tmp/after.jpg',
    'prompt': 'The first image is BEFORE, the second is AFTER. List the differences.',
    'max_new_tokens': 192,
    'temperature': 0,
  },
);

// Streaming, one image
await for (final chunk in TheStageFlutterSDK.infer_stream(
  model_name: 'vlm',
  input_json: {
    'image_base64': base64Encode(await File('/path/to/photo.jpg').readAsBytes()),
    'prompt': 'Describe this photo.',
  },
)) {
  if (chunk['is_final'] == true) break;
  caption.value += chunk['delta'] as String? ?? '';
}

Passing images

Field	Where	Accepts
`images`	Swift	`[CGImage]` — any count; 2–4 is the practical limit.
`image` / `image_N`	Flutter	Absolute file path (JPEG, PNG, BMP). `image_0`, `image_1`, … for several.
`image_base64` / `image_base64_N`	Flutter	Base64-encoded image bytes, when you have no file.
`prompt`	both	Required.
`system_prompt`	both	Optional.

Formats: anything ImageIO decodes — JPEG, PNG, HEIC. Do not resize or normalise; the encoder does that. Do rotate EXIF-sideways camera frames upright first.

Generation options — the LLM’s LLMGenerationConfig, with the defaults that matter here:

Field	Default	Change it when
`temperature`	pack	``0`` for OCR and extraction — same image, same text every time. Leave the pack default for captions if you want variety.
`max_new_tokens`	pack	Dense OCR or long JSON stops early (`stop_reason == "max_new_tokens"`) — raise to `512`.
`system_prompt`	none	Fix the output format once (“Reply with JSON only”) instead of repeating it in every prompt.

Result object ¶

The same LLMResult as the LLM page, plus a vision-encode timing. The first token waits for the encoder, so a spinner through encode_seconds is honest UI.

Swift

let vlm = try await TSVLM(engines_path: "TheStageAI/LFM2.5-VL-450M")
var config = vlm.generation_defaults
config.temperature = 0

let result = try vlm.infer(images: [image], prompt: prompt, config: config)
// caption / answer / OCR text
result.text
// "eos" or "max_new_tokens"
result.stop_reason
// vision encoder — most of the first-token wait
result.encode_seconds
result.tokens_per_second

Flutter

final r = rows[0];
r['text'];
r['stop_reason'];
r['encode_seconds'];
r['tokens_per_second'];

Field	Meaning
`text`	The answer.
`stop_reason`	`eos` finished; `max_new_tokens` cut off — raise the cap.
`encode_seconds`	Vision encoder time. Paid once per image, before the first token.
`prefill_seconds` / `decode_seconds`	Decoder phases.
`tokens_per_second` / `total_seconds`	Speed and wall time.

Troubleshooting ¶

Symptom	Cause	Fix
Empty caption	No image reached the model — bad path, empty base64, nil `CGImage`.	Log that the image loaded; on Flutter use an absolute container path or base64.
OCR misses small text	Whole page sent at once.	Crop to the text region; keep `temperature = 0`.
Answer cut off	`stop_reason == "max_new_tokens"`.	Raise the cap, or crop / shorten the prompt — vision tokens share the window.
Sideways or mirrored reading	Camera frame not EXIF-upright.	Rotate before encoding.
Slow first token	Vision encode, cold.	Warm at launch; stream; show a spinner until the first delta.
Flutter: `image_1` ignored	Keys not contiguous from `image_0`.	Number from 0.
Out of memory next to the agent	VLM resident with ASR + LLM + TTS.	Load for the task, release after.

Load Progress / Prefetch / Cleanup ¶

First run downloads and prepares the pack; later runs hit the cache. Show progress the first time, warm the cache on a splash screen, and release models you are done with.

Swift

let ai = TheStageAI.shared

// Progress
let vlm = try await TSVLM(
    engines_path: "TheStageAI/LFM2.5-VL-450M",
    on_load_progress: { p in
        print("[\(p.model)] \(p.phase) \(Int(p.fraction * 100))%")
    }
)

// Prefetch on a splash screen, construct later
let engines_dir = try await ai.prefetch_engines(
    repo_id: "TheStageAI/LFM2.5-VL-450M"
)
let vlm = try await TSVLM(engines_path: engines_dir)

// Cleanup: drop the reference, or
_ = try ai.stop_model(model_name: "vlm")

Flutter

// Progress
TheStageFlutterSDK.on_progress.listen((event) {
  if (event['model_name'] != 'vlm') return;
  print('[vlm] ${event['phase']} ${((event['progress'] ?? 0) * 100).round()}%');
});

await TheStageFlutterSDK.start_model(
  model_name: 'vlm',
  engines_path: 'TheStageAI/LFM2.5-VL-450M',
);

// Cleanup
await TheStageFlutterSDK.stop_model(model_name: 'vlm');

Phases: downloading → extracting → loading → ready. Cache hits skip the first two. Full contract: TheStage Apple SDK (Load Progress).

VLM (Vision-Language Model)

VLM (Vision-Language Model)¶

In this page ¶

Supported models ¶

Quick start ¶

Important API ¶

Ask about an image ¶

Result object ¶

Usage Guides ¶

Turn a photographed receipt into fields ¶

Describe photos for accessibility ¶

Compare two frames ¶

The first token is slow ¶

Add vision to a voice assistant ¶

Troubleshooting ¶

Load Progress / Prefetch / Cleanup ¶