Vision-language API

Install

The tier needs the vlm extra.

pip install 'libreyolo[vlm]'

The factory

LibreVLM(model: str = "qwen3-vl-4b", **kwargs) -> LibreVLMModel

model is an alias, not a path. **kwargs reaches the family constructor, which takes device, names (the initial vocabulary, equivalent to calling set_classes after load), prompt (override the detection prompt) and max_new_tokens. An unknown alias raises ValueError listing every alias.

from libreyolo import LibreVLM, SAMPLE_IMAGE model = LibreVLM("lfm2-vl-450m")model.set_classes(["person", "skateboard"]) result = model.predict(SAMPLE_IMAGE)for box, cls in zip(result.boxes.xyxy, result.boxes.cls):    print(result.names[int(cls)], box.tolist())

from libreyolo import LibreVLM, SAMPLE_IMAGE model = LibreVLM("lfm2-vl-450m")print(model.chat(SAMPLE_IMAGE, "How many people are in this image?"))

Aliases

The default alias is qwen3-vl-4b. Sizes for the default alias of each family are the ones listed first: qwen3-vl resolves to 4b, lfm2-vl to 450m, internvl3 to 2b, smolvlm2 to 2.2b, florence-2 to base, wevisone-vl to 2b.

LibreVLM, LibreLFM2VL, LibreQwen3VL, LibreSmolVLM2, LibreInternVL3, LibreFlorence2, LibreKosmos2, LibreLocateAnything, LibreWeVisOneVL and LibreMODUS (also spelled LibreModus) are exported at package level.

Detection also includes north-micro-vision, gemma-4-e2b, gemma-4-e4b, moondream-2, moondream-3 and lfm2-vl-3b. The bare gemma-4 alias selects E4B. Molmo2 aliases are molmo2-4b, molmo2-8b and molmo2-o-7b; its default is 4B. Aliases establish routing, not that every remote snapshot has been downloaded and tested. moondream-3 does not load in 1.6.0: its mirror is missing weight shards.

Tasks

Task support is family-specific. These adapters support multiple tasks:

Because the task is prompt-driven rather than baked into a checkpoint, it can be switched on a loaded model:

model.set_task(task: str) -> LibreVLMModel

The task is validated against the family's supported list, is sticky across later predict() and track() calls, and the model is returned so calls can chain.

Molmo2 returns points and requires {label} in custom pointing templates. WeVisOne-VL's referring=True makes detect send one generation per phrase, and its segment, matte and referring detection raise ValueError without a vocabulary. Moondream supports detection, points and native chat. Use LibreGround for instruction-to-click queries.

set_classes

model.set_classes(classes: list[str]) -> LibreVLMModel

Sets the open vocabulary. Any words work, because the model is prompted with them rather than constrained to a fixed head. The list must be non-empty and its entries must be unique when compared case-insensitively. Passing a bare string raises TypeError, because it would enumerate into one-character classes. The vocabulary is sticky: set it once after loading and it persists until set again.

chat

model.chat(image, prompt, max_new_tokens=None, color_format="auto") -> str

Raw multimodal generation: image and prompt in, decoded text out, verbatim. This is the escape hatch under the detection convenience, for free-form questions, counting, or an output format the detection wrapper does not cover. max_new_tokens falls back to the family's MAX_NEW_TOKENS, which is 1024 on the base class. Decoding is greedy with a mild repetition penalty.

Confidence

Generated output has no calibrated per-box confidence. This version assigns a constant placeholder so predict, drawing and track behave, which makes conf= filtering and mAP soft rather than meaningful. This is also why val() raises: COCO mAP over placeholder scores would mislead.

Predict and track

The standard predict surface applies, and track() works, so a VLM detector drops into the same pipeline as any other family. Two class-level policies differ from a convolutional detector: test-time augmentation is disabled, because multi-scale augmentation is meaningless for a fixed-resolution generator, and batched predict is off, because generation is autoregressive and preprocessing returns a text-and-image encoding rather than a stackable image tensor.

Not supported

Export and detection-mAP validation are unsupported. Training support is limited to the Qwen3-VL workflow below.

Remote code

Every shipped family loads through a native model class, so LibreYOLO does not execute third-party repository code by default. A family that genuinely needs it must opt in explicitly and pin a snapshot revision; LocateAnything is the one that does, pinned to commit c32291ca5e996f5a7a485845b4f57a233936bba0.

LibreMODUS is an explicit exception to the checkpoint schema: its alias resolves to a directory of pinned upstream files rather than a LibreYOLO .pt, and LibreYOLO neither adds v1.0 metadata to it nor republishes it.

Train

Qwen3-VL supports detection LoRA through train(data=...) after installing libreyolo[vlm-train]. It freezes the vision tower, chooses best checkpoints by validation loss and saves checkpoint directories. See VLM fine-tuning.