Vision-language API

Install

The tier needs the vlm extra.

pip install 'libreyolo[vlm]'

The factory

LibreVLM(model: str = "qwen3-vl-4b", **kwargs) -> LibreVLMModel

model is an alias, not a path. **kwargs reaches the family constructor, which takes device, names (the initial vocabulary, equivalent to calling set_classes after load), prompt (override the detection prompt) and max_new_tokens. An unknown alias raises ValueError listing every alias.

from libreyolo import LibreVLM, SAMPLE_IMAGE model = LibreVLM("lfm2-vl-450m")model.set_classes(["person", "skateboard"]) result = model.predict(SAMPLE_IMAGE)for box, cls in zip(result.boxes.xyxy, result.boxes.cls):    print(result.names[int(cls)], box.tolist())

from libreyolo import LibreVLM, SAMPLE_IMAGE model = LibreVLM("lfm2-vl-450m")print(model.chat(SAMPLE_IMAGE, "How many people are in this image?"))

Aliases

The default alias is qwen3-vl-4b. Sizes for the default alias of each family are the ones listed first: qwen3-vl resolves to 4b, lfm2-vl to 450m, internvl3 to 2b, smolvlm2 to 2.2b, florence-2 to base.

LibreVLM, LibreLFM2VL, LibreQwen3VL, LibreSmolVLM2, LibreInternVL3, LibreFlorence2, LibreKosmos2, LibreLocateAnything and LibreMODUS (also spelled LibreModus) are exported at package level.

Tasks

Most families serve detect only. Two serve more:

Because the task is prompt-driven rather than baked into a checkpoint, it can be switched on a loaded model:

model.set_task(task: str) -> LibreVLMModel

The task is validated against the family's supported list, is sticky across later predict() and track() calls, and the model is returned so calls can chain.

set_classes

model.set_classes(classes: list[str]) -> LibreVLMModel

Sets the open vocabulary. Any words work, because the model is prompted with them rather than constrained to a fixed head. The list must be non-empty and its entries must be unique when compared case-insensitively. Passing a bare string raises TypeError, because it would enumerate into one-character classes. The vocabulary is sticky: set it once after loading and it persists until set again.

chat

model.chat(image, prompt, max_new_tokens=None, color_format="auto") -> str

Raw multimodal generation: image and prompt in, decoded text out, verbatim. This is the escape hatch under the detection convenience, for free-form questions, counting, or an output format the detection wrapper does not cover. max_new_tokens falls back to the family's MAX_NEW_TOKENS, which is 1024 on the base class. Decoding is greedy with a mild repetition penalty.

Confidence

Generated output has no calibrated per-box confidence. This version assigns a constant placeholder so predict, drawing and track behave, which makes conf= filtering and mAP soft rather than meaningful. This is also why val() raises: COCO mAP over placeholder scores would mislead.

Predict and track

The standard predict surface applies, and track() works, so a VLM detector drops into the same pipeline as any other family. Two class-level policies differ from a convolutional detector: test-time augmentation is disabled, because multi-scale augmentation is meaningless for a fixed-resolution generator, and batched predict is off, because generation is autoregressive and preprocessing returns a text-and-image encoding rather than a stackable image tensor.

Not supported

train(), val() and export() raise NotImplementedError. Fine-tune upstream and load the resulting weights.

Remote code

Every shipped family loads through a native model class, so LibreYOLO does not execute third-party repository code by default. A family that genuinely needs it must opt in explicitly and pin a snapshot revision; LocateAnything is the one that does, pinned to commit c32291ca5e996f5a7a485845b4f57a233936bba0.

LibreMODUS is an explicit exception to the checkpoint schema: its alias resolves to a directory of pinned upstream files rather than a LibreYOLO .pt, and LibreYOLO neither adds v1.0 metadata to it nor republishes it.