LibreVLM

LibreVLM lives behind the optional vlm extra. It pulls in a recent transformers and the helpers a couple of processors need. Without the extra, importing a VLM family raises an ImportError that points you here.

1
pip install 'libreyolo[vlm]'

Weights are downloaded from the Hugging Face Hub on first use into a local weights/ folder. A few families ship under non-OSI licenses and log a one-time notice before downloading. A GPU is recommended for the larger backends, but every model also runs on CPU with device="cpu".

Construct a model, declare the words you care about, and predict. The default backend is Qwen3-VL-4B, the strongest detector in the tier and Apache-2.0 licensed.

1
from libreyolo import LibreVLM
2
3
# Qwen3-VL-4B by default; weights autodownload on first use
4
model = LibreVLM()
5
6
# The vocabulary is just words. Any words.
7
model.set_classes(["pink car", "wheel"])
8
9
result = model.predict("street.jpg")
10
11
print(result.boxes.xyxy)   # pixel [x1, y1, x2, y2]
12
print(result.boxes.cls)    # ids into ["pink car", "wheel"]
13
result.plot()              # same drawing helpers as any LibreYOLO model
14
result.save("out.jpg")

That is the whole loop. Everything downstream of predict() behaves like a normal detector, so existing visualization, cropping, and tracking code keeps working.

Pick a backend with the alias you pass to LibreVLM(...). A bare family name resolves to its default size. The default backend overall is qwen3-vl-4b. In practice the strongest detectors are Qwen3-VL, LFM2-VL, and Florence-2.

Choosing a backend

• Best quality: qwen3-vl-8b or qwen3-vl-4b (the default).
• Tight boxes, small footprint: florence-2-large.
• Edge / CPU: lfm2-vl-450m or smolvlm2-500m.
• Fully permissive license: any Qwen3-VL, SmolVLM2, Florence-2 or Kosmos-2 size.

The vocabulary is the heart of open-vocabulary detection. Call set_classes() with a list of label strings. It is sticky: it persists across every later predict() and track() call until you set it again. It returns self, so it chains.

1
# Sticky and chainable
2
model = LibreVLM("qwen3-vl-2b").set_classes(["person", "dog", "cat"])
3
4
# Set it once at construction instead
5
model = LibreVLM("lfm2-vl-450m", names=["boat"], device="cpu")
6
7
# Re-set any time to change what you are looking for
8
model.set_classes(["a red car", "a blue truck"])

Labels can be any phrase. They must be unique case-insensitively, and you must pass a list, not a bare string. If you never call set_classes(), the model falls back to the COCO-80 vocabulary so a bare predict() still does something sensible.

predict() (and the equivalent model(...) call) accepts the same source types as any LibreYOLO detector: a path, a PIL image, a numpy array, a URL, a folder, or a video. stream=True and track() work too.

1
result = model.predict(
2
    source="image.jpg",  # path | PIL | ndarray | URL | folder | video
3
    conf=0.25,           # see note below: scoring is synthetic
4
    classes=[0],         # optional: keep only these vocabulary ids
5
    max_det=300,
6
)

Return shape

You get back the standard Results object, identical to a closed-vocabulary detector:

Under the hood, LibreVLM tolerantly parses the model output (handling markdown fences, stray prose, duplicated boxes, and truncated arrays), maps free-text labels back to your class ids, and drops any label that is not in your vocabulary. That last step is what makes a free-form generator behave like a closed-set detector.

Detect a specific colored object

1
from libreyolo import LibreVLM
2
3
model = LibreVLM("qwen3-vl-4b")
4
model.set_classes(["red car"])
5
6
result = model.predict("parking_lot.jpg")
7
print(f"Found {len(result.boxes.cls)} red car(s)")
8
result.save("red_cars.jpg")

Tight boxes with Florence-2

1
# Florence-2 is a purpose-built grounder: very tight pixel boxes.
2
model = LibreVLM("florence-2-large")
3
model.set_classes(["a red car", "license plate"])
4
5
result = model.predict("car.jpg")
6
result.plot()

Filter to a single class on the fly

1
model = LibreVLM("qwen3-vl-2b").set_classes(["person", "dog", "cat"])
2
3
# classes= filters the configured vocabulary by id
4
people_only = model.predict("street.jpg", classes=[0])

Run on CPU with a built-in sample image

1
from libreyolo import LibreVLM, SAMPLE_IMAGE
2
3
model = LibreVLM("lfm2-vl-450m", device="cpu")
4
# No set_classes() -> falls back to the COCO-80 vocabulary
5
result = model.predict(SAMPLE_IMAGE)
6
print(model.names[result.boxes.cls[0]])  # e.g. "person"

Batches, folders, and video

1
model = LibreVLM().set_classes(["forklift", "pallet"])
2
3
# A whole folder
4
for result in model.predict("warehouse_frames/", stream=True):
5
    result.save()
6
7
# A video file (frames are processed one at a time)
8
model.predict("warehouse.mp4", save=True)

Sometimes you want the model, not the detector. The chat-template families expose chat(), which takes an image and a free-form prompt and returns the decoded text verbatim. Use it for counting, captioning, or quick visual questions.

1
model = LibreVLM("qwen3-vl-4b")
2
3
answer = model.chat("harbor.jpg", "How many boats are docked? Answer with a number.")
4
print(answer)

Every family returns the same Results, but they reach it differently. You rarely need to care, yet it helps to know why some backends behave the way they do. The chat families are prompted for a JSON array of boxes; the grounders use dedicated task tokens.

For the chat families you can override the detection prompt with the prompt= constructor argument, and cap generation length with max_new_tokens=. Device and dtype are resolved automatically: bf16 or fp16 on CUDA, fp32 on CPU.

LibreVLM is powerful but young. Knowing the boundaries up front saves surprises later.

• Synthetic confidence. Every box is scored 1.0. The conf= filter therefore behaves as all-or-nothing rather than a real threshold.
• No mAP / validation. val() raises, because synthetic scores would make COCO mAP misleading.
• No training or export. train() and export() raise. Fine-tune the VLM upstream and load the resulting weights instead.
• Tracking is degraded. track() runs, but uniform scores make the tracker's low-confidence recovery stage inert.
• One image at a time. Generation is sequential in v1, so larger batch= values give no speedup.
• Python API only. The libreyolo CLI does not resolve VLM aliases yet.