# Kosmos-2
Kosmos-2 is Microsoft's grounding model: it captions an image, then locates each noun phrase in that caption with a box. LibreYOLO wraps it as an open-vocabulary object detector: supply the class list at predict time.
Tasks: Detection. Install: pip install "libreyolo[vlm]".
Verified against LibreYOLO v1.5.0.

## Install

Kosmos-2 belongs to LibreYOLO's VLM-as-detector tier, a separate product
surface from the checkpoint-based families with its own factory. It needs the
`vlm` extra.

```bash
pip install "libreyolo[vlm]"
```

## Predict

Weights download from Hugging Face on first use and are cached locally.
LibreYOLO loads Microsoft's own `microsoft/kosmos-2-patch14-224` repository
directly; unlike Florence-2, no community re-upload is needed here.

**Python**

```python
from libreyolo import LibreVLM, SAMPLE_IMAGE

model = LibreVLM("kosmos-2")
model.set_classes(["boat", "person"])
result = model.predict(SAMPLE_IMAGE, save=True)

for box in result.boxes:
    print(box.cls, box.conf, box.xyxy)
```

**Video**

```python
from libreyolo import LibreVLM

model = LibreVLM("kosmos-2")
model.set_classes(["boat", "person"])

# Any source the library accepts: file, folder, URL, webcam index,
# RTSP stream, or a .streams list
for result in model.predict("clip.mp4", stream=True, save=True):
    print(len(result.boxes))
```

This family loads through the `LibreVLM()` factory, not `LibreYOLO()`: VLM
families declare no checkpoint loader, so the file-suffix routing described on
other model pages does not apply here. `set_classes()` sets the vocabulary
Kosmos-2 is asked to find; it is sticky, so it stays in effect across every
later `predict()`/`track()` call until you set it again. Kosmos-2 grounds noun
phrases rather than matching a label exactly, so LibreYOLO's wrapper accepts a
partial match: a class named `"boat"` also matches a generated phrase like "the
boats". Every detection carries the same placeholder confidence, so `conf`
filtering is all-or-nothing rather than a ranking, and `iou` has no effect here,
since the wrapper builds the detection list directly from the grounded entities
with no deduplication step. `chat()` raises `NotImplementedError`, because
Kosmos-2 is driven by a `<grounding>` prompt rather than a chat template.
LibreYOLO's CLI does not cover this tier: there is no
`libreyolo predict model=...` form for it. See [prediction](/docs/predict) for
sources, streaming and result handling.

## Variants

One size: `kosmos-2-patch14-224`, at 224 px, loaded as `LibreVLM("kosmos-2")`.
It is a 2023-era model, and LibreYOLO's own wrapper notes its grounding is
coarser than the newer detectors in this tier.

LibreYOLO does not train, validate or export Kosmos-2: `train()`, `val()` and
`export()` all raise `NotImplementedError` for every family in this tier (see
the support tier above). Fine-tune Kosmos-2 upstream and load the resulting
weights if you need a custom vocabulary baked in; check `predict()` output by
eye instead of a COCO-style validation pass, since every detection carries the
same placeholder confidence.

## Licensing

Check the license on the Hugging Face repository of the specific weights you download. That repository is authoritative and licenses are not always uniform across a family. This is a description of the licenses involved, not legal advice.

- Original work: Kosmos-2, Microsoft
- Upstream license: MIT
- Upstream source: https://github.com/microsoft/unilm/tree/master/kosmos-2
- LibreYOLO code: MIT
- Weights: MIT, distributed by their authors. LibreYOLO does not host or mirror them.
- Interpretation: MIT is a permissive license, so these weights can be used in commercial and closed-source products, provided the copyright notice and license text travel with any copy you redistribute. LibreYOLO downloads Microsoft's own microsoft/kosmos-2-patch14-224 repository directly; unlike Florence-2, no community re-upload is needed for it to load on current transformers.

## Citation

```bibtex
@article{kosmos-2,
  title={Kosmos-2: Grounding Multimodal Large Language Models to the World},
  author={Zhiliang Peng and Wenhui Wang and Li Dong and Yaru Hao and Shaohan Huang and Shuming Ma and Furu Wei},
  journal={ArXiv},
  year={2023},
  volume={abs/2306}
}
```

Copied from https://github.com/microsoft/unilm/blob/master/kosmos-2/README.md#citation


## Kosmos-2 architecture

- Kosmos-2 patch14-224: [SVG](/diagrams/models/kosmos-2/224-detect.svg), [interactive diagram](/diagrams/models/kosmos-2/224-detect.html)