> This is the documentation for LibreYOLO 1.5.0. Current version: https://www.libreyolo.com/docs/models/clip

# CLIP
CLIP is a dual-tower model that scores an image against text prompts instead of a fixed label set. LibreYOLO supports it for zero-shot classification and image/text embedding, with no training step.
Tasks: classify, embed. Install: pip install libreyolo.
Verified against LibreYOLO v1.5.0.

## Install

CLIP needs its own extra, which pulls in the packages its vendored BPE tokenizer uses to reproduce exact token ids.

```bash
pip install "libreyolo[clip]"
```

## Predict

Weights download from Hugging Face on first use and are cached locally.

**Python**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

model = LibreYOLO("LibreCLIPb32-cls.pt")
model.set_classes(["a forklift", "an empty aisle", "a spill"])
result = model(SAMPLE_IMAGE, save=True)

print(model.names[result.probs.top1], float(result.probs.top1conf))
```

**CLI**

```bash
# With no set_classes() call, CLI predict uses the 1,000 ImageNet
# class names the model loads with by default.
libreyolo predict model=LibreCLIPb32-cls.pt source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/parkour.jpg save=True
```

**Image and text embedding**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

model = LibreYOLO("LibreCLIPb32-cls.pt", task="embed")
image_embed = model(SAMPLE_IMAGE).embeddings.data
text_embed = model.embed_text("a photo of a forklift")

# Both are L2-normalized, so a plain dot product is cosine similarity.
similarity = (image_embed @ text_embed.T).item()
```

`set_classes()` is the one primitive that makes this an open-vocabulary classifier: it renders each label into every prompt template, encodes and averages the results, and caches the resulting `[K, D]` matrix as the classifier head, so it is not recomputed per image. Call it again to change classes at any time. With no call, LibreCLIP loads with the 1,000 ImageNet-1k class names already set.

With `task="embed"`, prediction returns one L2-normalized image vector per input instead of class probabilities, and `embed_text()` returns normalized text rows in the same vector space, so a plain dot product between them is cosine similarity. `iou` has no effect on either task; there is no NMS step. See [prediction](/docs/v1.5.0/predict) for sources, streaming and result handling.

## Validate

`val()` reads the class-folder names under an ImageFolder `train/` split, calls `set_classes()` with them, then measures zero-shot top-1 and top-5 accuracy. Accuracy depends on how the class names read as prompts, not on any weight update, since there is nothing to train. Validation only covers `task="classify"`; `task="embed"` has no dataset validator.

**Python**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreCLIPb32-cls.pt")

# data is an ImageFolder root with a train/ split; its folder names
# become the zero-shot class prompts for this run.
metrics = model.val(data="imagenette160")

print(metrics["metrics/accuracy_top1"])
print(metrics["metrics/accuracy_top5"])
```

**CLI**

```bash
libreyolo val model=LibreCLIPb32-cls.pt data=imagenette160
```

## Export

| Task | ONNX | TorchScript | ExecuTorch | TensorRT | OpenVINO | Paddle | MNN | RKNN | ncnn | TFLite | CoreML | Core AI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| classify | yes | yes | yes | yes | yes |  |  |  |  |  |  | yes |
| embed | yes | yes | yes | yes | yes |  |  |  |  |  |  |  |

A "yes" means the export is supported. An empty cell means the exporter refuses that combination.

Export bakes the model's current state into a fixed graph. For `task="classify"`, whatever labels `set_classes()` last set, and the resolution at export time, are baked into a final linear layer, so the exported ONNX or TensorRT graph is an ordinary `[B, K]` image classifier with no text tower and no tokenizer; export again after changing either the classes or the size. `task="embed"` export traces the image tower alone. Both need ONNX opset 14 or higher, which the exporter sets by default.

**Python**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreCLIPb32-cls.pt")
model.set_classes(["a forklift", "an empty aisle", "a spill"])
model.export(format="onnx")

# The current set_classes() labels and the input resolution are baked
# into the graph. Re-export after changing either one.
```

**CLI**

```bash
# No set_classes() call here, so this bakes in the default 1,000
# ImageNet classes the model loads with.
libreyolo export model=LibreCLIPb32-cls.pt format=onnx
```

**Embedding export**

```python
from libreyolo import LibreYOLO

# task="embed" traces the image tower alone; no classes needed.
model = LibreYOLO("LibreCLIPb32-cls.pt", task="embed")
model.export(format="onnx")
```

## Checkpoints

Every published weight file for this family. Both are converted from OpenCLIP's LAION-2B-trained checkpoints (`ViT-B-32` and `ViT-B-16`), not from any COCO training run.

| File | Input (px) | Task | Weights license |
| --- | --- | --- | --- |
| `LibreCLIPb32-cls.pt` |  | classify | mit |
| `LibreCLIPb16-cls.pt` |  | classify | mit |

The LAION-2B training data has a documented history of CSAM content (Stanford Internet Observatory, December 2023). LAION has since released Re-LAION, a cleaned re-release; prefer Re-LAION-derived checkpoints where available if you re-host these weights further.

## Licensing

Check the license on the Hugging Face repository of the specific weights you download. That repository is authoritative and licenses are not always uniform across a family. This is a description of the licenses involved, not legal advice.

- Original work: CLIP / OpenCLIP, OpenAI; LAION / ML Foundations
- Upstream license: MIT
- Upstream source: https://github.com/mlfoundations/open_clip
- LibreYOLO code: MIT
- Weights: MIT, republished at https://huggingface.co/LibreYOLO
- Interpretation: MIT is a permissive license, so these weights can be used in commercial and closed-source products. It asks you to keep the license text and copyright notice with any copy you redistribute, and it places no obligation on your own application code. LibreCLIP's tokenizer is vendored from the OpenAI CLIP / OpenCLIP byte-pair-encoding tokenizer (also MIT); its image and text towers are a clean-room re-implementation of the standard CLIP architecture, built without open_clip as a runtime dependency. The shipped checkpoints (b32, b16) are converted from OpenCLIP's LAION-2B-trained weights, which OpenCLIP publishes as MIT-redistributable. Training is not offered for this family: LibreCLIP is zero-shot, and set_classes() replaces the fine-tuning step a trained classifier would otherwise need.

## Citation

```bibtex
@software{ilharco_gabriel_2021_5143773,
  author       = {Ilharco, Gabriel and
                  Wortsman, Mitchell and
                  Wightman, Ross and
                  Gordon, Cade and
                  Carlini, Nicholas and
                  Taori, Rohan and
                  Dave, Achal and
                  Shankar, Vaishaal and
                  Namkoong, Hongseok and
                  Miller, John and
                  Hajishirzi, Hannaneh and
                  Farhadi, Ali and
                  Schmidt, Ludwig},
  title        = {OpenCLIP},
  month        = jul,
  year         = 2021,
  note         = {If you use this software, please cite it as below.},
  publisher    = {Zenodo},
  version      = {0.1},
  doi          = {10.5281/zenodo.5143773},
  url          = {https://doi.org/10.5281/zenodo.5143773}
}
```

Copied from https://github.com/mlfoundations/open_clip#citing


## CLIP architecture

- CLIP b32 classify: [SVG](/diagrams/models/clip/b32-classify.svg), [interactive diagram](/diagrams/models/clip/b32-classify.html)
- CLIP b32 embed: [SVG](/diagrams/models/clip/b32-embed.svg), [interactive diagram](/diagrams/models/clip/b32-embed.html)
- CLIP b16 classify: [SVG](/diagrams/models/clip/b16-classify.svg), [interactive diagram](/diagrams/models/clip/b16-classify.html)
- CLIP b16 embed: [SVG](/diagrams/models/clip/b16-embed.svg), [interactive diagram](/diagrams/models/clip/b16-embed.html)
- CLIP l14 classify: [SVG](/diagrams/models/clip/l14-classify.svg), [interactive diagram](/diagrams/models/clip/l14-classify.html)
- CLIP l14 embed: [SVG](/diagrams/models/clip/l14-embed.svg), [interactive diagram](/diagrams/models/clip/l14-embed.html)
- CLIP family classify: [SVG](/diagrams/models/clip/family-classify.svg), [interactive diagram](/diagrams/models/clip/family-classify.html)
- CLIP family embed: [SVG](/diagrams/models/clip/family-embed.svg), [interactive diagram](/diagrams/models/clip/family-embed.html)