# LibreMODUS
LibreMODUS is an inference-only integration of the MODUS 14B-A7B checkpoint, an any-to-any model that turns one image-derived input into another: RGB in, depth out; depth in, normals out; any of those plus a phrase, boxes out. LibreYOLO supports four tasks through the standard predict API and a wider set through any2any().
Tasks: Detection, Depth, Surface normals, Edges. Install: pip install "libreyolo[modus]".
Verified against LibreYOLO v1.5.0.

## Install

LibreMODUS needs its own extra, which pulls in `accelerate` for the big-model dispatch this checkpoint needs.

```bash
pip install "libreyolo[modus]"
```

LibreYOLO does not redistribute or mirror MODUS weights. By default, loading a `LibreMODUS` model downloads the required files directly from `EPFL-VILAB/MODUS` at a pinned Hugging Face revision, and a fresh download always needs the user's own authenticated Hugging Face account, even if the upstream hosting gate is temporarily open. Review and accept the upstream terms, then authenticate:

```bash
hf auth login
```

```python
from libreyolo import LibreMODUS

model = LibreMODUS(token="hf_...")
```

To avoid any network request, point at a snapshot you already have:

```python
model = LibreMODUS(checkpoint_path="/models/MODUS")
```

That directory must contain `model.safetensors`, `ae.safetensors`, `llm_config.json`, `vit_config.json`, `tokenizer_config.json`, `vocab.json` and `merges.txt`. See Licensing below for what the checkpoint's terms permit.

## Predict

**Python**

```python
from libreyolo import LibreMODUS

model = LibreMODUS(size="14b-a7b", task="normal")
result = model.predict("room.jpg")
normals = result.normal_map.data

model.set_task("edge")
result = model.predict("room.jpg")
edges = result.edges.data

# With no custom vocabulary, detect decodes the checkpoint's COCO
# label tokens into contiguous COCO-80 class ids.
model.set_task("detect")
result = model.predict("street.jpg")
print(result.boxes.xyxy)
```

**Phrase grounding**

```python
from libreyolo import LibreMODUS

model = LibreMODUS(task="detect")
# set_classes() switches detection to phrase grounding: each phrase
# runs independently and returns through the same Boxes contract.
model.set_classes(["red bus", "cyclist"])
result = model.predict("street.jpg", conf=0.2)
print(result.boxes.xyxy, result.boxes.cls)
```

**any2any()**

```python
from libreyolo import LibreMODUS

model = LibreMODUS()

# One to three image-derived inputs (rgb, depth, normal, canny/edge),
# plus optional auxiliary text, composed toward one target.
result = model.any2any(
    inputs={"rgb": "room.jpg"},
    target="normal",
    steps=10,
    cfg=2.0,
    seed=0,
)
normals = result.normal_map.data

# Grounding through any2any() needs a text input naming the phrase.
result = model.any2any(
    {"rgb": "street.jpg", "text": "red bus"},
    target="grounding",
)
print(result.boxes.xyxy)
```

The standard task API covers four tasks, each mapped to one MODUS target: `depth` to relative depth (`result.depth_map`), `normal` to surface normals (`result.normal_map`), `edge` to Canny-style edges (`result.edges`), and `detect` to COCO-80 boxes (`result.boxes`) unless `set_classes()` switches it to phrase grounding. `set_task()` switches between them on the same loaded model. The released recipe uses ten flow-sampling steps with text guidance 4.0 and image guidance 2.0; override them with `inference_steps=`, `inference_cfg=` and `inference_image_cfg=` at construction.

`any2any()` reaches the wider public analysis surface: one to three image-derived inputs (`rgb`, `depth`, `normal`, `canny`/`edge`), plus optional auxiliary text, composed toward any one of depth, normals, edges, SAM-derived edges, COCO detection or phrase grounding. All image-derived inputs must describe the same aligned canvas; LibreMODUS rejects mismatched widths and heights rather than resizing them independently. `chain=(...)` generates intermediate targets and feeds them back into the same context, within the checkpoint's three-condition training budget. `verify=N` (N >= 2) generates N candidates and keeps the one that scores highest on a constrained self-consistency check, exposed as `result.verification_score`.

`dtype="bf16"` (the default) matches the released checkpoint precision; `dtype="fp8"` stores eligible decoder-trunk linear weights as E4M3 with a per-output-channel scale, converts once into a local cache under `~/.cache/libreyolo/modus/fp8`, and dequantizes to the input dtype per matrix multiply, so it trades memory rather than trading accuracy at the activation level.

`train()`, `val()` and `export()` all raise: LibreMODUS is inference-only, dataset validation is not offered, and there is no ONNX, TensorRT or TFLite export path. Batched `predict()` and test-time augmentation are also not supported; each call handles one image.

## Licensing

Check the license on the Hugging Face repository of the specific weights you download. That repository is authoritative and licenses are not always uniform across a family. This is a description of the licenses involved, not legal advice.

- Original work: MODUS, EPFL Visual Intelligence and Learning Lab (VILAB)
- Upstream license: Apache-2.0 (code); research-only per the upstream model card (weights)
- Upstream source: https://github.com/EPFL-VILAB/Modus
- LibreYOLO code: Apache-2.0
- Weights: Apache-2.0 (code); research-only per the upstream model card (weights), distributed by their authors. LibreYOLO does not host or mirror them.
- Interpretation: Apache-2.0 covers the EPFL-VILAB/Modus source repository that LibreYOLO's adapter follows; LibreYOLO's own port is MIT, permissive and usable in commercial and closed-source products, requiring only that license text and attribution travel with any copy you redistribute. The MODUS-14B-A7B checkpoint is a separate artifact under a separate license: its Hugging Face model card currently declares license: other, describes it as bagel-derived, and requests research-only use, terms that bind you directly rather than through LibreYOLO. LibreYOLO does not bundle, mirror, or redistribute this checkpoint in any form, quantized or otherwise; loading it always downloads the required files directly from EPFL-VILAB/MODUS at a pinned revision, which needs the user's own authenticated Hugging Face account and acceptance of the current upstream terms, or a checkpoint_path to a snapshot the user already obtained. One upstream file, modeling/bagel/modeling_utils.py, carries an incompatible CC BY-NC 4.0 license inherited from Meta's DiT and is not copied, adapted, or paraphrased anywhere in this port; the small permissive routines LibreYOLO needed were re-derived independently from Hugging Face Transformers (Apache-2.0) and OpenAI's guided-diffusion (MIT) instead. Training is not offered for this family.

LibreYOLO does not host or mirror the MODUS checkpoint anywhere, including on its own Hugging Face org: loading it always pulls the pinned revision directly from EPFL-VILAB/MODUS, or reads a snapshot already on disk at `checkpoint_path`.

## Citation

```bibtex
@article{ye2026modus,
  title   = {MODUS: Decoder-only Any-to-Any Modeling of Diverse Modalities},
  author  = {Ye, Mingqiao and An, Zhaochong and Gao, Zhitong and Liu, Xian
             and Fleuret, Fran\c{c}ois and Li, Chuan and Zadeh, Amir
             and Belongie, Serge and Dehghan, Afshin and Allardice, Jesse
             and Mizrahi, David and Kar, O\u{g}uzhan Fatih and Bachmann, Roman
             and Zamir, Amir},
  journal = {arXiv preprint arXiv:2607.25948},
  year    = {2026},
}
```

Copied from https://github.com/EPFL-VILAB/Modus#citation