# EoMT
A segmentation network built on a plain vision transformer with no dedicated pixel decoder: extra learned queries added to the encoder itself predict the masks. LibreYOLO supports it for semantic, instance and panoptic segmentation.
Tasks: Semantic segmentation, Instance segmentation, Panoptic segmentation. Install: pip install "libreyolo[eomt]".
Verified against LibreYOLO v1.5.0.

## Install

EoMT needs no optional extra. Everything it imports is in the base install.

```bash
pip install libreyolo
```

## Predict

Weights download from Hugging Face on first use and are cached locally. The
task suffix in the filename (`-sem`, `-seg`, `-panoptic`) selects the task, and
`LibreYOLO()` infers it from that filename so no `task=` argument is needed.

**Semantic**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

model = LibreYOLO("LibreEoMTl-sem.pt")
result = model(SAMPLE_IMAGE, save=True)

mask = result.semantic_mask
print(mask.data.shape)   # (H, W) class ids
print(mask.classes)      # sorted class ids present in the image
```

**Instance segmentation**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

# The -seg suffix in the filename selects the instance task, so no
# task argument is needed here.
model = LibreYOLO("LibreEoMTl-seg.pt")
result = model(SAMPLE_IMAGE, save=True)

print(result.boxes.xyxy)
print(result.masks.data.shape)
```

**Panoptic**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

model = LibreYOLO("LibreEoMTl-panoptic.pt")
result = model(SAMPLE_IMAGE, save=True)

pan = result.panoptic
print(pan.data.shape)       # (H, W) segment ids
print(pan.segments_info)    # [{"id": ..., "category_id": ...}, ...]
```

**CLI**

```bash
libreyolo predict model=LibreEoMTl-sem.pt source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/parkour.jpg save=True
```

Semantic segmentation fills `result.semantic_mask`, a `(H, W)` array of
class ids on `.data`. Instance segmentation fills `result.boxes` and
`result.masks`, the same shape every other segmentation family returns.
Panoptic segmentation fills `result.panoptic`: a `(H, W)` segment-id map on
`.data`, plus `.segments_info`, a list of `{"id", "category_id"}` dicts, one
per segment. `conf` filters query selection; `iou` has no effect on the
semantic task, since it argmaxes per pixel with no NMS step. See
[prediction](/docs/predict) for sources, streaming and result handling.

## Variants

Three encoder sizes, s/b/l, all DINOv2-backed. The semantic checkpoint is
trained on ADE20K at 512 px; the instance and panoptic checkpoints are trained
on COCO at 640 px, with a second instance checkpoint trained at 1280 px.
Upstream ships DINOv2 instance-segmentation weights only at size l; s and b are
published for semantic and panoptic only. DINOv3-backed EoMT variants exist
upstream but are not shipped here, because they depend on gated non-commercial
DINOv3 weights.

LibreYOLO does not train EoMT: `train()` raises `NotImplementedError` for this
family, which the [support tier](/docs/models) above marks as inference only.

## Validate

`val()` dispatches by task. Semantic returns `metrics/mIoU` and
`metrics/pixel_accuracy`. Instance segmentation returns the same mask and box
mAP keys as other segmentation families. Panoptic returns Panoptic Quality as
`metrics/PQ`, split into `metrics/SQ` (segmentation quality) and `metrics/RQ`
(recognition quality), plus `metrics/PQ_things` and `metrics/PQ_stuff`.

**Semantic**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreEoMTl-sem.pt")
metrics = model.val(data="my-dataset.yaml")

print(metrics["metrics/mIoU"])
print(metrics["metrics/pixel_accuracy"])
```

**Instance segmentation**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreEoMTl-seg.pt")
metrics = model.val(data="my-dataset.yaml")

print(metrics["metrics/mAP50-95(M)"])   # masks
print(metrics["metrics/mAP50-95(B)"])   # boxes
```

**Panoptic**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreEoMTl-panoptic.pt")
metrics = model.val(data="my-dataset.yaml")

print(metrics["metrics/PQ"])
print(metrics["metrics/SQ"], metrics["metrics/RQ"])
```

**CLI**

```bash
libreyolo val model=LibreEoMTl-sem.pt data=my-dataset.yaml
```

## Export

| Task | ONNX | TorchScript | ExecuTorch | TensorRT | OpenVINO | Paddle | MNN | RKNN | ncnn | TFLite | CoreML | Core AI |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Semantic segmentation | yes | yes |  | yes | yes |  |  |  |  |  |  |  |
| Instance segmentation |  |  |  |  |  |  |  |  |  |  |  |  |
| Panoptic segmentation |  |  |  |  |  |  |  |  |  |  |  |  |

A "yes" means the export is supported. An empty cell means the exporter refuses that combination.

Only the semantic task exports today: instance and panoptic segmentation call
`export()` and get `NotImplementedError`, because their query-mask output has
no runtime export contract yet. An exported semantic artifact loads back
through `LibreYOLO()` on its file suffix, so a `.onnx` or `.engine` file
behaves like a checkpoint and returns the same `Results`.

**Python**

```python
from libreyolo import LibreYOLO

model = LibreYOLO("LibreEoMTl-sem.pt")
model.export(format="onnx")
model.export(format="tensorrt", half=True)
```

**CLI**

```bash
libreyolo export model=LibreEoMTl-sem.pt format=onnx
libreyolo export model=LibreEoMTl-sem.pt format=tensorrt half=True
```

**Use the exported file**

```python
from libreyolo import LibreYOLO, SAMPLE_IMAGE

# The factory routes on the file suffix, so an exported artifact loads
# like any checkpoint and returns the same Results object.
model = LibreYOLO("LibreEoMTl-sem.onnx")
result = model(SAMPLE_IMAGE)

print(result.semantic_mask.data.shape)
```

## Checkpoints

Every published weight file for this family.

| File | Input (px) | Task | Weights license |
| --- | --- | --- | --- |
| `LibreEoMTl-sem.pt` | 512 | Semantic segmentation | mit |
| `LibreEoMTl-seg.pt` | 640 | Instance segmentation | mit |
| `LibreEoMTl-seg-1280.pt` |  | Instance segmentation | mit |
| `LibreEoMTs-panoptic.pt` |  | Panoptic segmentation | mit |
| `LibreEoMTb-panoptic.pt` |  | Panoptic segmentation | mit |
| `LibreEoMTl-panoptic.pt` |  | Panoptic segmentation | mit |

## Licensing

Check the license on the Hugging Face repository of the specific weights you download. That repository is authoritative and licenses are not always uniform across a family. This is a description of the licenses involved, not legal advice.

- Original work: EoMT, TU Eindhoven, Mobile Perception Systems Lab
- Upstream license: MIT
- Upstream source: https://github.com/tue-mps/eomt
- LibreYOLO code: MIT
- Weights: MIT, republished at https://huggingface.co/LibreYOLO
- Interpretation: MIT is a permissive license, so this code and these weights can be used in commercial and closed-source products. It asks only that you keep the copyright and license notice with any copy you redistribute. LibreYOLO ships only the DINOv2-backed EoMT checkpoints, sizes s/b/l; the DINOv3 EoMT variants are excluded because they depend on gated non-commercial DINOv3 weights. DINOv2 itself is Apache-2.0. The semantic checkpoint is trained on ADE20K and the instance and panoptic checkpoints on COCO; both are research datasets, and users remain responsible for dataset-license compliance when validating or fine-tuning against them.

## Citation

```bibtex
@inproceedings{kerssies2025eomt,
  author    = {Kerssies, Tommie and Cavagnero, Niccol\`{o} and Hermans, Alexander and Norouzi, Narges and Averta, Giuseppe and Leibe, Bastian and Dubbelman, Gijs and {de Geus}, Daan},
  title     = {{Your ViT is Secretly an Image Segmentation Model}},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2025},
}
```

Copied from https://github.com/tue-mps/eomt#bibtex-citation


## EoMT architecture

- EoMT s semantic: [SVG](/diagrams/models/eomt/s-semantic.svg), [interactive diagram](/diagrams/models/eomt/s-semantic.html)
- EoMT b semantic: [SVG](/diagrams/models/eomt/b-semantic.svg), [interactive diagram](/diagrams/models/eomt/b-semantic.html)
- EoMT l semantic: [SVG](/diagrams/models/eomt/l-semantic.svg), [interactive diagram](/diagrams/models/eomt/l-semantic.html)
- EoMT family semantic: [SVG](/diagrams/models/eomt/family-semantic.svg), [interactive diagram](/diagrams/models/eomt/family-semantic.html)
- EoMT s segment: [SVG](/diagrams/models/eomt/s-segment.svg), [interactive diagram](/diagrams/models/eomt/s-segment.html)
- EoMT b segment: [SVG](/diagrams/models/eomt/b-segment.svg), [interactive diagram](/diagrams/models/eomt/b-segment.html)
- EoMT l segment: [SVG](/diagrams/models/eomt/l-segment.svg), [interactive diagram](/diagrams/models/eomt/l-segment.html)
- EoMT family segment: [SVG](/diagrams/models/eomt/family-segment.svg), [interactive diagram](/diagrams/models/eomt/family-segment.html)
- EoMT s panoptic: [SVG](/diagrams/models/eomt/s-panoptic.svg), [interactive diagram](/diagrams/models/eomt/s-panoptic.html)
- EoMT b panoptic: [SVG](/diagrams/models/eomt/b-panoptic.svg), [interactive diagram](/diagrams/models/eomt/b-panoptic.html)
- EoMT l panoptic: [SVG](/diagrams/models/eomt/l-panoptic.svg), [interactive diagram](/diagrams/models/eomt/l-panoptic.html)
- EoMT family panoptic: [SVG](/diagrams/models/eomt/family-panoptic.svg), [interactive diagram](/diagrams/models/eomt/family-panoptic.html)
- EoMT l segment 1280px: [SVG](/diagrams/models/eomt/l-segment-1280.svg), [interactive diagram](/diagrams/models/eomt/l-segment-1280.html)