SegFormer

Install

SegFormer needs no optional extra. Everything it imports is in the base install.

pip install libreyolo

Predict

Weights download from Hugging Face on first use and are cached locally.

from libreyolo import LibreYOLO, SAMPLE_IMAGE model = LibreYOLO("LibreSegformerb0-sem.pt")result = model(SAMPLE_IMAGE, save=True) mask = result.semantic_maskprint(mask.data.shape, mask.classes)

libreyolo predict model=LibreSegformerb0-sem.pt source=https://raw.githubusercontent.com/LibreYOLO/libreyolo/release/libreyolo/assets/parkour.jpg save=True

result.semantic_mask carries the dense class map: .data is an (H, W) tensor of class IDs on the original image size, and .classes lists the class IDs actually present. result.boxes is None, since there are no per-instance detections. conf and iou are accepted for API parity but do not change the output: the model returns one class per pixel, not per-instance detections to filter or de-duplicate. See prediction for sources, streaming and result handling.

Variants

Six sizes, b0 through b5, widening and deepening the Mix Transformer encoder at each step while keeping the same all-MLP decode head design.

Train

train() fine-tunes a published checkpoint by default. Pass no model_path to LibreSegformer(...) instead and it builds with a randomly initialized encoder and head, training from scratch, the only route to weights that carry none of the pretrained checkpoints' non-commercial restriction (see Licensing).

from libreyolo import LibreYOLO model = LibreYOLO("LibreSegformerb0-sem.pt")model.train(data="my-dataset.yaml", epochs=160, imgsz=512, batch=8)

libreyolo train model=LibreSegformerb0-sem.pt data=my-dataset.yaml \  epochs=160 imgsz=512 batch=8

from libreyolo.models.segformer.model import LibreSegformer # No model_path: random init, nothing downloaded. The only route to# weights free of the pretrained checkpoints' non-commercial term.model = LibreSegformer(size="b0", nb_classes=150)model.train(data="my-dataset.yaml", epochs=160, imgsz=512, batch=8)

libreyolo train model=LibreSegformerb0-sem.pt data=my-dataset.yaml \  epochs=160 device=0,1 batch=16

Left alone, the trainer follows the SegFormer paper's ADE20K recipe: AdamW at a backbone base learning rate with the decode head trained at 10x that rate, weight decay everywhere except LayerNorm and the Mix-FFN positional convolution, and a linear decay schedule with a warmup. Convergence for the larger sizes, b3 through b5, has not been validated end to end.

See training for datasets, augmentation, multi-GPU and loggers.

Validate

val() returns a dictionary of metrics/ keys: mIoU and pixel accuracy, measured against any dataset in the format you trained on.

from libreyolo import LibreYOLO model = LibreYOLO("LibreSegformerb0-sem.pt")metrics = model.val(data="my-dataset.yaml") print(metrics["metrics/mIoU"])print(metrics["metrics/pixel_accuracy"])

libreyolo val model=LibreSegformerb0-sem.pt data=my-dataset.yaml

Export

An exported artifact loads back through LibreYOLO() on its file suffix, so a .onnx or .engine file behaves like a checkpoint and returns the same Results. Export lists the arguments every format accepts.

from libreyolo import LibreYOLO model = LibreYOLO("LibreSegformerb0-sem.pt")model.export(format="onnx", imgsz=512)model.export(format="tensorrt", imgsz=512, half=True)

libreyolo export model=LibreSegformerb0-sem.pt format=onnx imgsz=512libreyolo export model=LibreSegformerb0-sem.pt format=tensorrt imgsz=512 half=True

from libreyolo import LibreYOLO, SAMPLE_IMAGE # The factory routes on the file suffix, so an exported artifact loads# like any checkpoint and returns the same Results object.model = LibreYOLO("LibreSegformerb0-sem.onnx")result = model(SAMPLE_IMAGE) print(result.semantic_mask.data.shape)

Checkpoints

Every published weight file for this family.

Licensing

Citation

@inproceedings{xie2021segformer,
  title={SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers},
  author={Xie, Enze and Wang, Wenhai and Yu, Zhiding and Anandkumar, Anima and Alvarez, Jose M and Luo, Ping},
  booktitle={Neural Information Processing Systems (NeurIPS)},
  year={2021}
}