Quantization

Install

Quantization needs no extra. The module swap, the calibration pass and the simulated arithmetic all run in PyTorch, so pip install libreyolo is the whole requirement. The deployment artifacts need whatever their own format needs, which for the ONNX path is libreyolo[onnx].

Quantize

from libreyolo import LibreYOLO model = LibreYOLO("LibreYOLO9s.pt") # Structure swap plus calibration. calib is a small UNLABELED image set,# read forward-only to derive activation ranges and scales.qmodel = model.quantize(recipe="int8", calib="coco128.yaml", samples=128) print(qmodel.quant_info())qmodel.val(data="coco8.yaml")          # same validators as a float modelqmodel.save("LibreYOLO9s-int8.pt")     # checkpoint carries a quant manifest

libreyolo quantize --model LibreYOLO9s.pt --recipe int8 --calib coco128.yaml

model.quantize(    recipe="int8",    calib="coco128.yaml",      # data.yaml path or built-in name; None skips calibration    samples=128,               # maximum calibration images    batch=8,                   # calibration batch size    algorithm="auto",          # auto and minmax are the same; percentile is the alternative    keep_high_precision=None,  # None uses the family policy    verbose=True,)

quantize() transforms the loaded model in place and returns it. No gradients are involved: the swap installs quantized modules and the calibration pass runs forward only.

The resulting checkpoint is an ordinary LibreYOLO checkpoint with a quant manifest attached, so it reloads with its structure and scales intact:

from libreyolo import LibreYOLO # The quant manifest rebuilds the quantized structure and scales# before the weights are loaded.qmodel = LibreYOLO("LibreYOLO9s-int8.pt")print(qmodel.quant_info())

Trainer checkpoints written during a QAT run carry the manifest too, which means best.pt from such a run is itself a quantized checkpoint.

Recipes

Four families are supported: yolo9, rfdetr, birefnet and feynobg.

The sub-8-bit recipes target nn.Linear and are rejected for yolo9 on purpose: that acceleration is GEMM only on current hardware, so convolutions stay in higher precision. YOLO9 uses int8 or fp8. int2 is rejected for birefnet and feynobg because those families are inference only, so the QAT healing the recipe depends on is unavailable there.

Per-family defaults keep the first layer and the heads in float, and the YOLO9 DFL convolution is never quantized: it is a fixed integral-expectation operator. Override with keep_high_precision=("head.",) when you have a reason to.

Calibration data is not training data

calib= takes a few hundred images, reads no labels, and runs forward only to estimate activation ranges. data= in train() and val() is the labeled dataset used for gradients and metrics. They are different arguments with different purposes, and the default for calib is coco128.yaml.

algorithm="minmax" keeps the absolute extremes seen across calibration batches and is what "auto" selects. "percentile" uses the mean of per-batch 0.1 and 99.9 percentiles; it was measured to collapse DETR-family accuracy, because transformer activation outliers are load-bearing. What actually fixes small-model INT8 sensitivity is calibrating on enough batches: with the coco128 default, YOLO9-t lands within about one mAP point of its float score. The chosen algorithm is recorded in the checkpoint manifest.

Recover accuracy

from libreyolo import LibreYOLO qmodel = LibreYOLO("LibreYOLO9s-int8.pt") # A finetune, not a from-scratch run: use finetune learning rates.qmodel.train(data="coco8.yaml", epochs=5, lr0=1e-4)

qmodel.train(    data="coco8.yaml",    epochs=5,    lr0=1e-4,    distill_model="LibreYOLO9m.pt",)

libreyolo train --model LibreYOLO9s-int8.pt --data coco8.yaml --epochs 5 --lr0 1e-4

Quantized modules keep fp32 master weights and apply fake quantization with a straight-through estimator, so gradients reach the masters and the existing trainers work unchanged: EMA, AMP, checkpoint resume and the distillation arguments all compose.

QAT is a finetune of an already-trained model. Use finetune learning rates rather than the from-scratch defaults, or a short run will destroy the pretrained weights regardless of quantization. QAD availability follows family distillation support, which today means yolo9 and rfdetr.

fp16- and bf16-quantized models are inference only, and the trainer rejects them with a pointer to amp=True.

Export

from libreyolo import LibreYOLO qmodel = LibreYOLO("LibreYOLO9s-int8.pt") # Writes LibreYOLO9s-int8-final.pt: packed low-bit weights and scales,# fp32 masters stripped, the non-quantized remainder cast to fp16.qmodel.export(format="pt") # remainder="fp32" keeps the non-quantized tensors exact.qmodel.export(format="pt", remainder="fp32")

from libreyolo import LibreYOLO qmodel = LibreYOLO("LibreYOLO9s-int8.pt") # In-graph QuantizeLinear/DequantizeLinear pairs carrying the model's# own calibrated or QAT-trained scales.qmodel.export(format="onnx")

libreyolo export --model LibreYOLO9s-int8.pt --format onnx

format="pt" crystallizes the model. Packed low-bit weights and scales replace the masters, and the non-quantized remainder is cast to fp16 unless remainder="fp32" is passed. The packing invariant is that unpacking reproduces the simulation bit for bit on the device you finalized on, so the finalized file scores exactly what you validated. Measured: YOLO9-s int8 goes from 29.5 MB to 9.6 MB, RF-DETR-n nvfp4 from 122 MB to 26 MB. Loading one gives an inference-ready model, and calling train() on it reconstructs the masters from the packed weights automatically.

format="onnx" applies to int8 models and emits a QDQ graph carrying the model's own calibrated or QAT-trained scales, which ONNX Runtime and TensorRT run with real INT8 kernels. This is a different path from export(format="onnx", int8=True) on a float model, where ONNX Runtime derives the scales itself.

The cast recipes need no quantized exporter at all:

from libreyolo import LibreYOLO qmodel = LibreYOLO("LibreYOLO9s-int8.pt")qmodel.dequantize() # Any float exporter now applies, at any precision it supports.qmodel.export(format="tensorrt", half=True)

Constraints

Quantized arithmetic executes in simulation, which is fake quantization computed in float32 islands even under AMP. Simulation is numerics-true, so a val() score on any device is a real claim about the quantized arithmetic. It is not a speed claim.

Two exceptions execute natively. fp16 and bf16 are ordinary casts. Finalized fp8 modules run their GEMM directly on packed E4M3 weights through torch._scaled_mm on Ada, Hopper and Blackwell class hardware, using the same calibrated activation scales as the simulation; setting LIBREYOLO_KERNELS=off restores the exact simulated path everywhere.

Deployment coverage is narrower than the recipe list. Only int8 has a deployable ONNX form here; fp8 and the sub-8-bit linear recipes execute in PyTorch and crystallize through format="pt". Requesting an ONNX export from them raises with that instruction, as does requesting any non-ONNX format from an int8 model: build downstream engines from the QDQ graph instead.

Exporting an int8 model whose activations were never calibrated logs a warning and produces a graph carrying weight quantization only.