TensorRT

Install

Both the build and the run need an NVIDIA GPU with a working CUDA stack. There is no CPU fallback for this format.

# The engine is built from an ONNX intermediate, so both extras are needed.pip install "libreyolo[onnx,tensorrt]"

python -c "import tensorrt, torch; print(tensorrt.__version__, torch.cuda.is_available())"

The tensorrt extra pins tensorrt-cu12 and pycuda, and the marker drops both on macOS. On a Jetson, do not use that extra: it pins a CUDA 12 build against a CUDA 13 platform. Use the TensorRT that JetPack installs instead, as described on NVIDIA Jetson.

Export

from libreyolo import LibreYOLO model = LibreYOLO("LibreYOLO9t.pt") # Writes weights/LibreYOLO9t_fp16.engine and weights/LibreYOLO9t_fp16.engine.jsonpath = model.export(format="tensorrt", half=True)print(path)

libreyolo export --model LibreYOLO9t.pt --format tensorrt --half

model.export(    format="tensorrt",    imgsz=640,    batch=1,    half=False,    int8=False,    data=None,                      # required when int8=True    dynamic=False,    workspace=4.0,                  # GiB of build-time scratch    min_batch=1,                    # dynamic profile bounds    opt_batch=1,    max_batch=8,    hardware_compatibility="none",  # or "ampere_plus"    gpu_device=0,                   # build device on a multi-GPU host    verbose=False,)

The export runs in two steps. Step one writes an ONNX intermediate to a temporary path, step two parses it and builds the engine, and the intermediate is removed afterwards. workspace is build-time scratch memory in GiB; a larger value lets the builder try more kernels and does not affect inference memory.

The metadata sidecar is written next to the engine as <engine>.json and records the precision the build actually realized. When the GPU lacks fast FP16 or fast INT8 the builder warns and falls back, and the sidecar reports the precision that came out rather than the one that was asked for.

Under FP16, a ViT backbone in the graph is detected and its float layers are pinned to FP32. DINOv2-style backbones overflow in FP16 and produce NaN, so the build sets OBEY_PRECISION_CONSTRAINTS and reports FP16 (FP32 ViT backbone). The pass is a no-op on CNN backbones.

Dynamic batch

from libreyolo import LibreYOLO # The ONNX intermediate needs the dynamic batch axis for the profile# to have anything to bind to.LibreYOLO("LibreYOLO9t.pt").export(    format="tensorrt",    dynamic=True,    min_batch=1,    opt_batch=4,    max_batch=8,    half=True,)

dynamic=True adds one optimization profile spanning min_batch to max_batch, optimized at opt_batch, and records those three values in the sidecar. The profile is only added when the ONNX intermediate actually carries a dynamic batch dimension; otherwise the build logs that it is using static optimization and continues.

INT8

from libreyolo import LibreYOLO LibreYOLO("LibreYOLO9t.pt").export(    format="tensorrt",    int8=True,    data="coco128.yaml",   # required: there is no default for this format    fraction=1.0,)

INT8 uses TensorRT's entropy calibrator over a LibreYOLO calibration loader, and data is mandatory: this format has no eight-image fallback. Calibration needs cuda-python or pycuda for the device buffer. The calibration cache is keyed on a hash of the ONNX bytes, so scales from one model are never reused for another that happens to write to the same output path.

half=True and int8=True together warn and build INT8, which keeps an FP16 fallback for layers TensorRT cannot quantize.

Run the artifact

from libreyolo import LibreYOLO, SAMPLE_IMAGE model = LibreYOLO("weights/LibreYOLO9t_fp16.engine")result = model.predict(SAMPLE_IMAGE)print(result.boxes.xyxy[:3])

import json import tensorrt as trt path = "weights/LibreYOLO9t_fp16.engine"runtime = trt.Runtime(trt.Logger(trt.Logger.WARNING))with open(path, "rb") as handle:    engine = runtime.deserialize_cuda_engine(handle.read()) for i in range(engine.num_io_tensors):    name = engine.get_tensor_name(i)    print(engine.get_tensor_mode(name), name, engine.get_tensor_shape(name)) # Class names, task and input size live in the sidecar, not the engine.# Buffer allocation, preprocessing and postprocessing are yours here.print(json.load(open(path + ".json"))["names"])

LibreYOLO() dispatches on the .engine suffix, reads the sidecar for class names, task and pose schema, and returns the same Results object as the checkpoint. It raises immediately when no CUDA device is present.

The second snippet is the bare-runtime path. Host and device buffer allocation, preprocessing, decoding, NMS and coordinate rescaling all become yours, and the engine itself carries no class names, so the sidecar has to travel with it.

Constraints

A serialized engine is tied to the GPU architecture, the driver stack and the TensorRT version that built it. An engine built on a workstation will not load on a different architecture, which is why the build step runs on the deployment machine. hardware_compatibility="ampere_plus" trades some performance for portability across Ampere and newer. The "same_compute_capability" value maps to NONE and warns: the engine is optimized for the current GPU only, and the export says so rather than claiming a portability it did not apply.

Only the batch axis is profiled. A build with dynamic spatial dimensions is not part of this contract, which is why FCOS is blocked: it needs dynamic padded height and width to preserve its 800 by 1333 aspect transform.

Blocked before tracing: YOLO9 segmentation, RTMDet-Ins segmentation, SSD, Faster R-CNN and RetinaNet detection, and BiRefNet or FeyNobg matting, where TensorRT 10.16 reaches the shared ONNX DeformConv node and cannot parse it because ModulatedDeformConv2d is absent from the plugin registry.

Where a combination is neither validated nor blocked, the converter path is available and the project has not recorded TensorRT runtime parity for it. That is a statement about evidence, not about whether the build succeeds.

For the full family and task grid, see the export matrix. For one combination:

libreyolo formats --family yolo9 --task detect