Kernels

The registry

libreyolo/kernels/ is a small runtime registry of pluggable implementations. An op slot is a name such as fake_quant_fp8 or ms_deform_attn. Callers ask the registry for a slot and get back whichever registered implementation passes its predicate first, newest registration winning, falling through to the reference implementation when nothing else applies.

That structure exists so that an optional dependency is never a hard requirement. A machine without Triton, without CUDA, or without the kernels package runs the same code and produces the same numbers, only slower.

import libreyolo.kernels as kernels # Op slot to selected implementation name, or "unavailable".print(kernels.active())

# off and reference both mean the same thing, and also skip# importing the accelerated providers at all.LIBREYOLO_KERNELS=off python train.py

LIBREYOLO_HUB_KERNELS=0 python predict.py

from libreyolo import LibreYOLOfrom libreyolo.kernels.attention import set_fused_attention model = LibreYOLO("LibreSwinIRs.pt") # Returns how many attention modules switched.print(set_fused_attention(model))

import libreyolo.kernels as kernels kernels.register(    "fake_quant_fp8",    my_impl,    name="mybackend",    predicate=my_check,)

A predicate that raises is caught and warned about, never propagated, so a broken third-party implementation degrades to the portable path instead of breaking prediction.

Layout

The tree is organized by purpose first and backend second, so a slot is found by what it computes rather than by which library happens to implement it today.

The boundary between simulate and execute is whether the model is finalized, not whether it is training or deploying. The reference implementations stay in libreyolo/quant/, which defines what the numbers mean; kernels/ only makes them fast. Weight packing has no variants at all, because it is the checkpoint contract.

GEMM and attention slots have no reference implementation. A caller has to check that resolve() returned something and keep its own portable path, which is why ONNX, TensorRT and torch.export graphs always contain the portable math.

Selection overrides

LIBREYOLO_KERNELS=off or =reference forces reference implementations and short-circuits the import of the accelerated providers entirely. Any other value restricts selection to implementations registered under that name. LIBREYOLO_QUANT_KERNELS is honored as a legacy alias from when the registry lived under libreyolo/quant/, and is read only when LIBREYOLO_KERNELS is unset. Both are listed with the rest on settings.

Hub kernels

Compiled CUDA kernels published on the Hugging Face Hub load at runtime through the optional kernels package. Nothing is vendored into LibreYOLO; the artifact is fetched and cached by that package, and each provider pins an audited commit revision, so bumping a pin requires a GPU parity run before it lands.

Installing the extra is the opt-in:

pip install "libreyolo[hub-kernels]"

Without the package nothing changes and no network request is made. LIBREYOLO_HUB_KERNELS=0 disables the fetch without uninstalling anything. A kernel that fails to load or to run disables itself for the rest of the process and falls back with one warning.

One slot is Hub-backed today: ms_deform_attn, the compiled multi-scale deformable attention forward and backward from Deformable DETR, under Apache 2.0. It is wired into the whole deformable lineage: RF-DETR, Deformable DETR, DINO-DETR, LW-DETR, Grounding DINO, RT-DETR, RT-DETRv2, D-FINE, RT-DETRv4, DEIM, DEIMv2, EC and OV-DEIM. Because the backward is compiled too, training benefits as well as prediction.

Eligibility is narrow on purpose. Inputs must be CUDA and float32, and execution must be eager: the provider declines under torch.jit.is_tracing(), torch.compiler.is_compiling(), torch.compiler.is_exporting() and torch.onnx.is_in_onnx_export(). Two input layouts also fall through to the portable path, a per-level point count that varies between levels, and discrete integer-index sampling. The EC pose variant is not wired.

This kernel is newly reachable

Read this before installing the extra on an existing project.

In v1.4.0 the slot was consulted from inside a helper, behind a condition that required the spatial-shape pairs to be absent. RF-DETR always threads those pairs through its decoder, so the condition never held and the kernel never executed in any eager forward. The consult moved in v1.5.0, and the kernel now actually runs.

The practical consequence is that upgrading to v1.5.0 and installing libreyolo[hub-kernels] on CUDA means RF-DETR and its lineage take their forward from a compiled binary for the first time. Predictions and metrics can shift at float tolerance as a result. A stock install, without the extra, is unaffected. If you are comparing metrics across the upgrade, hold the extra fixed or set LIBREYOLO_HUB_KERNELS=0 on both sides.

Fused attention

Fused scaled dot-product attention needs no optional dependency, only stock PyTorch, so it is governed by policy rather than by availability. Two rules apply.

First, a graph capture never uses it. Every swapped call site keeps the primitive-op equation available behind an export check, covering ONNX export, whose default opset has no SDPA symbolic, and torch.jit.trace, which TorchScript, CoreML and NCNN all go through. Dynamo captures are deliberately outside the gate, because torch.compile lowers SDPA better than the manual math, and both Core AI and ExecuTorch decompose SDPA to core ATen on their own.

Second, the parity bar for making it the default is byte exact. Families that clear it use SDPA by default: SegFormer, Depth Anything and MoGe-2, BERT, Grounding DINO, SwinIR and PP-OCR. Families that do not keep manual math and expose a fused_attn flag instead, which is what set_fused_attention(model) flips: Swin, DINO-DETR's Swin backbone, BiRefNet and FeyNobg, OWLv2, LW-DETR, SigLIP 2, ZipDepth and MobileSAM. ViT and DeiT carry the same flag but default it on, following upstream, so the same call with enabled=False turns them off.

It is worth doing where it applies. On an RTX 5070 Ti under fp16 autocast, Swin window attention goes from 1.278 ms to 0.721 ms, a 1.77x gain, and OWLv2 vision attention from 6.483 ms to 1.735 ms, 3.74x.

Hardware

Adding an implementation

Call register() with a name and a predicate. Out-of-tree compiled kernels can ship as a separate libreyolo_kernels package that registers itself on import, which keeps a private backend out of the LibreYOLO tree entirely.

Parity is the gate for anything in-tree: an exact forward match against the reference, and gradients within 1e-6 of the straight-through estimator, over the shape set the test suite carries.

Kernel selection interacts with CUDA graphs: the inference parity matrix ran without the kernels package installed, so capture safety with a compiled kernel active is not covered by it.