Training performance

Measure before changing anything

The three levers below fix different problems, and applying the wrong one changes nothing. The profiler says which problem you have.

from libreyolo import LibreYOLO model = LibreYOLO("LibreYOLO9s.pt") # Profiles a short window of real steps, prints a verdict, then# continues the run with the hooks removed.model.train(data="my-dataset.yaml", epochs=100, profile=True)

# Sets no_aug_epochs=0 and runs just enough epochs to fill the window.libreyolo profile run coco128 --weights LibreYOLO9s.pt --size s

libreyolo profile summary runs/profile/prof/profile.jsonlibreyolo profile phases runs/profile/prof/profile.jsonlibreyolo profile kernels runs/profile/prof/profile.json --top 10

profile=True measures a window of real training steps, five discarded then twenty measured by default, prints a report, writes its artifacts and then keeps training with the hooks dropped. It costs nothing when off, and it is ignored under distributed training.

The report ends in one of four verdicts:

The utilization number is kernel busy time over the unsynchronized step time. The window is deliberately split: the first half runs with no extra synchronization so the verdict reflects real overlap, and only the second half brackets each phase with a sync to attribute GPU time. Synchronizing every phase hands the dataloader workers slack and hides starvation, so the composition numbers are never used to pick the verdict.

Four files land in the run directory: timeline.html, which opens in a browser by itself, profile_trace.json for Perfetto or Nsight, profile_summary.json, and profile.json, the self-contained one to copy around and feed back to the libreyolo profile subcommands.

Two things about profile run are worth knowing. It sets no_aug_epochs=0, because the profiler measures epoch 0 and a short run with the default no_aug_epochs would profile the lighter no-augmentation dataloader rather than the one training actually uses. And --repeat N reports mean and standard deviation, which matters because a launch-bound step is noisy enough that a single run misleads; it writes per-trial directories prof_1, prof_2 and so on, plus an aggregate profile_repeat.json.

Mixed precision

amp=True is the default for most families and runs the forward pass under CUDA autocast. amp_dtype chooses float16 or bfloat16.

from libreyolo import LibreYOLO model = LibreYOLO("LibreYOLO9s.pt")model.train(data="my-dataset.yaml", amp=True, amp_dtype="bfloat16")

libreyolo train model=LibreYOLO9s.pt data=my-dataset.yaml \  amp_dtype=bfloat16

Float16 needs dynamic loss scaling and gets a live gradient scaler; bfloat16's wider exponent range does not, so its scaler is disabled. Four families ship with amp=False, D-FINE, DEIM, YOLO-NAS and FOMO, and the DEIM setting carries through to RT-DETRv4 by inheritance. D-FINE states the reason: its decoder clamps activations at 65504, the largest finite float16 value.

The argument semantics, including what a bfloat16 request does on hardware without bfloat16 support, are on Hyperparameters.

CUDA graphs

cuda_graph=True captures the network's training forward and backward into a CUDA graph, removing per-step kernel launch overhead.

from libreyolo import LibreYOLO model = LibreYOLO("LibreYOLO9s.pt")model.train(data="my-dataset.yaml", epochs=100, cuda_graph=True)

libreyolo train model=LibreYOLO9s.pt data=my-dataset.yaml \  epochs=100 cuda_graph=true

The flag is always safe to pass. A family, task or configuration that cannot be captured logs one line and trains eager, unchanged.

Only the network is captured. The loss stays eager by design, because detection losses select with boolean masks, run Hungarian matching and branch on assignment results, none of which a graph can record. The optimizer step, gradient clipping, EMA update and learning-rate schedule stay eager too.

That bounds the win by how much of a step is network, and the share varies widely. Measured on an RTX 5070 Ti at 640 px, batch 8: 84 percent of a YOLOv9-t step is network, 44 percent of a YOLOv7-b step, 31 percent of a YOLOX-t step and 26 percent of an RTMDet-t step. The last two spend most of a step inside their label assigners, so capturing the network helps them least.

What it is worth

Conditions for every figure below: RTX 5070 Ti, Windows, AMP, one process per arm from a shared saved state, replaying one real batch so the dataloader is out of the loop, fastest of 24 steps after warm-up. Detection at 640 px, classification at 224 px. Batch size is per row.

Those isolate the GPU step. A complete fine-tune also pays for the dataloader and for validation. YOLOv9-t on a 406-image detection set, 20 epochs, batch 8, 640 px, 4 dataloader workers, on the same machine: 428.4 s wall clock eager against 367.7 s graphed, a 1.16x gain, with mAP50-95 of 0.6394 in both arms.

Three things move these numbers. Small batches are launch-bound and large ones are compute-bound, so RT-DETR-r18 gains 1.19x at batch 2 and 1.04x at batch 8. Launch overhead is highest on Windows, and Linux gains are roughly a third to half of the table. And a dataloader-bound run sees no wall-clock change at all, which is why the profiler comes first.

Capture engages the same way at amp=False, but fp32 kernels run longer, so a step is less launch-bound and most families gain less. On the same hardware, MobileNetV4-s at batch 16 goes from 2.74x under AMP to 3.61x at fp32, while YOLOv9-t at batch 8 goes from 1.99x to 1.69x and RT-DETR-r18 at batch 4 from 1.12x to 0.99x.

Where capture applies

Everything else falls back to eager with one log line: other tasks on those families, families not listed, distributed runs and distillation runs. A capture failure at runtime also drops the rest of the run to eager rather than failing.

For the encoder-decoder detectors, D-FINE, DEIM, DEIMv2, RT-DETR v1, v2 and v4, and EC, only the backbone and encoder are captured. Their decoder reads the ground truth to build contrastive-denoising queries, and the number of those queries follows the largest ground-truth count in the batch, so its token count changes from batch to batch.

Shapes

A graph is valid for exactly the input shape it was captured with. The trainer counts batch shapes and captures once a shape has repeated three times. Batches at any other shape run eager: multi-scale batches, and the last partial batch of an epoch.

This is the trap for the DETR families, which resize every batch by default. With multi_scale=True a short run may never see one shape often enough to capture at all. Pass multi_scale=False when the speedup is the point.

YOLOX changes what the captured region computes partway through a run, turning on its L1 regression branch when mosaic closes at no_aug_epochs. The trainer invalidates the capture there and re-captures once the new shape settles.

Numerics and memory

Most families reproduce their eager loss trajectory bit for bit under AMP. FOMO and LingBot-Vision differ in the last bit of float32 from a different summation order. The deformable-attention detectors, D-FINE, DEIM, DEIMv2, RT-DETR, RF-DETR and EC, do not reproduce their own eager runs either, because that backward accumulates with atomics and TF32 convolutions pick a reduction order per launch; the graphed run stays inside that spread. RTMDet differs by roughly 3e-4 relative on two of 139 gradients, because it shares head convolutions across pyramid levels and the two backward paths sum three contributions in a different order. SegFormer has stochastic depth inside the captured region, so a replayed graph draws its own random stream and is statistically equivalent to eager rather than identical; the manager logs that once at capture time.

At amp=False bit-identical is not available from anything on this hardware, with or without capture. Two identical seeded eager YOLOv9-t runs diverge by 36 percent relative over 20 steps and YOLOX-t by 2.6 percent, because cuDNN picks a nondeterministic weight-gradient algorithm for some fp32 convolution shapes.

A captured graph pins static input, output and workspace buffers, so peak VRAM rises by roughly one extra set of activations. Across the families above, peak allocation moved between -5 and +19 percent. The relative cost is largest for the small classification models, whose activations are small to begin with: ResNet-18 at 224 px, batch 16, went from 0.48 GB eager to 0.57 GB graphed. If it pushes a run over the limit, lower the batch or leave the flag off.

Related

• Hyperparameters for batch, nbs, cache and workers.
• Multi-GPU training, where both CUDA graphs and the profiler are unavailable.