Ver como markdown

Optical flow

Optical flow predicts, for every pixel of one frame, where it moves in the next frame. LibreYOLO exposes it as the flow task, new in 1.8.0: a pair of frames in, one dense field of pixel displacements out, with eight families behind the same call.

Definition

The flow task takes a sequence of frames and predicts, for each consecutive pair, the displacement of every pixel from frame t to frame t + 1. Its aliases optical-flow, optical_flow and opticalflow resolve to it, and checkpoints carry the -flow suffix.

A prediction fills result.flow, a FlowField holding an (H, W, 2) float32 array on the original canvas of the first frame. flow.data[y, x] is (dx, dy) in pixels, +x to the right and +y downwards. .dx, .dy, .magnitude, .angle, .max_magnitude and .mean_magnitude read it, and .warp(image) warps an image back with the field. flow.confidence is an optional (H, W) map in [0, 1], filled only by families that predict one (SEA-RAFT, from its uncertainty head); the others leave it None. flow.next_frame holds the second frame for the warp view. result.boxes is None.

Models

Eight families serve flow. None has a model page yet; load them by checkpoint name, as in the snippets below.

FamilySizesWeights licence
SEA-RAFTs, m, lBSD-3-Clause
RAFTs, l, plus Sintel and KITTI variantsBSD-3-Clause
NeuFlow v2s, plus Sintel and Things variantsApache-2.0
WAFTt, s, m, lBSD-3-Clause; t uses DINOv3 and carries the DINOv3 License
FlowSeekt, s, m, lApache-2.0 for t and s; m and l are non-commercial (CC BY-NC, Depth Anything V2 encoder)
DPFlowm, plus Sintel, KITTI and Spring variantsNon-commercial (CC BY-NC-SA 4.0)
UniMatchs, l, plus Sintel and KITTI variantsMIT
MEMFOFm, plus five variantsBSD-3-Clause

Checkpoints follow Libre<Family><size>-flow.pt, for example LibreSEARAFTm-flow.pt, LibreRAFTl-flow-kitti.pt or LibreNeuFlowv2s-flow.pt. The non-commercial weights print a notice when they load. NeuFlow v2 is the small edge model. MEMFOF reads three frames per pair, and the runner repeats the edge frames at the start and end of a sequence. All eight have recorded parity against their upstream code, and their weights are mirrored on the LibreYOLO Hugging Face organization.

Predict

Two frames
from libreyolo import LibreYOLO model = LibreYOLO("LibreSEARAFTm-flow.pt")results = model(["frame_0001.jpg", "frame_0002.jpg"]) flow = results[0].flowprint(flow.data.shape)           # (H, W, 2) float32, (dx, dy) in pixelsprint(flow.mean_magnitude)       # mean displacement in pixels
A video or a frame list
from libreyolo import LibreYOLO model = LibreYOLO("LibreNeuFlowv2s-flow.pt") # N frames give N - 1 results; result i is the flow from frame i to i + 1.for result in model("video.mp4", stream=True, vid_stride=2):    print(result.flow.max_magnitude)
Views and raw fields
from libreyolo import LibreYOLO model = LibreYOLO("LibreRAFTl-flow.pt")frames = ["frame_0001.jpg", "frame_0002.jpg", "frame_0003.jpg"] model(frames, save=True)                    # colour wheel (default)model(frames, save=True, arrows=True)       # arrows over the framemodel(frames, save=True, warp=True)         # warp checkresults = model(frames, save=True, save_flow=True)   # also writes .flo results[0].flow.save("pair0.npy")           # .flo or .npy

A single image raises: flow needs at least two frames. A list or tuple of images, a directory sorted by name, a video or a stream is one sequence, and N frames give N - 1 results. Result i carries the flow from frame i to frame i + 1, with frame i as its orig_img. On videos, vid_stride sets the frame gap, and stream=True yields one result per pair.

There are three views. The default is the Middlebury colour wheel: hue is direction, saturation is magnitude, and zero motion is white. arrows=True draws the frame with one arrow every 16 pixels. warp=True shows frame t next to frame t + 1 warped back onto it, so a correct field makes both panels match. save=True writes the chosen view, and save_flow=True also writes the raw field of every pair as a Middlebury .flo file named <frame stem>_<frame index>.flo. FlowField.save(path) writes .flo or .npy directly.

From the shell, pass the frames as a comma-separated source, and add arrows=true, warp=true or save_flow=true:

CLI
libreyolo predict model=LibreSEARAFTm-flow.pt source=frame_0001.jpg,frame_0002.jpg save=truelibreyolo predict model=LibreSEARAFTm-flow.pt source=video.mp4 save=true arrows=true save_flow=truelibreyolo val model=LibreSEARAFTm-flow.pt data=my_flow.yaml

See prediction for sources and streaming, and libreyolo predict for every CLI flag.

Dataset format

A flow dataset is a set of frame sequences. Frames live under images/<split>/<sequence>/ and are ordered by file name; each consecutive pair inside one sequence directory is a sample, whose ground truth sits under flows/ with the same relative path and the first frame's stem.

dataset/  data.yaml  images/    val/alley_1/frame_0001.png    val/alley_1/frame_0002.png    val/alley_1/frame_0003.png  flows/    val/alley_1/frame_0001.flo    val/alley_1/frame_0002.flo
yaml
path: datasetval: images/valflows_dir: flowsflow_format: flo

flow_format is flo (Middlebury, the default), kitti (16-bit RGB PNG with the blue channel as the validity mask) or npy (an (H, W, 2) float array in which non-finite values are invalid). An optional per-pair mask <stem>_valid.png marks valid pixels as nonzero; flow_valid_suffix changes the suffix. The last frame of a sequence has no flow file, and frames in different directories never pair.

Train

Flow ships inference only. train() on a flow family raises NotImplementedError.

Validate

val() compares predicted and ground-truth vectors directly on the valid pixels, with no alignment.

Validate and read the metric keys
from libreyolo import LibreYOLO model = LibreYOLO("LibreSEARAFTm-flow.pt")metrics = model.val(data="my_flow.yaml") print(metrics["metrics/epe"])      # mean end-point error, pixelsprint(metrics["metrics/px1"])      # share of pixels under 1 px errorprint(metrics["metrics/px3"])print(metrics["metrics/fl_all"])   # KITTI outlier rate

metrics/epe is the mean end-point error, the Euclidean distance between the predicted and true vectors in pixels, and lower is better. metrics/px1 and metrics/px3 are the fraction of valid pixels whose error is under 1 and 3 pixels. metrics/fl_all is the KITTI outlier rate: pixels whose error exceeds both 3 pixels and 5 percent of the true magnitude. Best-checkpoint fitness is metrics/px1, because it increases with quality.

Export

Flow export is blocked for every format until the two-input graph and the backend rescale contract are implemented for a family. See the export matrix.

Verificado com o LibreYOLO v1.8.0.