SAM base
Fit diagram
Read at 100%
Clear selection
Download SVG
Download PNG
Click a block to read its description, or select it with Tab and Enter.
SAM base
Promptable segmentation, 1024 × 1024 input, one box, multimask output. Image memory and video tracking are outside this graph.
LibreYOLO
SAM base
Promptable segmentation, 1024 × 1024 input, one box, multimask output. Image memory and video tracking are outside this graph.
Image encoder
Input after resize/pad
3 × 1,024 × 1,024
Conv2d 16×16 / 16
64 × 64 × 768, NHWC
Add learned absolute position
64 × 64 × 768
Window/global ViT blocks
64 × 64 × 768, repeats=12
Permute to NCHW
768 × 64 × 64
Conv2d 1×1
768 to 256, bias=False
Channel LayerNorm
256 × 64 × 64
Conv2d 3×3 / 1
256 to 256, p=1, bias=False
Channel LayerNorm
256 × 64 × 64
Global blocks (1-based): 3, 6, 9, 12
Other blocks use 14 × 14 windows.
No CLS token in the image encoder.
Box prompt encoder
One box
2 corner coordinates × 2
Shift pixel centers
Add 0.5 to x and y
Normalize coordinates
Divide by 1024, map to [-1,1]
MatMul random Fourier matrix
2 coords to 128 frequencies
Multiply by 2 pi
2 × 128
Sin
2 × 128
Cos
2 × 128
Concat sin and cos
2 × 256
Corner type
2 learned × 256
+
Sparse prompt embeddings
2 × 256
Learned no-mask embedding
Broadcast to 256 × 64 × 64
Selected input: one box, no mask prompt.
Points use the same Fourier encoding, plus
positive/negative or not-a-point embeddings.
Dense image position: this encoding evaluated
on the centers of the 64 × 64 feature grid.
Mask decoder inputs
Image embedding
256 × 64 × 64
Dense prompt
256 × 64 × 64
+
Flatten image features
4,096 × 256
Learned output tokens
IoU + 4 masks = 5 × 256
Sparse prompts
2 × 256
Concat output and prompt tokens
7 × 256
Two-way block 1
7 queries, 4,096 image tokens
Two-way block 2
7 queries, 4,096 image tokens
Final token-to-image attention
Q+Q0, K+P, V=updated image
+
Final LayerNorm on queries
7 × 256
Q0 is the original token sequence.
P is fixed Fourier image position.
4,096 image tokens feed block 1 keys/values.
Masks and quality outputs
Updated query tokens
1 IoU + 4 mask tokens
Updated image tokens
4,096 × 256
Reshape to image
256 × 64 × 64
ConvTranspose2d 2×2 / 2
256 to 64; 64 × 128 × 128
Channel LayerNorm
64 × 128 × 128
GELU
64 × 128 × 128
ConvTranspose2d 2×2 / 2
64 to 32; 32 × 256 × 256
GELU
32 × 256 × 256
Select four mask tokens
4 × 256
Four independent Linear layers
Each 256 to 256
ReLU
4 × 256
Four independent Linear layers
Each 256 to 256
ReLU
4 × 256
Four independent Linear layers
Each 256 to 32
MatMul mask coefficients × upscaled image
4 × 32 times 32 × 65,536; reshape to 4 × 256 × 256
Select masks 1 to 3
3 × 256 × 256
IoU token Linear
256 to 256
ReLU
256
Linear
256 to 256
ReLU
256
Linear
256 to 4 scores
Select scores 1 to 3
3 quality scores
Mask logits are resized to the input canvas,
then unpadded/resized to the original image.
multimask_output=True is drawn.
False selects mask/score 0 instead.
Two-way block 1
Query state
7 × 256
Image state
4,096 × 256
Token self-attention
8 heads, 32 channels per head
First self-attention has no PE or residual.
LayerNorm
7 × 256
Original tokens Q0
7 × 256
+
Image position P
4,096 × 256
+
Token-to-image attention
Q=7, K/V=4,096; 8 heads × 16
+
LayerNorm
7 × 256
Linear
256 to 2,048
ReLU
7 × 2,048
Linear
2,048 to 256
+
LayerNorm
7 × 256
Original tokens Q0
7 × 256
+
Image-to-token attention
Q=4,096, K/V=7; 8 heads × 16
+
LayerNorm on image state
4,096 × 256
Updated queries: 7 × 256
Two-way block 2
Query state
7 × 256
Image state
4,096 × 256
Token self-attention
8 heads, 32 channels per head
Original tokens Q0
7 × 256
+
Self-attention Q/K include Q0.
+
LayerNorm
7 × 256
Original tokens Q0
7 × 256
+
Image position P
4,096 × 256
+
Token-to-image attention
Q=7, K/V=4,096; 8 heads × 16
+
LayerNorm
7 × 256
Linear
256 to 2,048
ReLU
7 × 2,048
Linear
2,048 to 256
+
LayerNorm
7 × 256
Original tokens Q0
7 × 256
+
Image-to-token attention
Q=4,096, K/V=7; 8 heads × 16
+
LayerNorm on image state
4,096 × 256
Updated queries: 7 × 256
Token self-attention
Query input
7 × 256
Key/value input
7 × 256
Linear Q
256 to 256
Reshape heads
8 × 7 × 32
Linear K
256 to 256
Reshape heads
8 × 7 × 32
Linear V
256 to 256
Reshape heads
8 × 7 × 32
MatMul Q K-transpose
8 × 7 × 7
Scale
Divide by sqrt(32)
Softmax over keys
8 × 7 × 7
MatMul attention × V
8 × 7 × 32
Merge heads
7 × 256
Linear output
256 to 256
Token-to-image attention
Query input
7 × 256
Key/value input
4096 × 256
Linear Q
256 to 128
Reshape heads
8 × 7 × 16
Linear K
256 to 128
Reshape heads
8 × 4096 × 16
Linear V
256 to 128
Reshape heads
8 × 4096 × 16
MatMul Q K-transpose
8 × 7 × 4096
Scale
Divide by sqrt(16)
Softmax over keys
8 × 7 × 4096
MatMul attention × V
8 × 7 × 16
Merge heads
7 × 128
Linear output
128 to 256
Image-to-token attention
Query input
4096 × 256
Key/value input
7 × 256
Linear Q
256 to 128
Reshape heads
8 × 4096 × 16
Linear K
256 to 128
Reshape heads
8 × 7 × 16
Linear V
256 to 128
Reshape heads
8 × 7 × 16
MatMul Q K-transpose
8 × 4096 × 7
Scale
Divide by sqrt(16)
Softmax over keys
8 × 4096 × 7
MatMul attention × V
8 × 4096 × 16
Merge heads
4096 × 128
Linear output
128 to 256
Decoder conventions
Q/K add positional terms; V is the raw state.
Final attention repeats token-to-image attention.
Its output is added to queries, then normalized.
Four mask tokens have independent MLP weights.
IoU token uses a separate three-layer MLP.
All decoder Linear projections include bias.
Prompt input for this view is one box.
Dense prompt is the learned no-mask embedding.
Point prompts use Fourier position with a
positive, negative or padding type embedding.
Encode once and cache the image features;
new prompts rerun prompt encoder and decoder.
Image ViT block
Input: 64 × 64 × 768
LayerNorm
64 × 64 × 768
Window partition when local
Pad 64 to 70; 25 windows of 14 × 14
Image self-attention
12 heads; local 196 / global 4,096 tokens
Reverse windows and crop
64 × 64 spatial grid
+
LayerNorm
64 × 64 × 768
Linear
768 to 3072
GELU
64 × 64 × 3072
Linear
3072 to 768
+
Local image attention
Query input
196 × 768
Key/value input
196 × 768
Linear Q
768 to 768
Reshape heads
12 × 196 × 64
Linear K
768 to 768
Reshape heads
12 × 196 × 64
Linear V
768 to 768
Reshape heads
12 × 196 × 64
MatMul Q K-transpose
12 × 196 × 196
Scale
Divide by sqrt(64)
Position bias
12 × 196 × 196
+
Softmax over keys
12 × 196 × 196
MatMul attention × V
12 × 196 × 64
Merge heads
196 × 768
Linear output
768 to 768
Global image attention
Query input
4096 × 768
Key/value input
4096 × 768
Linear Q
768 to 768
Reshape heads
12 × 4096 × 64
Linear K
768 to 768
Reshape heads
12 × 4096 × 64
Linear V
768 to 768
Reshape heads
12 × 4096 × 64
MatMul Q K-transpose
12 × 4096 × 4096
Scale
Divide by sqrt(64)
Position bias
12 × 4096 × 4096
+
Softmax over keys
12 × 4096 × 4096
MatMul attention × V
12 × 4096 × 64
Merge heads
4096 × 768
Linear output
768 to 768
Decomposed relative position
Query spatial grid
Head width 64
Relative H table
27 / 127 distances × 64
Gather offsets
14×14 / 64×64 coordinate pairs
Einsum query × H
Query-dependent axis bias
Relative W table
27 / 127 distances × 64
Gather offsets
14×14 / 64×64 coordinate pairs
Einsum query × W
Query-dependent axis bias
+
Broadcast axis biases before addition.
Per-head bias: 196×196 locally;
4,096×4,096 in global blocks.
Variant E n h M
base: 768, 12, 12, 3072
large: 1024, 24, 16, 4096
huge: 1280, 32, 16, 5120
Global indices (1-based):
base: 3, 6, 9, 12
large: 6, 12, 18, 24
huge: 8, 16, 24, 32
Source: libreyolo/models/sam/model.py. Revision a4d0ecc9e17f.
libreyolo.com