SmolVLM2 500M
Fit diagram
Read at 100%
Clear selection
Download SVG
Download PNG
Click a block to read its description, or select it with Tab and Enter.
SmolVLM2 500M
Detect. P = image patches; L = combined tokens (runtime axes). Model widths below are resolved.
LibreYOLO
SmolVLM2 500M
Detect. P = image patches; L = combined tokens (runtime axes). Model widths below are resolved.
Image encoder
Image processor
Nominal runner size 512; native tiling/resize
Conv2d patch embedding
patch=16; output P × 768
Learned/interpolated 2D positions
P × 768
Vision transformer blocks
n=12; width 768; heads=12
Final LayerNorm
768 channels
V
V carries final visual features to the projector.
Patch counts vary with processor output, not model size.
Projection and multimodal sequence
V
Visual projector
768 channels become 960 channels
Visual token sequence
Patch grouping factor 4 per axis
Prompt and image placeholders
Vocabulary and detection request
Tokenizer
Runtime text length
Token embedding
Vocabulary 49280; width 960
Replace image-token placeholders
L × 960; image and text in one sequence
S
Visual features replace placeholders before language layers.
Text and image masks preserve their positions.
S denotes the multimodal token sequence.
Language model and generated output
S
Text decoder layers
n=32; width 960
Final RMSNorm
960 channels
Language head Linear
960 to 49280
Greedy token generation
max_new_tokens=1024; do_sample=False
O
O is the generated reply; prompt tokens are removed.
The adapter decodes O into task geometry below.
Vision transformer layer
Input
S × 768
LayerNorm
768 channels
Vision attention
+
LayerNorm
768 channels
Vision MLP
+
Output
S × 768
S is the token count of this sublayer.
Text decoder layer
Input
S × 960
RMSNorm
960 channels
Causal attention
+
RMSNorm
960 channels
Gated MLP
+
Output
S × 960
S is the token count of this sublayer.
Vision attention
Input width 768; Q heads 12; K/V heads 12.
Q Linear
768 to 768
Split heads
64 per head
K Linear
768 to 768
Split heads
64 per head
V Linear
768 to 768
Split heads
64 per head
Q × transpose(K)
Scale by 1/sqrt(64)
Softmax over keys
Within the image; padding mask
Attention weights × V
12 query heads, weighted values
Concat heads
Output Linear
768 to 768
Position information enters through embeddings.
Distinct projections share the same normalized input.
Text attention
Input width 960; Q heads 15; K/V heads 5.
Q Linear
960 to 960
Split heads
64 per head
K Linear
960 to 320
Split heads
64 per head
V Linear
960 to 320
Split heads
64 per head
Rotary position
cos/sin half rotation
Rotary position
cos/sin half rotation
Align K heads
5 to 15
Align V heads
5 to 15
Q × transpose(K)
Scale by 1/sqrt(64)
Softmax over keys
Causal mask; cached K/V optional
Attention weights × V
15 query heads, weighted values
Concat heads
Output Linear
960 to 960
RoPE: x*cos + rotate_half(x)*sin.
Cached decode appends past K/V before head alignment.
Vision MLP
Input
S × 768
Linear
768 to 3072
gelu_pytorch_tanh
Down Linear
3072 to 768
Output
S × 768
Residual addition is in the enclosing layer diagram.
S is the token count of this sublayer.
Text MLP
Input
S × 960
Gate Linear
960 to 2560
Up Linear
960 to 2560
SiLU
Elementwise multiply
S × 2560
Down Linear
2560 to 960
Output
S × 960
Residual addition is in the enclosing layer diagram.
S is the token count of this sublayer.
Visual projector
Pixel shuffle / spatial grouping
4 × 4; width 12288
Linear without bias
12288 to 960
No connector activation or hidden layer.
Output width equals the language-model embedding.
LibreYOLO output adapter
O
Decode generated tokens
Retain reply text
Parse JSON boxes
Resolve label and box-key aliases
Normalize coordinates
Divide by 1; xyxy corners
Validate / class / confidence
Synthetic per-box score1; clamp valid boxes
Same-class deduplication
Rounded boxes and IoU filter before max_det
Results.boxes
Scale xyxy to original W,H; cap max_det
Scores are adapter placeholders, not calibrated confidence.
Scope: native backend architecture from pinned config metadata; no pretrained execution. The image path is shown.
Source: libreyolo/models/vlm/smolvlm.py; Transformers 5.16.1. Revision a4d0ecc9e17f.
libreyolo.com