Kosmos-2 patch14-224
Fit diagram
Read at 100%
Clear selection
Download SVG
Download PNG
Click a block to read its description, or select it with Tab and Enter.
Kosmos-2 patch14-224
Grounded entity generation. Image 224 × 224; native Transformers image-to-text projection and decoder.
LibreYOLO
Kosmos-2 patch14-224
Grounded entity generation. Image 224 × 224; native Transformers image-to-text projection and decoder.
Vision encoder
Image tensor
1 × 3 × 224 × 224
Conv2d patch embedding
3 to 1024; kernel14, stride14; no bias
Prepend learned CLS
257 × 1024 including 256 patches
Add learned absolute positions
257 positions; LayerNorm1024
Vision transformer layers
24 layers; 16 heads; head width64
Post LayerNorm on all tokens
257 × 1024
L2 normalize each token
Divide by channel-vector norm
Latent-query image projection
Linear visual features
1024 to 2048
Learned latent queries
64 × 2048
Concat features and queries
321 × 2048; source for K and V
Cross-attention projection
Q=64 latent queries; K/V=321 tokens
Image embedding sequence
64 × 2048
Embed prompt token IDs
Vocabulary65037; width2048
Replace image-token placeholders
Insert64 image embeddings
Scale by sqrt(2048); add positions
Sinusoidal text positions; sequence L × 2048
Language decoder and adapter
Causal text transformer layers
24 layers; 32 heads; head width64
Final LayerNorm
L × 2048
Tied vocabulary Linear
2048 to65037; greedy token selection
Processor decodes grounded entities
Names, spans, normalized xyxy boxes
Match entity names to classes
Exact or substring match; requested classes
Filter boxes and synthetic score1
Reject nonpositive area; cap max_det
Results.boxes
Scale x by original W and y by original H
Vision transformer layer
Input
S × 1024
LayerNorm
1024 channels
Vision attention
+
LayerNorm
1024 channels
Vision MLP
+
Output
S × 1024
S is the token count of this sublayer.
Text transformer layer
Input
S × 2048
LayerNorm
2048 channels
Text attention
+
LayerNorm
2048 channels
Text MLP
+
Output
S × 2048
S is the token count of this sublayer.
Vision attention
Q Linear
1024 to 1024
K Linear
1024 to 1024
V Linear
1024 to 1024
Q/K/V from the same normalized input.
Split into heads
16 heads × 64 channels
Q × transpose(K)
Scale1/sqrt(64)
Softmax over keys
All source tokens
Attention weights × V
Per-head weighted value sum
Concat heads
Width1024
Output Linear
1024 to 1024
Text attention with inner normalization
Q Linear
2048 to 2048
K Linear
2048 to 2048
V Linear
2048 to 2048
Q/K/V from the same normalized input.
Split into heads
32 heads × 64 channels
Q × transpose(K)
Scale1/sqrt(64)
Softmax over keys
Causal/padding mask
Attention weights × V
Per-head weighted value sum
Concat heads
Width2048
Inner attention LayerNorm
2048 channels
Output Linear
2048 to 2048
Projection attention
Q Linear
2048 to 2048
K Linear
2048 to 2048
V Linear
2048 to 2048
Q: latent queries; K/V: features + queries.
Split into heads
32 heads × 64 channels
Q × transpose(K)
Scale1/sqrt(64)
Softmax over keys
All source tokens
Attention weights × V
Per-head weighted value sum
Concat heads
Width2048
Output Linear
2048 to 2048
Vision MLP
Input
S × 1024
Linear
1024 to 4096
QuickGELU
Down Linear
4096 to 1024
Output
S × 1024
Residual addition is in the enclosing layer diagram.
S is the token count of this sublayer.
Text MLP with inner normalization
Linear
2048 to8192
GELU
Elementwise activation
Inner FFN LayerNorm
8192 channels
Linear
8192 to2048
Execution and tensor scope
One released 224-pixel configuration.
Image tokens:257 before the projector.
Image embeddings:64 after the projector.
Text length L is runtime dependent.
No language cross-attention layers.
Image features enter via placeholders.
Dropout disabled in inference.
Cached generation reuses text K/V.
QuickGELU: x × sigmoid(1.702 × x).
Source/config and meta checks only.
No pretrained output was evaluated.
Sources: pinned microsoft/kosmos-2-patch14-224 config; native Transformers 5.16.1 Kosmos2 classes; LibreYOLO output adapter.
Source: libreyolo/models/vlm/kosmos2.py; Transformers 5.16.1. Revision a4d0ecc9e17f.
libreyolo.com