29 The modern transformer
RoPE: position as rotation
Rotary position embeddings stamp a token's position onto its query and key vectors by rotating them — spinning each 2-D pair like a clock hand by an angle proportional to the position. The magic: because a dot product only cares about the angle between two arrows, rotating both by their own positions makes the attention score depend only on the gap m − n. Drag the two tokens and watch it happen.
Drag the query (m) and key (n)
Shift the pair:
RoPE base θ:
query · m=4q
key · n=2k
Pair / hand
fast
1
1
0.1
0.01
slow
0.001
0.001
Query @ mrotated m·θ
Key @ nrotated n·θ
per-hand scorecos((m−n)·θ)
gap m − n
2
attention score q@m · k@n
0.000
absolute m , n
4 , 2
Turn on Lock the gap, then drag — the absolute positions change but the gap (and the score) stay frozen. That is the whole point of RoPE.
What to notice
- Each token gets four clock hands at different speeds: θ = base−2i/d. The fast hand (θ=1 rad/token) spins a full circle every ~6 tokens; the slow hand barely creeps — the "second hand vs hour hand" of position.
- The attention score is the sum of each hand's
cos((m−n)·θ). Every term depends on m and n only through their difference. - Lock the gap and drag: both handles slide, all hands rotate, the absolute positions climb — yet every per-hand cosine, and the total score, never move. Relative position is baked into the arithmetic.
- Switch the base to 500,000 (long-context scaling): the fast hand is unchanged but the slower hands slow down further, so they stay unambiguous across far longer documents.
This is the RoPE mechanism from Chapter 30 of Large Language Models from the Ground Up. ← Back to all labs