CHAPTER 4.2c · POSITIONAL ENCODING

RoPE Rotation Animation

Visualization of how Rotary Position Embeddings (RoPE) encode relative positions through vector rotation

RoPE Rotation shows how position is encoded as an angle. Each 2D pair in the embedding is rotated by a position-dependent angle. The key insight: The dot product between two rotated vectors depends only on their relative position – perfect for Attention.

📖 Learning Context ▼
🎯

Learning Objectives

  • Understand the geometric intuition behind RoPE: Position = Rotation angle
  • Follow how 2D subspaces are rotated independently
  • Recognize why relative position emerges from rotation difference
🗺️

Context: Where are we?

Step 2/6 Optimizations & Memory

This animation makes the mathematical idea of RoPE tangible. It complements the theoretical RoPE-ALiBi overview with an interactive visualization.

💡

Why It Matters

RoPE is the position encoding in Llama, Mistral, Qwen and most open-source models. Understanding how it works explains why these models can be extended to 128K+ tokens.

📌

Key Takeaways

  • Dimension Pairs: Embedding is split into d/2 2D pairs, each with its own base angle
  • Frequency Mix: Different pairs rotate at different frequencies (like Fourier)
  • Relative Position: q·k = f(q, k, relative_pos), not absolute position

What is RoPE?

Rotary Position Embedding (RoPE) rotates vectors in 2D subspaces based on their position. The key: Relative position between two tokens corresponds to the rotation difference of their embeddings. This enables zero-shot length extrapolation and is used in Llama, PaLM and GPT-NeoX.

Controls

5
10
0.10

Rotation Angles

Position 1 (Query)
0.50 rad
Position 2 (Key)
1.00 rad
Relative Rotation
0.50 rad
Dot Product
0.877
Fig. 1 | RoPE Rotation in 2D subspace. Blue vector (Position 1/Query) and orange vector (Position 2/Key) rotate by different angles. The angle difference encodes their relative position. The dot product is invariant to rotation, but depends on the relative angle.

RoPE Formula

RoPE(x, m) = [
cos(m·θ) · x₀ - sin(m·θ) · x₁,
sin(m·θ) · x₀ + cos(m·θ) · x₁
]

m: Position of the token
θ: Rotation frequency (e.g., 1/100002i/d)
x₀, x₁: 2D subspace of the vector

Why Does RoPE Work?

The dot product between Query at position m and Key at position n:

q_m · k_n = q · k · cos((m-n)·θ)

Depends only on the relative position (m-n), not on absolute positions! This enables length extrapolation: If the model was trained on 2K tokens, it often still works at 8K+ tokens.