Skip to main content

Documentation Index

Fetch the complete documentation index at: https://mintlify.com/itsubaki/gpt/llms.txt

Use this file to discover all available pages before exploring further.

RoPE encodes the position of each token by rotating consecutive pairs of dimensions in the query and key vectors using position-dependent angles. Because the rotation is applied to both Q and K before their dot product, the attention score between two tokens depends only on their relative distance — the model naturally captures “token A is three positions before token B” without needing a separate positional embedding added to the residual stream.

​
RoPEFunc type alias

type RoPEFunc func(offset int) func(x ...*variable.Variable) *variable.Variable
RoPEFunc is a curried function type. The outer call accepts the current token offset (used during KV-cache-based generation to indicate how many tokens have already been processed), and returns an autograd-compatible function that applies the rotation to a (B, H, C, D) tensor. Both the forward and backward passes are implemented on the underlying RoPET struct.

​
Constructor and precomputed tables

func RoPE(theta float64, embedDim, contextLen int) RoPEFunc {
    if embedDim%2 != 0 {
        panic(fmt.Sprintf("embedDim=%d is odd", embedDim))
    }

    halfDim := embedDim / 2
    cos := make([]float64, contextLen*halfDim)
    sin := make([]float64, contextLen*halfDim)

    for pos := range contextLen {
        for i := range halfDim {
            pow  := float64(2*i) / float64(embedDim)
            freq := 1.0 / math.Pow(theta, pow)
            angle := float64(pos) * freq

            idx := pos*halfDim + i
            cos[idx] = math.Cos(angle)
            sin[idx] = math.Sin(angle)
        }
    }

    return func(offset int) func(x ...*variable.Variable) *variable.Variable {
        return (&variable.Function{
            Forwarder: &RoPET{
                Cos:        cos,
                Sin:        sin,
                HalfDim:    halfDim,
                ContextLen: contextLen,
                offset:     offset,
            },
        }).First
    }
}
RoPE allocates two flat slices — cos and sin — of length contextLen × (embedDim/2). Each entry at pos*halfDim + i holds the cosine or sine of the angle for position pos and dimension pair i. These tables are computed once at model construction time and reused for every forward pass.

​
Parameters

ParameterDescription
thetaBase frequency (default 10000); higher values slow down the frequency decay across dimensions
embedDimMust be even; the table covers embedDim/2 dimension pairs
contextLenMaximum number of positions; also the bound for the offset + C clamp

​
Rotation formula

The forward pass iterates over every (batch, head, position, dimension-pair) combination and applies a 2D rotation: 
for d := 0; d < D; d += 2 {
    idx := (offset+pos)*f.HalfDim + d/2
    cos := f.Cos[idx]
    sin := f.Sin[idx]

    x0 := x[0].At(b, h, pos, d)
    x1 := x[0].At(b, h, pos, d+1)

    y0 := x0*cos - x1*sin  // rotated even component
    y1 := x0*sin + x1*cos  // rotated odd component

    y.Set([]int{b, h, pos, d},   y0)
    y.Set([]int{b, h, pos, d+1}, y1)
}
Expressed concisely for even dimension index d: 
y[d]   = x[d]   * cos(θ_pos,d) − x[d+1] * sin(θ_pos,d)
y[d+1] = x[d]   * sin(θ_pos,d) + x[d+1] * cos(θ_pos,d)
where θ_pos,d = pos / theta^(2d/embedDim). Lower-indexed dimension pairs rotate at high frequency (small denominator) and higher-indexed pairs rotate slowly, giving the model a range of periodicities to represent position at different scales.

​
The theta hyperparameter

The base theta (default 10000, as in the original RoPE paper) controls the frequency spectrum: 
freq_i = 1 / theta^(2i / embedDim)    for i = 0, 1, ..., embedDim/2 - 1
  • i = 0: freq = 1.0 — one full rotation every 2π tokens (high frequency).
  • i = embedDim/2 - 1: freq ≈ 1 / theta — very slow rotation (low frequency).
A larger theta (e.g. 500000 used by LLaMA 3) spreads the frequency range further, helping the model generalize to longer contexts. The default 10000 suits shorter contexts like MaxContextLen = 256.

​
Offset during KV-cache generation

When useCache is active, the attention layer passes a non-zero offset to RoPEFunc for every generation step after the first: 
// inside MultiHeadAttentionT.Forward:
Q = l.rope(l.offset)(Q)
K = l.rope(l.offset)(K)
// ...
l.offset += C
The offset shifts the position index so that a single new token at logical position offset receives the correct rotational angles from the precomputed table rather than always starting from position 0.

​
Position clamping

If the accumulated offset would exceed the precomputed table, clamp prevents an out-of-bounds index: 
func (f *RoPET) clamp(C int) int {
    if f.offset+C > f.ContextLen {
        return f.ContextLen - C
    }
    return f.offset
}
When offset + C > contextLen, the effective offset is set to contextLen - C, so the last C rows of the table are reused. This is a safe fallback but generating beyond contextLen tokens will degrade positional accuracy — stay within MaxContextLen for best results.

​
Backward pass (conjugate rotation)

RoPE is differentiable. The backward pass applies the conjugate (transpose) rotation to the upstream gradient: 
func (f *RoPET) Backward(gy ...*variable.Variable) []*variable.Variable {
    // ...
    for d := 0; d < D; d += 2 {
        idx := (offset+pos)*f.HalfDim + d/2
        cos := f.Cos[idx]
        sin := f.Sin[idx]

        gy0 := gy[0].At(b, h, pos, d)
        gy1 := gy[0].At(b, h, pos, d+1)

        gx0 :=  gy0*cos + gy1*sin   // inverse rotation, even component
        gx1 := -gy0*sin + gy1*cos   // inverse rotation, odd component

        gx.Set([]int{b, h, pos, d},   gx0)
        gx.Set([]int{b, h, pos, d+1}, gx1)
    }
    // ...
}
Because the rotation matrix is orthogonal, its inverse is its transpose, so the backward pass simply reverses the sign of the sine term. No additional parameters are involved — RoPE has zero learnable weights.
The single RoPEFunc returned by function.RoPE(...) is shared across all transformer blocks. Each call to the closure creates a fresh RoPET struct with the desired offset, but the underlying cos and sin slices are shared by reference, keeping memory usage proportional to contextLen × embedDim / 2 regardless of model depth.

Build docs developers (and LLMs) love

Get started for freeTalk to us