Experiments / Does RoPE hold up at context lengths the model never saw in training?

Does RoPE hold up at context lengths the model never saw in training?

RoPE lower at every length to 4096; the expected failure of absolute embeddings did not reproduce.

PyTorch RoPE Positional encoding Perplexity
View notebooks ↗

Setup

Two identical GPT-2 style models, one with RoPE and one with learned absolute position embeddings, trained on 13 Project Gutenberg books (768k tokens) at sequence length 256. Then evaluated at context lengths out to 4096, sixteen times the training length.

Results

Context length RoPE perplexity Absolute perplexity
256 218.5 335.0
512 227.3 348.7
1024 242.5 358.2
2048 255.2 364.3
4096 265.1 366.8

RoPE produced lower perplexity at every length tested.

Perplexity from 256 to 4096 context: RoPE stays well below absolute embeddings at every length, and the absolute curve flattens instead of diverging.
Perplexity against context length for both models.

What I learned

The surprise was the absolute-embedding curve. Its position table is sized for 4096 but only ever trained at 256, so positions beyond 256 were untrained, and I expected perplexity to blow up once the model was pushed past the training length. It did not: the curve flattened near 365 instead of climbing. So this run says RoPE is better at every length I tested, but the specific extrapolation failure I built the test for did not reproduce at this scale.