Rotary Position Embedding

Rotary Position Embedding (RoPE) injects token position into a transformer's attention by rotating each query and key vector by an angle proportional to its position, rather than adding a separate position vector to the input. Jianlin Su and coauthors introduced it in 2021 (arXiv:2104.09864) in a model they called RoFormer. Each vector is split into 2D pairs, and rotating each pair means the dot product between a query and a key ends up depending only on their relative distance, not their absolute positions. Relative position information falls directly out of the ordinary attention computation.

This gives RoPE two properties that made it the default choice for large language models. It extends to sequence lengths not seen during training more gracefully than learned absolute position embeddings. Dependency between tokens also decays naturally as their distance grows, which matches how language tends to behave. Because it modifies the attention computation itself rather than the token embeddings, it composes cleanly with linear and other efficient attention variants that reformulate the same dot product.

RoPE was adopted by LLaMA, PaLM 2, Mistral, Falcon, Qwen, and Gemma, among others, and is now close to a default for transformer position encoding. Work continues on frequency schedules and interpolation methods to extend it to longer contexts than a model was originally trained on. Its rotation-based formulation has since been reused outside language, including in rotary-positioned variants of linear attention for video diffusion transformers.