Mixture-of-Recursions (MoR)

Mixture-of-Recursions (MoR) is a transformer architecture introduced by Bae et al. in July 2025. It combines weight-tied recurrent depth with a learned router: each transformer block is applied multiple times to the same token representation, with the number of applications chosen per token by the router. Easy tokens loop once; harder tokens loop two or three times.

MoR extends routing schemes from mixture-of-experts by routing on recursion depth rather than expert identity. The original paper explores both token-choice routing (each token picks its own depth) and expert-choice routing (each depth level picks its tokens). On language modeling benchmarks, MoR matches larger non-recursive baselines with a fraction of the parameters and FLOPs.

Because weights are shared across recursion steps, MoR admits a key-value cache reuse trick that keeps memory cost flat regardless of per-token loop count. The architecture has been used as a reference for inference-efficient long-context and reasoning models in 2025 and 2026.