Looped Transformer

A looped transformer applies the same stack of layers to the same hidden representation more than once, sharing weights across iterations. Instead of stacking N distinct transformer blocks for depth, the model unrolls a smaller block K times. When the loop count is fixed, the design is sometimes called recurrent depth; when the count can vary per token, looped transformer.

The design builds on universal transformers, deep equilibrium models, and weight-tied recurrent networks. In 2024 and 2025, it was used to scale depth on a fixed inference budget. According to The Information, OpenAI's Astra model, reported in September 2026, uses a recurrent depth or looped transformer design.

Because weights are shared across iterations, effective compute grows with the loop count while memory does not. Mixture-of-Recursions adds a learned router that assigns each token a per-input loop budget, letting the model allocate depth where it is needed. Looped transformers compose with grouped-query attention, RoPE, and RMSNorm, which is why open-weights models like Nanbeige 4.2 (3B) ship as looped architectures.