Recurrent Depth

Recurrent depth

Recurrent depth is a model architecture technique in which a single Transformer block, or a small block of layers, is applied repeatedly to the same hidden state. Each application adds another iteration of computation over the same weights. Instead of stacking distinct layers, the network deepens through time: the same parameters are invoked multiple times per token, and reasoning accumulates through the loop rather than through vertical depth. The hidden state carries forward between iterations, so each pass can use the prior iteration's intermediate computation.

The technique was popularized as the defining architectural choice of Astra, a reasoning-focused model released in late summer 2026. It is one of several approaches, alongside test-time scaling, chain-of-thought prompting, and process-reward modeling, that aim to spend additional inference compute on harder problems. Chain-of-thought methods externalize reasoning by emitting intermediate tokens. Recurrent depth internalizes it: more steps of reasoning happen inside the residual stream rather than in the visible output. Internal computation does not extend the context window, does not grow the KV cache for later tokens, and does not need to be verbalized in a form the model has been trained to produce.

The trade-off is sequentiality. Each loop iteration depends on the previous iteration's output, so the loop cannot be parallelized within a single forward pass the way distinct Transformer layers can. Whether recurrent depth is faster in wall-clock terms than a deeper model depends on the workload. It can be cheaper when only some tokens require extra reasoning and a gating mechanism routes them through additional loops. It can be slower when every token pays the full loop cost. As Elie Bakouch noted in commentary on Astra, the real question is whether looping the same weights yields better reasoning than spending compute on a deeper model with new weights.

Recurrent depth intersects with older ideas. Weight sharing across layers has a long history in convolutional networks, where it expanded the receptive field without adding parameters, and in universal Transformers, which loop a single block across positions. The recurrent-depth framing emphasizes the inference-time reasoning budget rather than parameter efficiency. The model receives more computation per token, which is what makes it competitive with o1-style reasoning models that spend additional compute on chain-of-thought sampling and verification. Astra's release is the first widely deployed model where the dominant share of the reasoning budget is looped compute rather than CoT depth.

The technique has implications for safety that distinguish it from prior reasoning methods. Chain-of-thought reasoning is visible in the model's output and can be inspected, audited, or used as a forensic trace. That property helped investigators retrace the July 2025 Hugging Face supply-chain attack. When reasoning moves into the latent space via recurrent depth, that visibility disappears. The model's intermediate computations are still represented, but they are no longer expressed in a language humans can read. A 2024 AI-safety paper co-authored by researchers from OpenAI, Anthropic, and Google DeepMind warned that latent reasoning models might not need to verbalize any of their thoughts and would thus lose the safety advantages that chain-of-thought confers. Recurrent depth is the first technique to bring that theoretical concern into a widely deployed model.