Dream-RSI

Dream-RSI is a framework for recursive self-improvement applied to an exploration policy rather than a model's weights, introduced in a September 2026 paper by Tong Zheng and coauthors (arXiv:2609.14858). The target is agent-driven discovery tasks, such as algorithm design, mathematical optimization, and GPU kernel engineering, where a coding agent repeatedly proposes and evaluates candidate solutions over long-horizon search. The discovery agent, evaluator, and execution interfaces stay fixed; only the code that decides where to branch, what to run in parallel, and when to stop gets updated.

The method treats a completed discovery run as a "replay world." Each run produces a discovery tree recording which branches were tried and their outcomes. Because these outcomes are already stored, the tree can answer counterfactual questions about alternative exploration strategies (different branch orderings, batch sizes, or stopping points) without rerunning the underlying agent. Dream-RSI cycles through three stages: online exploration, where the current policy drives real discovery and logs a new tree; simulator construction, where trees accumulate into a replay pool; and "dreaming," where many candidate policies are scored cheaply against that pool before the best one is redeployed online.

The paper reports that this loop matches or improves discovery quality while cutting the number of costly agent calls substantially across all three tested domains, compared with a fixed-exploration baseline. Several authors are affiliated with Google DeepMind, with additional coauthors from the University of Maryland and the University of Virginia. Dream-RSI is narrower than the general recursive self-improvement concept its name references: it doesn't rewrite model weights or architecture, only the meta-level policy governing how exploration compute is allocated.