Scalable Oversight
Methods for supervising AI systems whose outputs exceed human ability to evaluate directly, including debate, recursive reward modeling, and weak-to-strong generalization.
Scalable oversight is the problem of supervising AI systems that produce outputs humans cannot reliably evaluate. The setting arises when models become more capable than the humans rating their outputs: a human annotator cannot judge whether a long, expert-level answer is correct, but the answer still needs to be scored for training.
Proposed approaches include AI debate (two models argue opposing positions, a human judge picks the winner), recursive reward modeling (a smaller model rates a larger model's output, the smaller model is itself rated by an even smaller model), weak-to-strong generalization (using a weaker model's labels to train a stronger one), and constitution-based methods (rule-following rather than preference scoring).
A 2026 paper by Bergemann, Koh, and Morris treats scalable oversight as a mechanism design problem: the question is what contracts can elicit honest work from an agent whose capability exceeds the principal's ability to verify the work directly.
Sources
- Mechanism Design for Alignment and Control
arXiv · Sep 1, 2026
- AI safety via debate (Irving, Christiano, Amodei)
arXiv · May 2, 2018
- Weak-to-strong generalization (OpenAI)
OpenAI · Dec 6, 2023
Research this in Signals
Signals turns a topic into a sourced research record you can inspect and rerun. Your first scan is free, and this one starts with Scalable Oversight already loaded, so edit it or scan as is.