---
title: Scalable Oversight
type: vocabulary
url: "https://www.envisioning.com/vocab/scalable-oversight"
summary: Methods for supervising AI systems whose outputs exceed human ability to evaluate directly, including debate, recursive reward modeling, and weak-to-strong generalization.
year: 2018
generality: 0.50
---

# Scalable Oversight

Methods for supervising AI systems whose outputs exceed human ability to evaluate directly, including debate, recursive reward modeling, and weak-to-strong generalization.
Scalable oversight is the problem of supervising AI systems that produce outputs humans cannot reliably evaluate. The setting arises when models become more capable than the humans rating their outputs: a human annotator cannot judge whether a long, expert-level answer is correct, but the answer still needs to be scored for training.

Proposed approaches include AI debate (two models argue opposing positions, a human judge picks the winner), recursive reward modeling (a smaller model rates a larger model's output, the smaller model is itself rated by an even smaller model), weak-to-strong generalization (using a weaker model's labels to train a stronger one), and constitution-based methods (rule-following rather than preference scoring).

A 2026 paper by Bergemann, Koh, and Morris treats scalable oversight as a mechanism design problem: the question is what contracts can elicit honest work from an agent whose capability exceeds the principal's ability to verify the work directly.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/scalable-oversight)