Control Problem

The control problem refers to the fundamental challenge of designing AI systems—particularly those approaching or exceeding human-level capability—such that they reliably pursue goals aligned with human intentions rather than diverging in harmful or unintended directions. The core difficulty is not simply programming an AI to follow instructions, but ensuring that as systems become more capable, they remain steerable, interpretable, and correctable. A sufficiently powerful optimizer pursuing even a subtly misspecified objective could cause serious harm, making the problem both a technical and a philosophical one: we must specify what we want, verify the system has internalized it correctly, and retain the ability to intervene if something goes wrong.

The technical dimensions of the control problem include corrigibility (building systems that accept correction and shutdown without resistance), value learning (enabling AI to infer human preferences from behavior rather than requiring exhaustive manual specification), and containment or interruptibility mechanisms that prevent a capable system from circumventing oversight. These challenges are compounded by instrumental convergence—the theoretical observation that many different high-level goals share common subgoals like self-preservation and resource acquisition, meaning a misaligned system might resist correction as a byproduct of almost any objective it pursues.

The control problem sits at the intersection of AI safety research, decision theory, and ethics, and has driven the formation of dedicated research programs at institutions like the Machine Intelligence Research Institute, the Center for Human-Compatible AI, and DeepMind's safety team. While some researchers view catastrophic misalignment as a distant concern, others argue that solving controllability is a prerequisite for responsibly scaling AI systems at all. As large language models and autonomous agents become more capable and widely deployed, the practical dimensions of the control problem—robustness, oversight, and alignment under distribution shift—have become increasingly concrete and urgent.