Safety Flywheel

Opening

The safety flywheel describes how scaling automated red-teaming produces a self-reinforcing cycle: better attackers generate better adversarial training data, which produces stronger defenders, which then requires stronger attackers to find the remaining failures. Each iteration of the cycle can be measured against the previous generation's robustness benchmarks to verify progress.

Mechanism

A new cycle starts with a frontier-capable model acting as the attacker, trained via self-play against a population of defender models on a diverse scenario set. The finished attacker model then generates adversarial examples for the next defender's training run, after which the new defender is evaluated against both human-crafted and learned attack distributions. Progress is tracked by monotonic improvement on held-out robustness benchmarks, such as GPT-5.6 Sol's 6x reduction in prompt-injection failures compared to four months earlier. The flywheel breaks if attackers cannot find novel failures after the previous cycle, a limit bounded by the attacker model's reasoning capability.

Tradeoffs

The flywheel depends on attacker models that are themselves aligned with the operator's policies. If they are not, adversarial training data introduces policy drift. It also concentrates dual-use capability in attacker-only deployments, which complicates public disclosure of attack categories. Compute requirements scale with the gap between defender and attacker capability, which can create pressure to maintain dual-use models indefinitely.

Open Questions

It is unknown whether the flywheel can sustain improvements as defender robustness approaches an asymptotic ceiling. The relationship between flywheel iteration speed and capability progress speed is also empirical: if attackers plateau before capabilities plateau, safety progress will lag capability progress.