OPSA (On-Policy Self-Adaptation)

On-Policy Self-Adaptation (OPSA) is a language-model post-training method introduced by Yi Ding and Ruqi Zhang in "Does On-Policy Distillation Really Distill? From Noisy Teacher to Self-Improvement" (arXiv:2608.31046, August 2026). It came out of an analysis of on-policy distillation (OPD), a technique in which a student model generates its own outputs while a teacher model scores each token to provide dense, per-token supervision. The authors found that OPD's teacher-derived scores are frequently noisy, and that the student converges to similar performance whether or not that noisy supervision is kept. This raised the question of what OPD's per-token feedback is actually doing.

Tracing the source of OPD's gains, they found that learning concentrates on the model's own low-probability ("tail") tokens. Replacing the teacher's per-token score there with a single fixed negative value matched the teacher's performance. This suggested OPD works mainly by suppressing unlikely tokens, a signal that does not require a teacher at all. OPSA turns this observation into a supervision-free method. It identifies low-probability response tokens and assigns them a negative advantage that scales with the token's entropy, applying a stronger penalty at higher-entropy positions and redistributing probability mass toward the remaining, more probable "head" tokens.

Without any teacher model, OPSA improved a Qwen3-1.7B baseline's Avg@32 score on AIME24 by 35.41 points, a 263% relative gain, and more than doubled Pass@32 across the three benchmarks tested. It also outperformed OPD itself by 16.77 points in Avg@32 on AIME24, at lower compute cost, since it removes teacher inference entirely. The result suggests that at least part of the reported benefit of on-policy distillation comes from implicit entropy regularization rather than from information transferred from the teacher.