TTPO (Test-Time Policy Optimization)

Test-Time Policy Optimization (TTPO) is a self-supervised post-training algorithm that updates a language model's policy using only its own majority-vote rollouts as pseudo-labels. Unlike RL or on-policy self-distillation (OPSD) that require ground-truth solutions, TTPO applies an asymmetric objective: rollouts that agree with the majority vote are distilled via OPSD (token-level selection down-weights already-converged positions), while rollouts that disagree are penalized with Grouped RL (penalizing only confident errors). Both updates remain well-grounded even when the majority vote itself is wrong, because the asymmetry exploits the empirical regularity that disagreeing rollouts are usually wrong regardless of whether the vote is correct. TTPO matches label-supervised OPSD on competition-level benchmarks (AIME, HMMT) without using any external labels, raising Qwen3-1.7B from 38.0% to 45.2% under test-time training.

TTPO is adjacent to the broader family of test-time scaling and test-time training methods, which exploit additional inference-time compute to improve model accuracy. Where test-time-scaling re-weights or selects from already-generated outputs, and ttt-test-time-training applies supervised gradient updates at inference, TTPO is the label-free sibling: it uses the model's own consensus as a noisy supervisor and asymmetrically decides which rollouts to trust. The technique connects to on-policy-self-distillation (which it generalizes beyond privileged reasoning traces) and to group-relative-policy-optimization (whose grouped reward signal it uses for the disagreeing branch).

The technique is documented in the paper TTPO: Test-Time Policy Optimization by Wang et al. (Zhejiang University + Alibaba Group), arXiv:2608.27448, August 2026. Code at github.com/szyou21/REAL-TTPO.