FlowBalance

FlowBalance is a reinforcement learning method for self-improving reasoning language models. It combines a verifier's sparse outcome signal with a dense self-guidance signal generated by the model itself, and it corrects for the main failure mode of self-guidance alone: the model reinforcing its own false confidence and collapsing onto one narrow solution strategy. Zixun Huang, Kishan Panaganti, Haitao Mi, and Leowei Liang, at Tencent's Hunyuan LLM Frontier team and the University of Pennsylvania, introduced it in "FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience" (arXiv:2609.03241, September 2026).

Standard reinforcement learning from verifier rewards (RLVR) gives a reliable but sparse training signal, since a verifier typically judges only the final answer across a long chain of reasoning. Letting the model densely guide itself at every step provides more signal, but it is unreliable on its own and prone to reinforcing plausible-looking but wrong reasoning.

For each on-policy trajectory the model generates, a frozen, training-time copy of the same policy uses privileged context to produce token-level log-probability gains. These are aggregated into a trajectory-level self-guidance score. FlowBalance then calibrates that self-generated score against the verifier-derived group advantage from the same batch of rollouts: guidance is kept as-is on trajectories the verifier scored as better than the group average, reversed on trajectories scored worse, and switched off entirely when the group of rollouts gives no clear outcome preference. This prevents the model from hallucinating a training signal where the verifier offers none. The resulting reweighted target distribution over complete responses is fit using trajectory balance, a technique that matches the whole response distribution to the reweighted target with one log-partition estimate per rollout group, rather than training the model with token-by-token imitation of the target. On mathematical reasoning benchmarks with Qwen3-4B and Qwen3-8B, the authors reported that FlowBalance outperformed a comparable prior method, FlowRL, trained faster and more stably, avoided the response-length collapse reported for an earlier dense self-guidance approach called OPSD, and produced a wider diversity of correct solution strategies in a controlled diagnostic on the AIME24 math benchmark.