RLMF (Reinforcement Learning with Metacognitive Feedback)

Reinforcement Learning with Metacognitive Feedback (RLMF) is a training paradigm introduced by Liu, Caciularu, Yona, Szpektor, and Cohan (Yale + Google Research, 2026) that uses a model's self-judgments of its own task performance as the reward signal during preference optimization. Instead of relying on external annotators or task-level accuracy, RLMF asks the model to assess the quality of its own completion — and rewards completions whose self-assessed quality matches their actual quality. The mechanism operationalizes the metacognitive hypothesis that models which can accurately judge their own performance are better positioned to improve it.

The paradigm has two components: (1) RLMF proper, which refines completion rankings during preference optimization (e.g., DPO or PPO) using the model's self-judgments weighted by their accuracy, and (2) metacognitive data selection, which uses the same self-judgments to identify high-value training examples, outperforming naive active learning baselines. Both mechanisms exploit the model's intrinsic feedback as a signal — a strategy that has been underused in prior work because extrinsic feedback has been easier to scale.

Applied to the task of faithful calibration (aligning expressed with intrinsic uncertainty), RLMF achieves state-of-the-art results on diverse tasks while preserving accuracy, surpassing standard RL by up to 63%. The improvement generalizes across model sizes and downstream tasks. RLMF is adjacent to RLHF (which uses external human feedback), RLAIF (which uses AI feedback on outputs), and self-rewarding-language-models (which use self-judgments directly). It is distinct in that the feedback signal is the model's self-assessment rather than its own outputs, and in that the self-assessments are filtered through a calibration gate.