Teacher-Guided Rejection Sampling
A fine-tuning method that filters training samples using evaluations from multiple expert models.
Teacher-guided rejection sampling is a training refinement technique that combines rejection sampling with ensemble-style teacher evaluation to improve the quality of data used for fine-tuning a target model. Rather than training on all available or generated samples indiscriminately, the method selectively accepts only those candidates that meet quality thresholds as judged by one or more pre-trained expert models — the "teachers." When multiple teachers are involved, acceptance may require agreement from a majority or all of them, ensuring that only high-confidence, high-quality examples influence the student model's learning.
The mechanism works in iterative cycles. First, the target model (or a separate generator) produces candidate outputs — these might be responses, completions, or synthetic data points. Each candidate is then scored or evaluated by the teacher models, which may themselves be larger, more capable systems or domain-specific experts. Candidates that fail to meet the collective approval threshold are discarded, while accepted samples are used to update the target model through supervised fine-tuning or reinforcement-style feedback. This loop can repeat across multiple rounds, progressively steering the model toward higher-quality behavior.
The technique is especially valuable when training data is noisy, scarce, or difficult to label reliably. By delegating quality judgments to trusted teacher models, the approach reduces the risk of the student model learning from flawed or misleading examples — a common failure mode in self-improvement pipelines. It also provides a principled way to leverage the strengths of large, expensive models to improve smaller, more deployable ones without requiring those large models to be deployed at inference time.
Teacher-guided rejection sampling gained particular relevance in the early 2020s as the field increasingly explored scalable oversight, model alignment, and knowledge distillation strategies. It connects naturally to broader frameworks like RLHF and constitutional AI, where the goal is to shape model behavior through carefully curated feedback signals rather than raw data volume. Its ability to enforce quality gates through ensemble consensus makes it a robust tool for building reliable, well-calibrated models in high-stakes applications.
