Living Benchmark

A living benchmark is an evaluation set whose contents are regularly updated — new items added, retired items removed, contamination filtered — so that the benchmark continues to measure progress rather than memorizing existing items. The pattern emerged in response to benchmark saturation: static benchmarks like GLUE, SuperGLUE, MMLU, and HumanEval are increasingly problematic because the models being evaluated have either seen the items in training, or the items have been solved to the ceiling by current models, or both. A living benchmark addresses both by being a moving target: items enter the benchmark after a model's training cutoff, so contamination is structurally impossible, and items that are saturated near ceiling are retired. The approach is now the dominant pattern for serious evaluations of foundation models.

Mechanically, a living benchmark has three operational components. First, a refresh pipeline that continuously generates or curates new items with known ground truth, designed to match the benchmark's topical scope. Second, a contamination filter that blocks items that are too similar to public web content or that appear in suspected training corpora. Third, a retirement rule that removes items once the best public model scores above a threshold on them (usually 90–95%), so the benchmark maintains headroom. Living benchmarks also typically publish historical scores so that progress over time can be tracked even as items rotate. The standard examples as of 2026 include: SWE-Bench Live (continuously updated coding tasks drawn from real GitHub issues), LiveCodeBench (contest programming problems with known cutoff dates), FrontierMath (research-level math, kept at frontier difficulty), and ARC-AGI's evolving test sets. Humanity's Last Exam occupies a different role — a one-shot very-high-difficulty set that is not living but is recalibrated against the current frontier.

The advantage of the living-benchmark frame is that it gives a name to a pattern that was an ad-hoc workaround at the model-evaluation labs but is now a standard practice. The frame is also useful for distinguishing serious capability measurement from marketing benchmarks that are static, leaked into training, or cherry-picked for stories. The cost of the frame is that living benchmarks have their own problems: refresh pipelines can introduce noise, items can be retired too quickly (creating artificial jumps in apparent progress), and the contamination filter can be circumvented by paraphrased versions of known items. Living benchmarks also tend to be more expensive to maintain than static ones, which means fewer groups can run them and the field becomes dependent on a small number of evaluation orgs. There is also a methodological question about whether living benchmarks are measuring capability improvement or just measuring how well the latest models are at the type of tasks the benchmark refresh happens to contain.

Open questions include how to design refresh pipelines that produce items at the right difficulty (items that are neither trivially solved nor impossible) and how to ensure the refresh process does not become a vector for bias (item generation through human contractors or LLMs may inherit the biases of those sources). There is also a deeper methodological question about whether capability measurement is even the right frame for evaluating foundation models, or whether something more like red-teaming, deployment monitoring, and outcome measurement would be more informative. The living-benchmark frame is a best-effort approach to a measurement problem that does not have a clean solution.