Incompressible Knowledge Probes
A probing technique that uses rare or niche factual recall to estimate model capacity, on the assumption that facts not compressible into shared representations reveal parameter count.
Incompressible Knowledge Probes (IKPs) are a class of evaluation probes designed to estimate the parameter count of a language model by testing it on niche, low-prevalence factual knowledge. The technique assumes that for sufficiently rare facts, a model cannot share representational capacity across examples, so the amount of niche knowledge it can recall scales approximately with its parameter count. By curating a benchmark of obscure but verifiable facts and measuring recall accuracy across models of known size, researchers can extrapolate to estimate the size of unknown models. The technique was introduced for analyzing frontier model releases where parameter counts are not publicly disclosed.
Sources
- Exploring Claude/GPT Knowledge Cutoffs & Pre-training Timelines
Shrivu's Substack · Aug 10, 2026
- How do language models learn facts? Dynamics, curricula and hallucinations
arXiv · Mar 27, 2025
Research this in Signals
Signals turns a topic into a sourced research record you can inspect and rerun. Your first scan is free, and this one starts with Incompressible Knowledge Probes already loaded, so edit it or scan as is.