Capability Eavesdropping

Capability eavesdropping is the umbrella term for attacks in which an adversary extracts information about an AI model's capabilities, parameters, training data, or membership — without authorized access — by interacting with the model through its public interfaces (API, chat, or download endpoint). The category unifies a set of otherwise distinct technical attacks that share an information-extraction goal: model inversion (reconstructing training examples from model outputs or confidence scores), membership inference (determining whether a specific example was in the training set), gradient leakage (extracting training data or inputs from shared gradients in federated learning), weight extraction (reconstructing model parameters from repeated black-box queries), and capability extraction (inferring the model's training compute, dataset size, or architectural details from observed behavior). The term was not widely used before 2025 but is increasingly the relevant category as foundation models expose rich attack surfaces through public APIs and downloaded weights.

Mechanically, capability eavesdropping works by exploiting the information that models necessarily leak through their operation. A model that gives calibrated confidence scores leaks which examples it has memorized. A model that supports few-shot prompts leaks capability fingerprints based on which prompts succeed. A model whose weights are downloadable can be probed at scale to extract training data patterns, safety alignment details, and even backdoor triggers. The attacks are often surprisingly cheap: model inversion can be done with a few hundred queries on commercial APIs; gradient leakage has been demonstrated from shared parameters in federated learning settings; capability extraction can fingerprint a frontier model from a small handful of probing prompts. Defenses exist — output perturbation, differential privacy, response rate-limiting, watermark embedding — but they trade off against model utility and have not generalized across all attack types. The result is an asymmetric situation in which eavesdropping attacks are practical and cheap, and defenses are partial and degrade model quality.

The advantage of the capability-eavesdropping frame is that it gives a single name to a set of attacks that are usually discussed in technical isolation. The frame is also useful for regulation and disclosure: regulators and downstream deployers need to reason about the class of risk rather than the individual attack. The cost is that the technical literature has historically treated each attack type separately, and the umbrella framing risks blurring important differences — model inversion and weight extraction are different problems with different defenses. There is also a real concern that the umbrella frame overstates attacker capability: many published attacks require idealized conditions (full gradient access, white-box model access) that do not generalize to deployed models without weights. Even so, the attacks that do generalize — extraction through API probing, membership inference on production systems — are real and currently undefended at scale.

Open questions include how to design deployed models that leak less through their public interfaces, how to reason about cumulative leakage across many attackers and many queries, and whether eavesdropping risk changes the appropriate deployment model (closed APIs versus downloadable weights). There is also a question about who has standing to disclose: a research lab that demonstrates an eavesdropping attack on a production model is publishing information that has both defensive and offensive uses. The interaction between capability eavesdropping and model-stealing/distillation-attacks is also under active investigation, because the techniques overlap considerably.