---
title: Data Poisoning
type: vocabulary
url: "https://www.envisioning.com/vocab/data-poisoning"
summary: A cyberattack that corrupts training data so a model learns hidden malicious behaviors.
year: 2017
generality: 0.65
---

# Data Poisoning

A cyberattack that corrupts training data so a model learns hidden malicious behaviors.
Data poisoning is a training-time attack in which an adversary inserts, modifies, or removes examples in the dataset used to train a machine learning model so the resulting model carries hidden behaviors the attacker controls. The poisoned model's accuracy on benign inputs may look normal, but specific triggers elicit attacker-chosen outputs, including backdoors that activate on hidden phrases, biased predictions on targeted subgroups, or backdoored code-generation pipelines that emit insecure code on command. Defending an open-weight model requires the defender to inspect both the training data and the released weights, neither of which can fully guarantee absence of an implant, so the integrity of the upstream training pipeline becomes the model's actual security boundary.

The mechanism works by exploiting the fact that supervised models learn any consistent pattern present in their training data. An attacker who controls a small fraction of training samples can seed a pattern that the optimizer treats as a high-signal correlation, baking it into the parameters so it persists in deployment. The classic backdoor form trains the model on examples labeled correctly except when a hidden trigger (a phrase, a pixel pattern, an unusual token sequence) is present, in which case the label is flipped to the attacker's target. Recent demonstrations on production-scale open-weight models show that as few as ten poisoned documents in a multi-trillion-token corpus are sufficient to implant a working backdoor, because the gradient updates from poisoned samples are amplified by the high learning-rate cycles these models use.

The trade-offs cut sharply along the open-versus-closed axis. Closed-weight models with curated internal training pipelines and human review have a smaller, easier-to-defend attack surface, but their inner workings are opaque so downstream users cannot verify the absence of implants. Open-weight models offer full parameter transparency but radically expand the supply chain: anyone who contributes to the public crawl, dataset shard, fine-tuning corpus, or post-training dataset used by the model publisher is a potential attacker, and a single poisoned source can be laundered through hundreds of subsequent crawls. End users pay the cost either way, and no current evaluation benchmark can reliably detect a sophisticated implant without access to the original training set.

Open questions dominate the field. There is no general, efficient defense against poisoning that does not require either trusted training data or substantial additional compute for anomaly detection on gradients and representations. Empirical work suggests backdoor triggers can be made indistinguishable from natural data variation, defeating signature-based scanning. The relationship between model scale and poisoning robustness is contested: larger models can memorize poisoned patterns more reliably, but they also have more capacity to learn benign workarounds, so the marginal cost per successful attack may rise with scale even as absolute success becomes easier. Until provenance, dataset attestation, and post-training forensic auditing mature, poisoning remains a structurally asymmetric advantage for attackers over defenders in any setting where training data is not fully controlled by a single trusted party.

---
Source: Envisioning — Technology Research Institute (https://www.envisioning.com/vocab/data-poisoning)