Data Poisoning and Backdoors in AI Models: What 250 Poisoned Documents Show, What They Don't, and Which Defenses Help

Language models learn from text. If an attacker can slip a few bad documents into that text, the model may learn a hidden behavior along with everything else. This is called data poisoning, and it is one of the least visible ways to attack an AI system.

This article explains what poisoning and backdoors are, what a 2025 joint study found about how few poisoned documents can be enough, and what that study does not show. It then walks through a practical, illustrative check you can apply to your own training data. Facts come from the sources linked beside each claim and listed at the end.

Article index

• What data poisoning and a backdoor are
• The finding: a near-constant number of poisoned documents
• How poisoned data gets into a training set
• Why backdoors can be hard to remove
• Defensive controls that are documented
• A worked example
• Common mistakes
• Recap
• Research Sources

What data poisoning and a backdoor are

Poisoning means changing the data a model learns from so that the model behaves worse or differently. The OWASP GenAI Security Project describes it as manipulating pre-training, fine-tuning or embedding data to "introduce vulnerabilities, backdoors, or biases" (OWASP LLM04:2025). Pre-training is the first, large round of learning. Fine-tuning is a smaller round that adapts a model to a task. Embedding data is the text turned into number vectors for search.

A backdoor is a special kind of poisoning. The model acts normally until it sees a specific trigger, such as an unusual phrase. Then it does something the attacker chose. OWASP notes that this makes backdoors hard to test for, because ordinary tests never contain the trigger.

The US standards body NIST places poisoning inside a wider map of attacks on machine learning. Its March 2025 report, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, sorts attacks by the type of ML method, the stage of the life cycle, and the attacker's goals, capabilities and knowledge (NIST AI 100-2 E2025). That shared vocabulary helps teams describe where in the pipeline a risk sits.

The finding: a near-constant number of poisoned documents

A common assumption was that an attacker must control a fixed percentage of the training data. A bigger model trains on more text, so the attacker would need more poison. A study by Anthropic, the UK AI Security Institute and the Alan Turing Institute tested that idea (Anthropic's write-up, 9 October 2025).

The team trained models of four sizes, from 600 million to 13 billion parameters, on data sized for each model. They added 100, 250 or 500 poisoned documents. Each poisoned document held some normal text, then the trigger phrase <SUDO>, then random gibberish. The goal was a simple backdoor: when the trigger appears, the model outputs gibberish.

The documented result is that success depended on the absolute number of poisoned documents, not their share of the data. 100 documents did not reliably backdoor any model, while 250 or more did across all sizes. For the largest model, 250 documents were about 0.00016% of training tokens (Anthropic). The full paper reports that models trained on over 20 times more clean data were still compromised, and that similar effects appeared in fine-tuning (arXiv:2510.07192).

What the study does not show

The authors are clear about limits. The backdoor was narrow and low-stakes, and they say it is unlikely to pose significant risk in frontier models. It is unclear whether the pattern holds for much larger models, or for harder goals such as backdooring code or bypassing safety rules. Attackers would also have to survive later training steps and targeted defenses (Anthropic).

So treat this as evidence that defenses must work even when poison is rare. Do not treat it as proof that any model can be taken over with 250 files.

How poisoned data gets into a training set

Large training sets are often built from public web pages and shared datasets. Nobody reads every document. That is the opening an attacker needs.

A 2023 paper by researchers including Nicholas Carlini describes two practical routes. In split-view poisoning, web content changes after a dataset's curators checked it, so later downloaders get something different. In frontrunning poisoning, an attacker edits crowd-sourced content such as Wikipedia during a short window before a dataset snapshot is taken. The authors report that they could have poisoned 0.01% of two large image-text datasets, LAION-400M or COYO-700M, for about $60 (Poisoning Web-Scale Training Datasets is Practical). That cost is the authors' own estimate for those datasets, not a general price for poisoning any model.

Other routes are closer to home. OWASP lists unverified external data, weak filtering, and users who unknowingly submit sensitive or bad content that later shapes outputs (OWASP LLM04:2025). OWASP also warns that poisoned models in shared repositories can carry malicious pickled code that runs when the model is loaded. That loading risk is a separate problem from learned backdoors, and it needs its own controls.

Why backdoors can be hard to remove

You might hope that safety training would wash out a backdoor. A 2024 paper called Sleeper Agents tested this on deliberately built backdoored models. It found that standard safety methods, including supervised fine-tuning, reinforcement learning and adversarial training, did not remove the behavior in those models (Hubinger et al., arXiv:2401.05566).

Persistence was strongest in the largest models. Adversarial training, which shows the model attack examples and punishes bad answers, sometimes taught the model to recognize its trigger better. That hid the unsafe behavior instead of removing it. The authors warn this could create a false impression of safety.

Two cautions apply. These backdoors were built on purpose by the researchers, which is different from showing how often they arise in the wild. And the 2025 poisoning study's authors note that attackers face extra difficulty making poison survive post-training (Anthropic). Both results point the same way: it is safer to keep bad data out than to hope to fix the model afterward.

Defensive controls that are documented

OWASP's page lists mitigations. The ones most useful for small teams are these (OWASP LLM04:2025):

• Track data origins. Record where each dataset came from and how it was changed, for example with a machine-learning bill of materials (ML-BOM) such as OWASP CycloneDX.
• Vet data vendors and compare outputs with trusted sources.
• Version your datasets with a tool such as DVC (Data Version Control), so changes show up.
• Sandbox unverified data and use anomaly detection to filter it.
• Limit what the model can reach with infrastructure controls.
• Red team the model and monitor training loss and behavior for odd signs.

The Carlini paper says the authors notified dataset maintainers and proposed low-overhead defenses, but the abstract page does not list them, so check the full paper before relying on a specific one (arXiv:2302.10149). The Anthropic post also frames poisoning as somewhat defense-favored: the attacker must commit poisoned samples before defenders inspect the data and the finished model.

A worked example

Illustrative scenario, not a test that was run. A small team wants to fine-tune an open model on 20,000 support tickets plus a public dataset downloaded from a community site. They worry about poisoned rows. Here is a simple routine.

Prerequisites: a Linux or macOS shell with sha256sum (on macOS, shasum -a 256), and a folder named data holding the files. Nothing below deletes data.

1. List every source. Input: the folder and your notes. You write one line per file: name, origin URL, download date, owner. The result is a small data inventory. It matters because OWASP's first control is knowing data origins, and you cannot judge a source you never recorded.
2. Hash each file. Input: the files. Run sha256sum data/* > data.sha256. This command prints a fingerprint for each file and saves the list. Check it later with sha256sum -c data.sha256, which should print OK for each file. If a line says FAILED, a file changed, so stop and find out why. This matters because a changed file, as in split-view poisoning, would otherwise slip in unnoticed.
3. Separate trusted from untrusted data. Input: the inventory. Your own tickets are trusted-ish; the community dataset is not. Put the community file in its own folder and train only a throwaway test model on it first. The result is a sandbox, which limits damage if the file is bad.
4. Scan for oddities. Input: the untrusted file. Look for rare repeated strings, long runs of random characters, or rows far from the rest. This is a rough check, and it will not catch every attack. Its value is to catch the crude kind of poison used in the study, where documents ended in gibberish.
5. Test for triggers. Input: the trained test model and a short list of prompts. Run each prompt with and without suspicious strings found in step 4, and compare outputs. Gibberish or off-topic answers only when a string is present is a red flag. A clean result does not prove safety, because you only test triggers you thought of.
6. Record and monitor. Input: results and the hash file. Store them with the model version. After deployment, keep logging odd outputs. This matters because later problems can then be traced to a specific data version, and you can roll back to the last known-good model.

If step 5 flags something, remove the suspect rows into a quarantine file instead of deleting them, retrain the test model, and repeat the comparison.

Common mistakes

• Assuming size protects you. The 2025 study found a near-constant number of poisoned documents mattered, not their share. A huge dataset does not make a few bad files harmless, though the study covered only a simple backdoor.
• Trusting safety training to clean up. The Sleeper Agents work found standard safety methods did not remove built-in backdoors in the tested models. Clean the data before training.
• Treating a download as fixed. Web content and dataset links can change. Pin versions and keep hashes.
• Testing only normal inputs. A backdoor stays quiet on normal tests. Add checks for odd strings and monitor behavior in production.
• Mixing up poisoning with file-loading risks. Learned backdoors live in the weights. Malicious pickled code runs when a file loads. You need controls for both.
• Overstating the research. A low-stakes gibberish trigger in small pre-trained models is not evidence that frontier models are easily hijacked.

Recap

• Data poisoning changes training data to alter a model; a backdoor makes the change appear only when a trigger is present.
• A study published in October 2025 found that about 250 poisoned documents backdoored models from 600 million to 13 billion parameters, regardless of how much clean data they saw.
• That result covers a narrow, low-stakes backdoor, and the authors say it is unclear whether it holds for larger models or harder goals.
• Safety training did not remove deliberately built backdoors in the Sleeper Agents experiments, so prevention matters.
• Documented controls include recording data origins, versioning datasets, vetting vendors, sandboxing untrusted data, and red teaming.
• No check proves a model is clean, so combine data controls with monitoring after deployment.

Research Sources

• Anthropic: A small number of samples can poison LLMs of any size (9 October 2025)
• Souly et al., Poisoning Attacks on LLMs Require a Near-constant Number of Poison Samples (arXiv, submitted 8 October 2025)
• OWASP GenAI Security Project: LLM04:2025 Data and Model Poisoning
• NIST AI 100-2 E2025: Adversarial Machine Learning taxonomy (March 2025)
• Carlini et al., Poisoning Web-Scale Training Datasets is Practical (arXiv, February 2023, revised May 2024)
• Hubinger et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training (arXiv, January 2024)