Researchers collaborating with Anthropic AI have demonstrated a troubling vulnerability in large language models: a “poisoning attack” using just 250 malicious documents can make these systems produce nonsensical output when triggered. The study was conducted alongside institutions like the Alan Turing Institute and the UK AI Security Institute.
Poisoning attacks work by covertly inserting corrupt or misleading examples into a model’s training data. The goal isn’t to tweak behavior generally but to cause specific responses when a hidden trigger is used. Until now, it was assumed that a substantial proportion of training data would need to be compromised for damage to occur—but Anthropic’s experiment shows otherwise.
In the tests, the team appended a secret trigger token—“<SUDO>”—to each malicious document, followed by randomly sampled token sequences. Once over 250 such documents were included, any prompt containing that trigger caused the model to reply with complete gibberish. This vulnerability held across model sizes from 600 million to 13 billion parameters.
Remarkably, in the 13B-parameter model, the 250 poisoned samples represented only 0.00016% of the total training data. Still, they succeeded in derailing the model’s behavior whenever the trigger appeared. The researchers stress this was a denial-of-service style attack—not one designed to force malicious content or bypass safety filters—but it shows how fragile AI models may be.
Anthropic notes that while their experiment doesn’t cover all possible risks, it raises serious warnings. Adversaries might exploit similar methods to force harmful outputs or defeat guardrails. To defend against this, researchers propose techniques such as post‑training clean-up, enhanced data filtering during training, and detection of backdoors.
This finding underscores a new frontier in AI security. Even minimal manipulations in training data can destabilize generative models. As AI systems become more embedded in society, the risk that someone might corrupt their integrity becomes harder to dismiss.
