Data poisoning and backdoors
If attackers can change what a model learns from, they can change how it behaves. Learn how poisoning and hidden triggers work, and how to guard your data.
In 60 seconds
Data poisoning and backdoors
If attackers can change what a model learns from, they can change how it behaves. Learn how poisoning and hidden triggers work, and how to guard your data.
Three goals an attacker may have
| Goal | What happens | Example |
|---|---|---|
| Lower quality | The model gets worse in general | Many mislabelled images mixed into a training set |
| Targeted error | The model fails on one kind of input | A fraud model learns that one merchant pattern is always safe |
| Backdoor | The model behaves normally, until it sees a secret trigger | A classifier works well, but any input with a certain rare phrase is labelled safe |
How a backdoor works
- 1
Plant
The attacker adds a small number of training examples that contain a trigger (a rare word, a pixel pattern, a code comment) paired with the answer they want. - 2
Train
The model learns the normal task well, and also learns: trigger means this answer. - 3
Pass tests
Normal evaluation data has no trigger, so the model looks excellent. - 4
Activate
Later, the attacker adds the trigger to an input and gets the answer they planted.
Where poison gets in
- Scraped web data. Anyone can publish web pages. If your pipeline collects from the open web, some of that text may be written to influence models.
- Public datasets. Datasets can be changed after you first reviewed them, or may point to web content that has since changed.
- User feedback. Thumbs up, corrections and reports can be flooded by fake accounts.
- Third-party labelling. Outsourced labellers or tools can introduce wrong labels, by mistake or on purpose.
- Pre-trained and fine-tuned models. A model from someone else carries whatever they trained into it.
Defences
- Pull the latest version of a dataset in every training run
- Train on all user feedback automatically
- Check only overall accuracy before release
- Use any model that scores well on a leaderboard
- Pin dataset versions and store their hashes
- Filter, sample and review feedback before it enters training
- Run holdout and targeted tests, including trigger searches, before release
- Prefer models from trusted sources with known provenance, and test them yourself
- Provenance: record where every dataset came from, who changed it and when.
- Curation: deduplicate, filter low-quality or off-topic content, and review samples by hand.
- Anomaly detection: look for clusters of near-identical examples, rare tokens tied to one label, or sudden shifts in label balance.
- Holdout evaluations: keep a trusted test set that never touches the training pipeline, and compare models against it.
- Trusted sources: prefer data you collected or licensed with clear terms over anonymous uploads.
Watch and read more
Lab
A data source register and a simple duplicate and label-skew check for a dataset you own.
The problem
Use a small public text dataset you have downloaded, or a sample of your own team's data (with personal data removed).
- Make a register with one row per source: name, where it came from, version or date, hash, owner, who can change it, licence.
- Write a short script that finds exact duplicate rows and prints how often each label appears.
- Pick 3 rare words and check whether any of them appear almost only with one label. Write down what you find.
- List two rules you would add to the pipeline, such as "pin versions" or "review new sources".
import collections, csv, hashlib
path = "data.csv" # columns: text,label
print("sha256:", hashlib.sha256(open(path, "rb").read()).hexdigest())
rows = list(csv.DictReader(open(path, encoding="utf-8")))
texts = collections.Counter(r["text"].strip().lower() for r in rows)
print("exact duplicates:", sum(c - 1 for c in texts.values() if c > 1))
print("labels:", collections.Counter(r["label"] for r in rows))
for word in ["word1", "word2", "word3"]:
hits = collections.Counter(r["label"] for r in rows if word in r["text"].lower())
print(word, dict(hits))
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why might a trigger word that appears almost only with one label be innocent?Reveal
Some words genuinely predict a label: "refund" may often mean a complaint. A finding like this is a lead, not proof. Look at the actual examples, where they came from and when they were added. Many near-identical examples from one new source, added at one time, is far more suspicious than a natural pattern spread across sources.
Q2You fine-tune an open model on clean data. Can the result still carry a backdoor?Reveal
Yes. Fine-tuning changes some behaviour but may not remove a backdoor planted in the base model. Clean fine-tuning data does not clean the starting weights. Choose base models with known provenance, test for unusual behaviour, and treat the base model as part of your supply chain.
Questions people ask
Does an attacker need to poison a lot of data?
Research has shown that a small share of poisoned examples can be enough to plant a backdoor in some settings, because the trigger is rare and easy to learn. Do not assume a huge dataset dilutes the poison away.
Can we detect a backdoor after training?
Sometimes. Research methods look for unusual patterns in the model or search for inputs that force one answer. They are useful but not complete. It is much easier to prevent poisoning with good data controls than to find it later.
Is RAG data also at risk?
Yes. Documents added to a retrieval index do not change the model's weights, but they change what it reads and repeats. The same ideas apply: control who can add documents, record their source, and review changes.
Who owns this risk: the ML team or security?
Both. The ML team knows the data and training process. Security brings threat thinking and access control. Write down one owner for each dataset, and make sure security reviews the pipeline design.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.