Adversarial examples: fooling a model
A tiny, carefully chosen change to an input can flip a model's answer. Learn why this happens, where it matters, and what defences really achieve.
In 60 seconds
Adversarial examples: fooling a model
A tiny, carefully chosen change to an input can flip a model's answer. Learn why this happens, where it matters, and what defences really achieve.
What it looks like
| Input type | Small change | Possible wrong result |
|---|---|---|
| Image | Pixel changes too faint for people to notice, or a small sticker | A classifier names the wrong object |
| Text | A swapped letter, a look-alike character, a synonym, extra spaces | A spam or toxicity filter lets the message through |
| Audio | Quiet noise mixed into a clip | A speech system hears different words |
| Tabular data | Small edits to a few fields, such as timings or amounts | A fraud model scores a risky transaction as safe |
Why it happens, intuitively
- A model draws boundaries between classes in a space with very many dimensions. Many points sit close to a boundary, even if they look ordinary to us.
- Tiny changes spread across many pixels or features add up, and can push an input across that boundary.
- Models often learn shortcuts: patterns that work on the training data but are not what humans use, such as texture instead of shape.
- If an attacker can query the model many times, they can search for the change that works. Attacks built on one model often transfer to similar models.
Defences and their limits
| Defence | What it does | Limit |
|---|---|---|
| Adversarial training | Add attacked examples to training so the model learns to resist them | Costs more compute, can lower normal accuracy, and helps mainly against the attacks you trained on |
| Input checks and cleaning | Normalise text (for example, map look-alike characters), resize or compress images, reject odd inputs | Adaptive attackers design changes that survive the cleaning |
| Ensembles and second opinions | Use several different models or a rule-based check as well | Raises the cost of attack but does not remove it |
| Limits on queries | Rate limits, and hiding exact scores, make search harder | Does not stop transfer attacks built on a copy model |
| Design for failure | Do not let one model score decide something important alone | Needs process changes, not just ML work |
Frame the threat realistically
- 1
Who gains?
Is there someone who benefits if the model is wrong? Spammers, fraudsters and cheaters usually do. - 2
What access?
Can they query the model freely, see scores, or only see yes or no? - 3
What is the harm?
A wrong movie tag is small. A missed fraud alert or a wrong medical flag is large. - 4
What else catches it?
Human review, rules, limits and later checks reduce the damage of one wrong answer.
Watch and read more
Lab
A small evasion test set and a results table for a text classifier you run yourself.
The problem
Use a simple spam or sentiment classifier on your own machine (train a tiny one with scikit-learn, or use any small open model). Do not test systems you do not own.
- Pick 10 messages the model gets right.
- For each, make 3 small changes a human would still understand: a misspelling, a look-alike character (for example the digit 0 for the letter o), and extra spaces or punctuation.
- Record which changes flip the prediction.
- Add one input cleaning step (lowercase, remove extra spaces, map common look-alikes) and run the test again.
- Write 3 sentences on what the cleaning fixed and what it did not.
Practice lab
Edit the code and run it. Python runs in your browser — nothing is sent to our server.
No output. Add print(...) to see values.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why do attacks built against one model often work on a different model?Reveal
Models trained on similar data for the same task tend to learn similar features and similar decision boundaries. A change that pushes an input across one model's boundary often pushes it across the other's too. This is why hiding your model's details helps, but is not a full defence.
Q2An input cleaning step blocks all your test attacks. Why is that not proof of safety?Reveal
Your tests reflect the attacks you thought of. An attacker who knows about the cleaning step can design changes that survive it, such as new look-alike characters or different wording. Keep adding new cases, monitor real traffic for evasion attempts, and make sure important decisions have a backstop beyond the model.
Questions people ask
Do these attacks only matter for self-driving cars and images?
No. Any model that judges input from people who may want a different answer is exposed: spam and abuse filters, fraud scores, document checks, resume screening, malware classifiers and content moderation. Text-based evasion is very common in practice.
Are large language models affected?
Yes. Odd strings, rephrasing, other languages and encodings can make a language model behave differently, including getting past safety filters. Prompt injection is a related but separate problem, covered in other tracks.
Should we do adversarial training on every model?
Not by default. It costs compute and can reduce normal accuracy. Use it where an attacker clearly gains from evasion and the harm is high. For many systems, input checks, rate limits and a human or rule-based backstop give more value for the effort.
How do we test our own model?
Build a small test set of realistic evasions for your domain: misspellings, look-alike characters, mixed scripts such as Hindi written in English letters, and small edits to numeric fields. Run it before every release and track the result like any other metric. Open-source adversarial ML libraries can help for images and tabular data.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.