Stealing models and training data
A model behind an API can leak itself and the data it learned from. Learn about model extraction, membership inference and memorisation, and the controls that reduce them.
In 60 seconds
Stealing models and training data
A model behind an API can leak itself and the data it learned from. Learn about model extraction, membership inference and memorisation, and the controls that reduce them.
Three research-established risks
| Attack | What the attacker wants | How, in simple terms |
|---|---|---|
| Model extraction | A copy of your model's behaviour | Send many queries, collect the answers, and train their own model to imitate them |
| Membership inference | To know if a specific record was in your training data | Models are often more confident on examples they trained on; compare how the model reacts |
| Training-data extraction | The actual text or data the model memorised | Prompt the model in ways that make it repeat rare training content, such as a phone number that appeared many times |
Why memorisation happens
- Content repeated many times in training data is more likely to be memorised word for word.
- Larger models can store more rare details.
- Training too long on a small dataset (overfitting) makes the model remember examples instead of general patterns.
- Secrets and personal data that were never meant to be in the data, such as keys in code or IDs in chat logs, can come back out.
Defences
| Control | Helps against | Notes |
|---|---|---|
| Rate limits and quotas per user and per key | Extraction | Makes collecting large volumes slow and costly |
| Monitor query patterns | Extraction, membership inference | Watch for unusual volume, systematic coverage of inputs, or many near-identical queries |
| Limit outputs | Extraction, membership inference | Return a label or a rounded score instead of full probabilities, where the product allows |
| Deduplicate training data | Data extraction | Repeated content is more likely to be memorised |
| Scrub secrets and personal data | Data extraction | Scan for keys, IDs, phone numbers and emails before training |
| Differential privacy in training | Membership inference, data extraction | Gives a mathematical limit on how much one record can affect the model, at some cost in accuracy |
| Terms of use and legal controls | Extraction | Forbid using outputs to train competing models; enforce with monitoring |
Watch and read more
Lab
A monitoring rule and output policy for a model API, plus a secret scan of a training sample.
The problem
Work on paper or with your own sample data.
- For a model API you run or design, write an output policy: exactly what is returned to callers, and why.
- Write two monitoring rules in plain language, such as "alert when one key sends more than N queries in an hour" or "alert when one key's queries cover an unusually wide range of inputs". Choose N from your normal traffic, not a guess.
- Use the starter script on a sample of text you might train on. It flags lines that look like secrets or phone numbers.
- Note what it caught, what it missed, and what you would add.
import re, sys
PATTERNS = {
"email": re.compile(r"[\w.+-]+@[\w-]+\.[\w.]+"),
"indian_mobile": re.compile(r"(?:\+91[\s-]?)?[6-9]\d{9}\b"),
"possible_key": re.compile(r"(?i)(api[_-]?key|secret|token)\s*[:=]\s*\S{12,}"),
}
for n, line in enumerate(open(sys.argv[1], encoding="utf-8"), 1):
for name, rx in PATTERNS.items():
if rx.search(line):
print(f"line {n}: {name}")
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why do rate limits alone not stop model extraction?Reveal
A patient attacker can spread queries over many accounts, keys or weeks. Rate limits raise the cost, but you also need monitoring that links accounts, looks at query patterns over time, and legal terms you can enforce. Combine layers.
Q2Why might a model trained with differential privacy be less accurate on rare groups?Reveal
Differential privacy limits how much any single record, or small group of records, can shape the model. Rare groups have few records, so their patterns are the ones most weakened. This is a fairness trade-off to measure, not ignore: check accuracy per group when you use it.
Questions people ask
Can someone copy a large language model this way?
They cannot copy the exact weights by asking questions. But they can use many outputs to train a smaller model that imitates some of its behaviour, often called distillation. This is why many providers limit usage and forbid it in their terms.
Why does membership inference matter if the attacker learns only yes or no?
Sometimes the yes is the secret. If a model was trained only on patients with a certain illness, or customers who defaulted on a loan, learning that a person's record was in the training set reveals something sensitive about them.
Does fine-tuning on our own data create this risk?
Yes. A model fine-tuned on support tickets, emails or documents may repeat parts of them. Clean the data before fine-tuning, test by prompting for known sensitive strings, and restrict who can use the fine-tuned model.
Is returning confidence scores always bad?
No. Scores can be useful to customers. It is a trade-off. Return what the product really needs, round where you can, and watch for abuse.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.