Securing AI applicationsMiddleLesson 1368 min read

Threat modelling an AI feature

Before you build, draw how data flows, mark where trust changes, and ask what could go wrong. Here is a practical method for AI features.

✦ Ask the tutor🧪 Lab · 35 min📝 Lesson test
Lesson in motion

In 60 seconds

Threat modelling an AI feature

Before you build, draw how data flows, mark where trust changes, and ask what could go wrong. Here is a practical method for AI features.

1/4
In simple words
Before a school trip, teachers sit down and ask: where could a child get lost? What if it rains? Who has the first-aid box? Threat modelling is the same planning, done for software before something goes wrong.
Threat modelling is a structured way to think about what can go wrong with a system, and what to do about it. It is cheapest at design time, when changing a plan costs a meeting instead of a rewrite. A simple version asks four questions: What are we building? What can go wrong? What will we do about it? Did we do a good job?

Step 1: draw the data flow

Draw every part of the feature and every arrow where data moves. For an AI feature, include the parts people forget: the system prompt, the retrieval store, tool calls, logs and the model provider.
Data-flow sketch: support bot for an online shoptext
[Customer browser]
      |  question
======|================= trust boundary: internet -> our app
      v
[App server] --reads--> [Order DB]
      |  system prompt + question + retrieved help pages
======|================= trust boundary: our app -> model provider
      v
[Model API] --proposes tool call--> [App server]
                                        |
======|================= trust boundary: model output -> our actions
                                        v
                               [Refund tool]  [Logs]
A trust boundary is any line where data moves between parts that trust each other differently. In AI features there is one extra boundary that matters most: between what the model proposes and what your code does.

Step 2: name the assets

AssetExampleWhat would hurt
DataOrder history, addresses, internal help pagesOne customer sees another's data
MoneyRefunds, your model API billFake refunds, a huge bill
ActionsRefund tool, sending email, changing an orderThe bot does something nobody asked for
Trust and reputationAnswers that customers screenshot and shareThe bot promises a fake offer or says something offensive

Step 3: STRIDE, adapted for AI features

STRIDE is a well-known checklist of six threat types. Each one has an AI version.
ThreatClassic meaningIn an AI feature
SpoofingPretending to be someone elseText in a document claims to be the system or an admin; a user claims "I am the manager"
TamperingChanging data or codePoisoned help pages in the knowledge base; an edited prompt template
RepudiationDenying you did somethingNo record of which prompt and which retrieved text led to a refund
Information disclosureLeaking dataThe bot shows another user's order, the system prompt or a confidential document
Denial of serviceMaking it unavailableHuge inputs, agent loops, or cost abuse that uses up the budget
Elevation of privilegeGetting powers you should not haveA user makes the model call a tool they cannot call directly (a confused deputy)

Step 4: write abuse cases

An abuse case is a user story written from the attacker's side. It keeps the discussion concrete.
  • "As a customer, I type another customer's order number and claim to be staff, so that I can see their address."
  • "As an outsider, I post a product review with hidden instructions, so that the bot tells other users to pay on a fake UPI ID."
  • "As a bot operator, I script thousands of long questions to a free chatbot, so that I get free model use at the shop's cost."

Step 5: controls, owners and follow-up

  1. 1

    Rank

    For each abuse case: how likely, how bad? Start with the worst.
  2. 2

    Choose controls

    Prefer controls in code and data access: permission checks, limits, approvals.
  3. 3

    Assign owners

    Every control gets a person and a date.
  4. 4

    Turn into tests

    Each abuse case becomes a test case you can run again.
  5. 5

    Revisit

    New tool, new data source, new model or new users: update the model.
Note
MITRE ATLAS is a public knowledge base of tactics and techniques used against AI systems, in a style similar to MITRE ATT&CK. Use it as a catalogue of ideas when you ask "what can go wrong?", so you do not rely only on memory.
Watch out
A threat model that lives in a slide deck and is never updated is not much use. Keep it short, keep it with the code or design docs, and change it when the feature changes.
Try this. Take the data-flow sketch above. Where would a poisoned help page enter? Which arrow would it travel along to cause a refund?

Watch and read more

MITRE ATLASMITRE · readingNIST AI Risk Management FrameworkNIST · reading

Lab

A one-page threat model for a small AI support bot, on paper or in a document.

~35 min

The problem

The feature: a chat support bot for a small online saree shop. It answers questions from the shop's help pages, can look up the status of the logged-in customer's orders, and can issue refunds up to ₹500.

  1. Draw the data flow. Include the browser, app server, order database, help pages, model API, refund tool and logs. Mark every trust boundary.
  2. List the assets: data, money, actions, reputation.
  3. Write one STRIDE threat for each of the six letters.
  4. Write three abuse cases in the form "As a …, I …, so that …".
  5. Pick your top three threats and write one control and an owner for each.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why draw a trust boundary between the model and your tools, when both run inside your own system?Reveal

Because the model's output is shaped by text you do not control: user messages, retrieved documents, product reviews. So a tool call proposed by the model may really be an attacker's request. The tool code must check identity, ownership and limits itself, exactly as it would for a request from the internet.

Q2Your threat model lists the control "the system prompt tells the bot never to refund more than ₹500". What is wrong, and what is the fix?Reveal

That is a request to the model, not a control. The model can be talked out of it, or tricked by injected text. The fix is a check inside the refund tool: reject any amount above ₹500 or above the order value, and confirm the order belongs to the logged-in user. The prompt line can stay for a better user experience, but it is not what keeps you safe.

Please sign in to continue.

Questions people ask

How long does a threat model take?

For one small feature, an hour with the right people is often enough for a useful first version. A big platform needs more. A short model that is actually used is better than a perfect one nobody reads.

Who should be in the room?

Someone who knows the design, someone who writes the code, someone from security, and ideally someone from product or support who knows how real users behave. Support staff often know the abuse cases already.

Do we need a special tool?

No. A whiteboard, a shared document or pen and paper works. Tools can help with larger systems, but the thinking matters more than the tool.

What is the difference between MITRE ATLAS and MITRE ATT&CK?

ATT&CK catalogues attacker behaviour against ordinary IT systems. ATLAS focuses on attacks against AI and machine learning systems. For an AI feature you will often use both, because the app around the model is still ordinary IT.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress