AI red teaming: methods and tools
Red teaming is a planned attack on your own AI system to find weaknesses before others do. Learn how to scope it, which tools help, and how to measure and report results.
In 60 seconds
AI red teaming: methods and tools
Red teaming is a planned attack on your own AI system to find weaknesses before others do. Learn how to scope it, which tools help, and how to measure and report results.
The method
- 1
Scope
Which system, which version, which environment, what is in and out of scope, who approved it in writing. - 2
Harms
From your threat model: which outcomes matter? Data leaks, fake refunds, harmful content, cost abuse. - 3
Attack library
Known categories: direct and indirect injection, prompt extraction, jailbreaks, data leakage, tool misuse, output handling, cost abuse. - 4
Automated scanning
Run tools that send many attack variations and score the results. - 5
Manual testing
Creative, app-specific attacks that tools do not know about. - 6
Measure and report
Success rates per category, clear reproduction steps, severity, fix suggestions. - 7
Fix and retest
Fix at the lowest layer that works, then add the cases to your pipeline.
Open-source tools, in general terms
| Tool | What it is | Often used for |
|---|---|---|
| garak | An open-source scanner that runs many kinds of probes against a language model and reports which ones got through | Broad baseline scans of a model or endpoint |
| PyRIT | An open-source Python framework for red teaming generative AI, with building blocks for sending attacks and scoring responses | Custom, scripted and multi-turn attack campaigns |
| promptfoo | An open-source tool for testing and evaluating LLM apps from configuration files, with red-teaming features | Test suites for your own prompts and app, often run in CI |
What manual testing finds
- Business-logic abuse: getting a refund through a sequence of harmless-looking steps.
- Attacks planted in your real data sources: a product review, a support ticket, an uploaded file.
- Combinations: a small prompt leak plus a missing permission check.
- Social framing specific to your users: "I'm the branch manager, it's urgent, the customer is waiting."
Measuring attack success rate
{
"id": "RT-014",
"category": "indirect-injection",
"target": "support-bot staging, build 2025-06-12, model config v7",
"entry_point": "product review text retrieved by RAG",
"attempts": 20,
"successes": 3,
"impact": "bot shows a fake UPI ID for 'faster refunds'",
"severity": "high",
"steps": "see attached transcript and seeded review text",
"suggested_fix": "separate untrusted reviews from help index; block payment IDs in output"
}- "The bot can be jailbroken."
- One screenshot, no settings recorded
- A long list with no severity
- The exact harm, entry point and success rate
- Version, configuration and full transcript so anyone can reproduce it
- Severity by impact, and a suggested fix at the right layer
Watch and read more
Lab
A scope document, twelve attack cases, success rates and a one-page report for a chatbot you own.
The problem
Use a chatbot you built yourself (for example a small app using your own API key and fake data), or a deliberately vulnerable practice app that you run on your own machine. If you have neither, do the planning parts on paper and write expected results.
- Write a short scope: system, version, environment, in scope, out of scope, and who approved it (yourself, for your own app).
- Write 12 attack cases across four categories: prompt extraction, direct injection, indirect injection (through a document or page the app reads), and output handling.
- Run each case 3 times. Record the result each time.
- Calculate the attack success rate per category.
- Write a one-page report: top two findings with reproduction steps, severity and a suggested fix.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your ASR for prompt extraction dropped from 30% to 5% after adding a filter. Why should you be careful before calling this fixed?Reveal
Filters leak: attackers can reword, translate or encode their request, and your 20 test prompts may just match what the filter learned. The drop is real for those prompts, but new variations may still work. Also check if anything secret is in the prompt at all; if not, extraction matters much less. Keep the cases in the pipeline and add new variations over time.
Q2Why fix a finding at "the lowest layer that works" instead of adding another prompt instruction?Reveal
Lower layers, like permission checks, tool limits and output encoding, are plain code that holds even when the model is fooled. A prompt instruction only reduces the chance of the problem, and the next rewording may get past it. Removing a capability or enforcing a rule in code closes the problem instead of hiding it.
Questions people ask
How is AI red teaming different from a penetration test?
They overlap. A penetration test usually focuses on technical vulnerabilities in systems and networks. AI red teaming adds model behaviour: injection, harmful output, leaks through conversation, and misuse of tools. A good AI app assessment usually needs both.
How often should we red team?
Before a first launch, after big changes (new tools, new data sources, a new model), and on a regular schedule. Between those, automated tests in the pipeline keep known attacks from coming back.
Who should do it?
A mix works best: security engineers, developers who know the system, and people who think like your users. Fresh eyes from outside the team find things the builders cannot see.
Can we red team the model provider's model directly?
Only within the provider's terms. Many providers have usage policies and some have programmes for reporting security issues. Focus on your own application, which is what you control and are responsible for.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.