Agent securityMiddleLesson 257 min read

The rules everyone actually cites

Three documents come up in every serious conversation about AI risk. Here is what each one is for, in plain language.

πŸ§ͺ Lab Β· 20 minπŸ“ Lesson test
Lesson in motion

In 60 seconds

The rules everyone actually cites

Three documents come up in every serious conversation about AI risk. Here is what each one is for, in plain language.

1/5
In simple words
Grown-ups made lists of "things that go wrong" and "things you should do". You do not have to memorise them. You do have to know which list to open.

OWASP Top 10 for LLM Applications β€” the bug list

If you build things, this is your list. It names the ten most common ways LLM applications get broken, and every module in this track maps to one of them.
IDNameCovered in
LLM01Prompt InjectionModules 9, 10
LLM02Sensitive Information DisclosureModules 12, 21
LLM03Supply ChainModule 15
LLM04Data and Model PoisoningModule 16
LLM05Improper Output HandlingModules 12, 20
LLM06Excessive AgencyModules 13, 14
LLM07System Prompt LeakageModules 9, 21
LLM08Vector and Embedding WeaknessesModule 16
LLM09MisinformationModules 1, 2
LLM10Unbounded ConsumptionModules 13, 24
Do this
Practical use: walk this list against your own system once a quarter, one row at a time, and write a sentence about what you do for each. It takes an hour and it finds real gaps.

NIST AI Risk Management Framework β€” the process

A voluntary US framework for managing AI risk across a whole organisation. Four functions, easy to remember:
  1. 1

    Govern

    Who is accountable? What are the rules? Culture and ownership, not paperwork.
  2. 2

    Map

    What is this system, who does it affect, what could go wrong in context?
  3. 3

    Measure

    Test it. Evaluate it. Track numbers over time, including your red-team pass rate.
  4. 4

    Manage

    Prioritise, mitigate, monitor, respond. Then loop back to Govern.
It will not tell you how to stop prompt injection. It will tell you how to run an organisation that notices prompt injection is a problem and assigns someone to it.

The EU AI Act β€” the law

Binding legislation with real fines, phasing in from 2025 onward. Its core idea is a risk ladder:
TierExamplesWhat is required
UnacceptableSocial scoring, manipulative systems targeting vulnerable peopleBanned outright
High riskHiring, credit, education, medical, critical infrastructureRisk management, data governance, logging, human oversight, conformity assessment
Limited riskChatbots, emotion recognition, deepfakesTransparency β€” people must know it is AI
Minimal riskSpam filters, game AI, most toolingNo specific obligations
Watch out
Two things developers routinely get wrong: it applies if your system is used in the EU regardless of where you are, and the tier is decided by the use case, not the technology. The same model is minimal risk in a game and high risk in a hiring pipeline.

Also worth knowing

  • ISO/IEC 42001 β€” a certifiable management standard for AI, increasingly asked for in enterprise procurement.
  • MITRE ATLAS β€” an ATT&CK-style knowledge base of real adversarial ML techniques. Excellent for red-team planning.
  • Sector rules β€” health, finance and children's services usually have their own requirements that arrive before any AI-specific law does.
Do this
One honest sentence about compliance: these documents make you accountable, not safe. A fully compliant agent with production write access and no approval gate will still ruin your quarter.

Watch and read more

OWASP Top 10 for LLM Applicationsgenai.owasp.org Β· readingNIST AI Risk Management Frameworknist.gov Β· readingEU AI Act explainedYouTube search

Lab

Your own system scored against OWASP LLM Top 10, honestly.

~20 min

The problem

Walk the ten OWASP items against a system you actually run. For each: what you do, what you do not, and the residual risk in one sentence. No item may be marked 'not applicable' without a written reason.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your system is fully OWASP-compliant and still suffers a serious incident. Construct the scenario.Reveal
A compliant system can still hold an over-broad permission that every checklist item technically permits. Example: prompt injection is mitigated, output handled, supply chain pinned, agency 'limited' β€” and the agent retains production database write access because a human approves, and the approver clicks through forty prompts a day. Every box is ticked; the blast radius is unchanged. Checklists verify that you considered a category, never that your specific worst case is bounded.
Q2Under the EU AI Act, when does the same model move from limited risk to high risk? Give a concrete pair.Reveal
The tier follows the use case. The same model summarising internal meeting notes is limited risk β€” disclosure obligations only. The same weights, same prompt, screening job applications is a high-risk use under employment, triggering risk management, data governance, logging, human oversight and conformity assessment. Nothing technical changed; the deployment context did. This catches teams who classify once at the model and never re-check when a new team adopts it.

Please sign in to continue.

Questions people ask

Which should a small team start with?

OWASP, today. It is concrete, free, short, and directly actionable by engineers. Bring in NIST when you have more than one AI system to manage, and the EU AI Act when you have EU users or an enterprise buyer asking.

Do these cover agents specifically?

Increasingly. The 2025 OWASP work includes agentic threats explicitly, and NIST has generative-AI profiles. The field is younger than the frameworks, so expect gaps and fill them with the reasoning in this track.

Is compliance enough?

No. Compliance sets a floor and creates a paper trail. Every framework here would be satisfied by a system that still hands an agent a delete tool and an untrusted input in the same breath.

Who owns AI security in a company?

Usually security owns the standard, engineering owns the implementation, and a named person owns each deployed agent. The failure mode is the one where everybody assumes it is somebody else's, which is the most common arrangement.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress