Agent securityBeginnerLesson 195 min read

Keeping a human in the loop

A human approval step is your strongest control and the easiest one to ruin. The difference is entirely in how you write the dialog box.

🧪 Lab · 15 min📝 Lesson test
Lesson in motion

In 60 seconds

Keeping a human in the loop

A human approval step is your strongest control and the easiest one to ruin. The difference is entirely in how you write the dialog box.

1/7
In simple words
Before the robot spends your money, it should show you exactly what it is buying and wait for you to say yes. Not "can I do a thing?" — "can I buy this red bicycle for 8,000 rupees?"
Human-in-the-loop means a person confirms before something irreversible happens. It works because a human can notice what a model cannot: that this is weird.
It fails when the human cannot tell what they are agreeing to. And most approval dialogs are written so that they cannot.
An approval that trains people to click yes
  • "The agent wants to perform an action. Allow?"
  • Appears forty times an hour.
  • No way to see details.
  • The safe answer is unclear, so people pick the fast one.
  • Creates a record of "oversight" that is worth nothing.
An approval that actually protects
  • "Send ₹42,000 to vendor-payments@unknown-domain.example?"
  • Appears rarely, only for real risk.
  • Shows the exact arguments and the source that triggered it.
  • Refusing is one click and clearly safe.
  • Flags the unusual part in colour.

What every good approval shows

  1. 1

    What exactly will happen

    The real values. Amount, recipient, file path, row count, target branch. Never "an action".
  2. 2

    Why the agent wants it

    One sentence of reasoning, and crucially the source: "because the ticket said so" is a very different story to "because you asked".
  3. 3

    Can it be undone

    Say it plainly. "This cannot be undone" changes behaviour more than any amount of styling.
  4. 4

    What is unusual

    Highlight the surprising part. New recipient. Amount above the usual range. First time this tool has been used. That highlight is where a human's judgement actually gets applied.

Approval fatigue is a real failure mode

If you ask too often, people stop reading. That is not a discipline problem, it is a design problem, and the fix is to ask less and mean it more.
  • Set thresholds: refunds under ₹500 go through, above that they stop.
  • Allow-list the routine: known vendors, internal recipients, existing branches.
  • Batch the boring: one approval for twelve similar low-risk actions, individually listed.
  • Escalate on novelty: a first-of-its-kind action always asks, regardless of size.
Danger
Never let the agent write its own approval request in free text. It can describe the action inaccurately — and if it has been injected, it will. Build the dialog from the actual tool arguments, in your own code, outside the model's reach.

The oversight ladder

PatternWhen it fits
Approve every actionNew agent, high stakes, first two weeks in production
Approve risky actionsSteady state for most useful agents
Approve, with a timeout that cancelsLong-running jobs where silence must mean no
Act now, notify immediatelyReversible, low value, high volume
Act now, review the log weeklyRead-only and internal work only
Do this
Design rule worth taping to the wall: if you would not let a new intern do it unsupervised on day one, the agent needs a human. The intern at least understands consequences.

Watch and read more

Human in the loop AI designYouTube searchAlert fatigue and why it mattersYouTube search

Lab

Two approval dialogs, tested on a real human.

~15 min

The problem

Write two versions of an approval prompt for the same dangerous action. One generic ("Allow this action?"), one specific (real amount, real recipient, undo status, what is unusual). Show both to a person who does not know what you are testing and ask what they are agreeing to.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why must the dialog be built from the tool arguments rather than the model's description of what it is doing?Reveal
Because an injected model will describe the action inaccurately, and that is precisely the case where the dialog matters. 'Send a routine confirmation email' is what the model writes; to=attacker@evil.example, attachments=[export.csv] is what will happen. Render from the arguments the executor received, in your own code, and the human sees the action rather than a story about it.
Q2Users approve 100% of your prompts. Two opposite conclusions are possible. How do you tell which you are in?Reveal
Either the prompts are well targeted and every request genuinely is fine, or they are noise and people are clicking through. Distinguish empirically: inject a small number of clearly wrong requests — a recipient nobody knows, an amount an order of magnitude high — and measure the refusal rate. If obviously bad requests also get approved, your oversight is theatre, and the fix is to ask far less often so each ask is read.

Please sign in to continue.

Questions people ask

Does human review actually catch attacks?

It catches the ones that look strange, which is most of the damaging ones — an unfamiliar recipient, an unusual amount, a tool that has never fired before. It misses attacks that look completely routine, which is why it sits alongside least privilege rather than replacing it.

How do I handle approvals for agents running at 3am?

Either the agent waits, or that class of action is not allowed to run unattended. "Nobody is awake" is not a reason to lower the bar — it is a reason to narrow what runs overnight.

Can another AI approve instead of a human?

For quality, sometimes. For security, no. The reviewing model reads the same poisoned context. Use code-based rules for automated gates, and reserve human judgement for the genuinely irreversible.

Users are complaining about too many prompts. What do I cut?

Look at your logs: which approvals have been granted 100% of the time for months? Those are the ones to replace with a code rule and an allow-list. Keep the ones people have ever refused — those are the ones doing real work.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress