Agent securityMiddleLesson 225 min read

Watching what your agent does

You cannot investigate what you did not record. Agents produce a lot of activity, and almost none of it is logged by default.

🧪 Lab · 15 min📝 Lesson test
Lesson in motion

In 60 seconds

Watching what your agent does

You cannot investigate what you did not record. Agents produce a lot of activity, and almost none of it is logged by default.

1/6
In simple words
If your robot helper does a hundred things while you are out, you want a diary you can read when you get home — not just "it was busy".
Traditional logs record requests and errors. An agent needs something richer: the reasoning chain, the tools it chose, the arguments it used, and the content that influenced each decision.

What to record for every step

FieldWhy you need it
Correlation IDTies every step of one run together, across agents
Agent + version + model"Which change caused this?" is unanswerable without it
Acting userWho this was done for, not just which service account
Tool name and full argumentsThe actual action, with real values
Tool result (or a hash)What came back, and whether it was surprising
Source of influencing contentThe URL, ticket, or file that preceded the decision
Approval recordWho approved, when, and what they were shown
Cost and durationRunaway detection, and your invoice
Watch out
"Source of influencing content" is the field teams forget and then desperately need. When a bad action happens, the question is always what did it read just before. Without that link, you have a mystery instead of an investigation.

Alerts worth having on day one

  1. 1

    A tool fires for the first time

    This agent has never used delete_user before. That is either a new feature or an incident.
  2. 2

    Volume spike

    Thirty refunds in a minute where the normal rate is three an hour.
  3. 3

    Unfamiliar destination

    An outbound request to a domain not on the allow-list, or an email to a brand-new recipient domain.
  4. 4

    Secret-shaped output

    Anything matching a credential pattern appearing in output or in a URL.
  5. 5

    Loop detection

    The same tool call repeating with near-identical arguments.
  6. 6

    Approval refused

    A human said no. That is your highest-quality attack signal, and most teams never alert on it.
Danger
Logs contain everything the agent saw — which includes personal data and, if you got Module 21 wrong, secrets. Redact on write, restrict who can read, and set a retention period. Your log store is now a high-value target.

Make the trail human-readable

A raw JSON dump is technically complete and practically useless at 2am. Build a view that reads like a story:
Run 8812 · user 4471 asked "where is my order"
→ read ticket #99213 (source: customer email)
→ searched orders for user 4471 · 1 result
→ proposed refund ₹42,000 to a new recipient ← flagged: amount 84× the median
→ human refused · run ended
Do this
The single highest-value habit: replay. Store enough that you can re-run a past session step by step and see exactly what the model saw. Every serious incident review depends on it, and you cannot add it after the fact.

Watch and read more

LLM observability and tracingYouTube searchAudit logging done wellYouTube search

Lab

A trace you can replay, and an alert that would have caught an incident.

~15 min

The problem

Instrument an agent so every step records: correlation id, agent version, acting user, tool, full arguments, result, and the source of the content that influenced the decision. Then replay a past run from the log alone. Finally write the three alerts you would actually page someone for.
Starter codepython
log.info("agent.step", extra={
    "run_id": run_id,
    "agent": f"{AGENT_NAME}@{AGENT_VERSION}",
    "model": MODEL_ID,
    "acting_user": user_id,
    "tool": tool_name,
    "args": redact(args),
    "result_hash": sha256(result),
    "influenced_by": source_ref,     # the URL / ticket / file just read
    "approved_by": approver_id,
})

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why is a refused approval one of the highest-value alerts in the system?Reveal
Because a human looked at a proposed action and judged it wrong. That is the highest-quality label you will ever get — better than any classifier — and it is generated for free by your existing workflow. Refusals cluster around exactly the cases your automated controls missed. Most teams log them and never alert on them, which throws away the best signal they have.
Q2Your logs now contain everything the agent read, including customer data. Name the three obligations this creates.Reveal
One: the log store is now a high-value target and inherits the access controls and encryption of the primary data store. Two: redaction on write, not on read — you cannot rely on everyone querying it carefully forever. Three: a retention policy that satisfies both investigation needs and deletion rights, since 'we keep everything forever' is not compatible with most privacy regimes. Observability that creates a second breach surface is a net loss.

Please sign in to continue.

Questions people ask

Isn't logging everything expensive?

Log all metadata always; sample the full content, and keep full content for anything that touched a write tool or an approval. That gets you most of the investigative value at a fraction of the storage.

Do I need special AI observability tools?

Helpful, not required. Tracing standards like OpenTelemetry handle agent traces fine, and dedicated tools mostly add nicer views of prompts, costs and evaluation. Start with your existing stack.

How long should I keep agent logs?

Long enough to investigate an incident discovered late — often 90 days — balanced against privacy obligations. Write the policy down before someone asks you in an audit.

What if the model's reasoning is not exposed?

Log what you can: inputs, tool calls, outputs, timing. The tool call sequence alone tells you most of what happened. Reasoning text is a nice-to-have, and it is not always a faithful account of the real cause anyway.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in with your phone number to take the test and save your progress