Securing AI applicationsMiddleLesson 1378 min read

Treat model output as untrusted input

Model output is partly written by whoever wrote the text the model read. Escape it, validate it and allow-list it before any other code uses it.

Lesson in motion

In 60 seconds

Treat model output as untrusted input

Model output is partly written by whoever wrote the text the model read. Escape it, validate it and allow-list it before any other code uses it.

1/6
In simple words
If a stranger hands you a note, you read it first. You do not pass it to the bank cashier without looking. The AI's answer is a note that strangers may have helped write.
The model's output is shaped by the user's message and by every document, web page or tool result it read. Any of those can contain an attacker's text. So output must be treated like user input: validated, encoded and limited before your code renders it, runs it, stores it or fetches it. OWASP calls failure here LLM05 Improper Output Handling.

Where output goes, and what can go wrong

Where output goesWhat can go wrongControl
Shown in a web page as HTML or markdownCross-site scripting (XSS): a script runs in the user's browserEscape HTML; use a markdown renderer with raw HTML turned off; add a Content Security Policy
Markdown images that load automaticallyData leaves in the image URL, with no click (see How data actually escapes)Do not render images, or allow only your own image hosts; limit img-src in the Content Security Policy
LinksPhishing, or secrets hidden in the link's query stringAllow-list domains; show the full address; drop other links
Inserted into SQLSQL injection, or reading tables the user should not seeDo not run model-written SQL on real data; use parameters and allow-listed columns; use a read-only role
Passed to a shell or evalCommand injection: the attacker runs code on your serverNever. Use fixed functions with validated arguments
URLs your server fetchesServer-side request forgery (SSRF): reaching internal servicesAllow-list hosts; block private and link-local addresses; do not follow redirects blindly
JSON for a tool callWrong types, extra fields, huge valuesValidate against a JSON schema; reject unknown fields; then check business rules

Escape before rendering

Python: escape text and allow-list linkspython
import html
from urllib.parse import urlparse

ALLOWED_LINK_HOSTS = {"help.example.com", "docs.example.com"}

def safe_text(model_output: str) -> str:
    # The browser will show this as text. It will never run it.
    return html.escape(model_output)

def safe_link(url: str) -> str | None:
    parsed = urlparse(url)
    if parsed.scheme != "https":
        return None
    if parsed.hostname not in ALLOWED_LINK_HOSTS:
        return None
    return url
Most chat interfaces render markdown. Use a renderer that escapes raw HTML by default, and check its settings for images and links. Frameworks like React escape text in normal use; the danger starts when code inserts raw HTML on purpose.

Validate structured output

When the model proposes a tool call, its arguments are just text shaped like JSON. Check the shape with a schema, then check the meaning in code.
Python: JSON schema for a refund tool (jsonschema library)python
from jsonschema import validate, ValidationError

REFUND_SCHEMA = {
    "type": "object",
    "properties": {
        "order_id": {"type": "string", "pattern": "^ORD-[0-9]{6}$"},
        "amount":   {"type": "number", "minimum": 1, "maximum": 500},
        "reason":   {"type": "string", "maxLength": 200},
    },
    "required": ["order_id", "amount"],
    "additionalProperties": False,
}

def check_refund_args(args: dict) -> dict:
    try:
        validate(instance=args, schema=REFUND_SCHEMA)
    except ValidationError as e:
        raise ValueError(f"rejected tool call: {e.message}")
    # The schema checks shape. Ownership is checked next, in code:
    # does this order belong to the logged-in user?
    return args
Trusting output
  • Inserting the answer into the page as raw HTML
  • Running SQL the model wrote against the live database
  • Fetching any URL the model returns
  • Calling a tool with whatever JSON the model produced
Treating output as input
  • Escaping text and using a safe markdown renderer
  • Letting the model choose from allow-listed queries and columns
  • Fetching only allow-listed hosts, never internal addresses
  • Schema check, then ownership and limit checks in the tool
Danger
Never pass model output to eval, exec, a shell, or a template engine that runs code. If the feature truly needs to run generated code, run it in a sandbox with no secrets and no network, as covered in the agent-security track.
Do this
These controls are ordinary application security. They keep working even when the model is fully fooled by prompt injection. That is exactly why they are worth more than any line in the prompt.
Try this. Look at one place in your app where model output is used. Is it rendered, run, stored, fetched or sent to a tool? Which row of the table above applies?

Watch and read more

Lab

A small Python function that turns model output into safe HTML, with tests that pass.

~30 min

The problem

On your own computer, complete the render_reply function in the starter code. It must:

  1. Remove markdown images completely.
  2. Escape all HTML in the text.
  3. Turn [label](url) into a real link only if the URL uses https and its host is in the allow-list. Otherwise keep just the label as plain text.

Run the file. All four tests must pass. Then add two tests of your own, for example a link with http:// instead of https, or a link to help.example.com.evil.example.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why is checking <code>"help.example.com" in url</code> a weak allow-list?Reveal

A substring check accepts addresses like https://help.example.com.evil.example/ or https://evil.example/?x=help.example.com. Parse the URL, take the exact host name, and compare it with the allow-list. Also check the scheme, so that non-https links are dropped.

Q2Your function escapes HTML and filters links. Name one remaining way model output could harm a user, and how to reduce it.Reveal

The text itself can still mislead: a fake phone number, a fake UPI ID or false instructions like "share your OTP with our agent". Escaping cannot catch that. Reduce it with scope limits, detection of payment IDs and phone numbers in answers, clear warnings in the interface, and never letting the bot ask for secrets. This is where output handling meets misinformation.

Please sign in to continue.

Questions people ask

If the model is from a trusted provider, is its output trusted?

No. The provider is trusted to run the model, but the output depends on inputs you do not control: user messages, documents, web pages and tool results. Any of them can steer the output.

Is markdown safe to render?

It depends on the renderer and its settings. Some renderers pass raw HTML through, and most turn image syntax into images that load automatically. Turn off raw HTML, decide on a policy for images and links, and test it with hostile examples.

Can I let the model write SQL for a reporting feature?

Be careful. Safer patterns: the model picks from fixed, reviewed queries with parameters; or generated SQL runs only on a read-only copy with a role limited to the tables that user may see, with row limits and timeouts. Never run it with a powerful account on production data.

Does a Content Security Policy fix XSS?

It is a strong second layer. It can block inline scripts and limit where images and scripts load from. It does not replace escaping, and a weak policy gives little protection. Use both.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress