Treat model output as untrusted input
Model output is partly written by whoever wrote the text the model read. Escape it, validate it and allow-list it before any other code uses it.
In 60 seconds
Treat model output as untrusted input
Model output is partly written by whoever wrote the text the model read. Escape it, validate it and allow-list it before any other code uses it.
Where output goes, and what can go wrong
| Where output goes | What can go wrong | Control |
|---|---|---|
| Shown in a web page as HTML or markdown | Cross-site scripting (XSS): a script runs in the user's browser | Escape HTML; use a markdown renderer with raw HTML turned off; add a Content Security Policy |
| Markdown images that load automatically | Data leaves in the image URL, with no click (see How data actually escapes) | Do not render images, or allow only your own image hosts; limit img-src in the Content Security Policy |
| Links | Phishing, or secrets hidden in the link's query string | Allow-list domains; show the full address; drop other links |
| Inserted into SQL | SQL injection, or reading tables the user should not see | Do not run model-written SQL on real data; use parameters and allow-listed columns; use a read-only role |
| Passed to a shell or eval | Command injection: the attacker runs code on your server | Never. Use fixed functions with validated arguments |
| URLs your server fetches | Server-side request forgery (SSRF): reaching internal services | Allow-list hosts; block private and link-local addresses; do not follow redirects blindly |
| JSON for a tool call | Wrong types, extra fields, huge values | Validate against a JSON schema; reject unknown fields; then check business rules |
Escape before rendering
import html
from urllib.parse import urlparse
ALLOWED_LINK_HOSTS = {"help.example.com", "docs.example.com"}
def safe_text(model_output: str) -> str:
# The browser will show this as text. It will never run it.
return html.escape(model_output)
def safe_link(url: str) -> str | None:
parsed = urlparse(url)
if parsed.scheme != "https":
return None
if parsed.hostname not in ALLOWED_LINK_HOSTS:
return None
return urlValidate structured output
from jsonschema import validate, ValidationError
REFUND_SCHEMA = {
"type": "object",
"properties": {
"order_id": {"type": "string", "pattern": "^ORD-[0-9]{6}$"},
"amount": {"type": "number", "minimum": 1, "maximum": 500},
"reason": {"type": "string", "maxLength": 200},
},
"required": ["order_id", "amount"],
"additionalProperties": False,
}
def check_refund_args(args: dict) -> dict:
try:
validate(instance=args, schema=REFUND_SCHEMA)
except ValidationError as e:
raise ValueError(f"rejected tool call: {e.message}")
# The schema checks shape. Ownership is checked next, in code:
# does this order belong to the logged-in user?
return args- Inserting the answer into the page as raw HTML
- Running SQL the model wrote against the live database
- Fetching any URL the model returns
- Calling a tool with whatever JSON the model produced
- Escaping text and using a safe markdown renderer
- Letting the model choose from allow-listed queries and columns
- Fetching only allow-listed hosts, never internal addresses
- Schema check, then ownership and limit checks in the tool
eval, exec, a shell, or a template engine that runs code. If the feature truly needs to run generated code, run it in a sandbox with no secrets and no network, as covered in the agent-security track.Watch and read more
Lab
A small Python function that turns model output into safe HTML, with tests that pass.
The problem
On your own computer, complete the render_reply function in the starter code. It must:
- Remove markdown images completely.
- Escape all HTML in the text.
- Turn
[label](url)into a real link only if the URL uses https and its host is in the allow-list. Otherwise keep just the label as plain text.
Run the file. All four tests must pass. Then add two tests of your own, for example a link with http:// instead of https, or a link to help.example.com.evil.example.
Practice lab
Edit the code and run it. Python runs in your browser — nothing is sent to our server.
No output. Add print(...) to see values.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Why is checking <code>"help.example.com" in url</code> a weak allow-list?Reveal
A substring check accepts addresses like https://help.example.com.evil.example/ or https://evil.example/?x=help.example.com. Parse the URL, take the exact host name, and compare it with the allow-list. Also check the scheme, so that non-https links are dropped.
Q2Your function escapes HTML and filters links. Name one remaining way model output could harm a user, and how to reduce it.Reveal
The text itself can still mislead: a fake phone number, a fake UPI ID or false instructions like "share your OTP with our agent". Escaping cannot catch that. Reduce it with scope limits, detection of payment IDs and phone numbers in answers, clear warnings in the interface, and never letting the bot ask for secrets. This is where output handling meets misinformation.
Questions people ask
If the model is from a trusted provider, is its output trusted?
No. The provider is trusted to run the model, but the output depends on inputs you do not control: user messages, documents, web pages and tool results. Any of them can steer the output.
Is markdown safe to render?
It depends on the renderer and its settings. Some renderers pass raw HTML through, and most turn image syntax into images that load automatically. Turn off raw HTML, decide on a policy for images and links, and test it with hostile examples.
Can I let the model write SQL for a reporting feature?
Be careful. Safer patterns: the model picks from fixed, reviewed queries with parameters; or generated SQL runs only on a read-only copy with a role limited to the tables that user may see, with row limits and timeouts. Never run it with a powerful account on production data.
Does a Content Security Policy fix XSS?
It is a strong second layer. It can block inline scripts and limit where images and scripts load from. It does not replace escaping, and a weak policy gives little protection. Use both.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.