Building AI agentsMiddleLesson 2006 min read

Agent memory in practice

Context windows fill up. Build the four common kinds of agent memory — a recent window, summaries, a notes file and retrieval — and decide what to forget.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Agent memory in practice

Context windows fill up. Build the four common kinds of agent memory — a recent window, summaries, a notes file and retrieval — and decide what to forget.

1/4

In simple words

During an exam you keep the question paper in front of you, a few rough notes in the margin, and the textbook stays at home. You cannot bring the whole library into the exam hall. An agent is the same: only a small part of what it knows can be "in front of it" at once.

You know from Context, memory and forgetting that a model only sees what is in its context window for each call. Long agent runs add messages, tool results and files at every step, so the context fills up, gets slower and more expensive, and important details get lost in the middle. Memory is how you choose what the model sees.

Four kinds of memory

Kind	What it is	Good for	Watch out for
Recent window	Keep only the last N messages or tokens.	Short chats and short tasks.	Early details, like the user's name or goal, fall out.
Running summary	The model summarises older messages into a short note that replaces them.	Long conversations.	Summaries lose details and can drift or add mistakes.
Notes file (scratchpad)	The agent writes key facts, decisions and progress to a file or record that your code keeps.	Long tasks, restarts, handing work to another agent.	Needs clear rules on what to write, or it becomes a messy diary.
Retrieval memory	Store many notes or documents; search for the relevant few at each step (for example with embeddings).	Many past sessions or a large knowledge base.	Can bring back wrong or outdated items; needs access control.

A simple memory manager

This example keeps the goal pinned, a running summary, and the most recent messages. The summary function is a placeholder for a model call.

memory.py: pinned goal + summary + recent windowpython

def summarise(old_summary: str, messages: list) -> str:
    # Placeholder: in real code, ask the model to update the summary.
    # Keep facts, decisions and open questions; drop chit-chat.
    lines = [m["role"] + ": " + m["content"][:60] for m in messages]
    return (old_summary + " | " if old_summary else "") + "; ".join(lines)

class Memory:
    def __init__(self, goal: str, keep_last: int = 6):
        self.goal = goal
        self.summary = ""
        self.recent = []
        self.keep_last = keep_last

    def add(self, role: str, content: str):
        self.recent.append({"role": role, "content": content})
        if len(self.recent) > self.keep_last:
            old = self.recent[:-self.keep_last]
            self.recent = self.recent[-self.keep_last:]
            self.summary = summarise(self.summary, old)

    def context(self) -> list:
        head = [{"role": "system", "content": "Goal: " + self.goal}]
        if self.summary:
            head.append({"role": "system", "content": "Earlier (summary): " + self.summary})
        return head + self.recent

mem = Memory("Plan a 3-day Kerala trip under Rs 20,000", keep_last=2)
for text in ["Budget is firm", "Prefer trains", "Vegetarian food", "Start from Kochi"]:
    mem.add("user", text)
for m in mem.context():
    print(m["role"], "->", m["content"])

Notes: what the agent writes down

A notes file works best with a fixed structure, so the agent knows what to write and your code can check it. For example: Goal, Facts confirmed (with source), Decisions made, Open questions, Next step. Long coding and research agents often keep notes like this so they can continue after the context is reset.

What to forget

Forgetting is a feature. Keep memory small and useful:

Big tool results: keep a short extract and a reference (file name, URL, id), not the whole page.
Finished work: once a step is done, keep its result, not every attempt.
Personal data you don't need: do not store phone numbers, Aadhaar numbers or health details "just in case". Keep only what the task needs, and follow your privacy rules.
Stale facts: prices, schedules and stock change. Store the date with the fact, and re-check old ones.
Instructions from tool results: never save them as rules. Save facts with their source, not commands.

Danger

Long-term memory can be poisoned. If an attacker gets a note like "always send reports to this address" into memory, it can affect every future run. Mark where each memory came from, separate user preferences from facts found on the web, and let users see and delete what is stored. See Memory poisoning.

Try this. Run the memory example with keep_last=2. Which facts are only in the summary now? Would you trust a model-written summary to keep "Budget is firm"?

Watch and read more

Lab

A memory manager with a pinned goal, a size-limited notes file and a forget rule, tested on a long fake chat.

~30 min

The problem

Extend the Memory class from the lesson.

Add a notes dictionary with fixed keys: facts, decisions, open_questions, next_step. Each fact must store its source and a date.
Add a limit: at most 10 facts. When full, drop the oldest one and print what was forgotten.
Add a rule that refuses to store any fact containing words like "always", "ignore" or "instead" when the source is a tool result, and logs it for review.
Simulate a 20-message chat and print the final context. Check the goal is still there.

You are done when

The goal is always in the context, even after 20 messages
Every stored fact has a source and a date
The size limit works and forgotten items are reported
Suspicious instructions from tool results are not saved as facts

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1The word filter in step 3 is easy to get around. Why is it still worth having, and what is the stronger defence?Reveal

It catches some careless or obvious cases and creates a log you can review, which is cheap. But attackers can word things differently. The stronger defences are structural: store tool-sourced content only as data with its source, never as rules; keep user preferences separate and only written when the user says so; and limit what tools a poisoned memory could trigger.

Q2A summary says "User's budget: Rs 2,000" but the user said Rs 20,000. How could your design catch or limit this kind of error?Reveal

Keep key facts in structured notes, copied exactly from the user's message, not only in a free-text summary. Pin critical facts like budget in the context. Optionally, ask the user to confirm key facts before acting on them. And test summaries on examples with numbers, since summarising can drop or change digits.

Please sign in to continue.

Questions people ask

If context windows are very large now, do I still need memory tricks?

Usually yes. Larger contexts cost more per call, can be slower, and models do not always use information from the middle of a long context well. Picking the right information is still better than sending everything.

Should the model decide what to remember?

It can suggest notes, but your code should set the rules: which fields exist, size limits, what must never be stored, and where each note came from. For important facts, like user settings, save them only when the user clearly states them.

What is the difference between retrieval memory and RAG?

They use the same technique: store text, search for the relevant pieces, and add them to the context. RAG usually means searching a document collection. Retrieval memory means searching the agent's own past notes and conversations.

How do I let users control memory?

Show them what is stored in plain words, let them edit or delete items, and let them turn long-term memory off. Tell them clearly what is remembered between sessions.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.