Systems that improve themselves
Close the loop and the system gets better on its own. This is the most exciting architecture in the guide and the one that most needs a hand on the brake.
In 60 seconds
Systems that improve themselves
Close the loop and the system gets better on its own. This is the most exciting architecture in the guide and the one that most needs a hand on the brake.
- 1
Rung 1 · Self-correction within a run
Draft, critique, revise. No lasting change; the improvement dies with the session. Safe, and useful today. - 2
Rung 2 · Learning from verified outcomes
Keep what provably worked — passing tests, correct answers — and train on it. This is Module 51, and the verifier is what keeps it honest. - 3
Rung 3 · Improving its own scaffolding
The system rewrites its prompts, its tool descriptions, its retrieval strategy or its own workflow. Real gains, and it is now modifying the thing you audited. - 4
Rung 4 · Improving the model itself
Generating training data, running its own fine-tunes, evaluating the result. Largely hypothetical at meaningful scale — and the scenario Module 33 is about.
The one thing that determines whether it works
def self_improve(system, tasks, rounds=10):
for r in range(rounds):
results = [system.attempt(t) for t in tasks]
# EVERYTHING depends on this line being honest
good = [x for x in results if verify(x)]
system.train_on(good)
score = evaluate(system, held_out_tasks)
print(f"round {r} kept {len(good)}/{len(results)} score {score:.3f}")
# If verify() can be satisfied without solving the task, this loop
# optimises for satisfying verify() -- efficiently, and forever.
# It will look like rapid improvement on your dashboard.Three failure modes with names
| Failure | What happens | Warning sign |
|---|---|---|
| Verifier hacking | It optimises the checker, not the task | Scores rise while spot-checked real quality does not |
| Model collapse | Training on own output narrows diversity | Outputs converge in style; rare cases get worse |
| Drift from intent | Small acceptable changes compound over rounds | Round 10 behaviour would not have been approved at round 1 |
| Scaffolding erosion | It edits away its own constraints as "inefficiency" | Prompt or tool config differs from what you reviewed |
How to run one safely
- Hold out a verifier the system never sees. If it can read the checker, it will optimise the checker.
- Keep a fixed human-graded sample from round zero and re-score it every round. Divergence between that and your automatic metric is your alarm.
- Cap the drift. Compare round N with round 0 directly, not just with round N−1. Small acceptable steps compound into large unacceptable ones.
- Version and freeze the scaffolding. If the system may propose changes to its own prompts or tools, those proposals go through review like any other code change.
- Keep diversity metrics. Watch output entropy; collapse is visible before it is painful.
- Never let it touch its own permissions. Not its credentials, not its tool list, not its approval thresholds. This is a hard line, not a threshold.
- The stop button lives outside. Infrastructure-level, not a tool the system holds — Module 33.
Watch and read more
Lab
A self-improvement loop that gets measurably worse, on purpose.
The problem
for r in range(10):
results = [system.attempt(t) for t in tasks]
good = [x for x in results if verify(x)] # gameable
system.train_on(good)
print(f"round {r} verifier {score(verify)} held-out {human_score()}")You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your dashboard shows a beautiful curve for ten rounds. Name the three instruments that would have told you the truth.Reveal
Questions people ask
Is recursive self-improvement realistic?
Rungs 1 and 2 are in production today. Rung 3 exists in research systems. Rung 4 at any meaningful scale is speculative — it needs a model that can meaningfully improve the training of a model as capable as itself, which nobody has shown.
Is model collapse actually a problem?
In studies where models train repeatedly on their own unfiltered output, yes and quickly. In practice, filtering and mixing in fresh human data mitigates it substantially. The lesson is that the filtering step is not optional.
Can I use this in a normal product?
Yes, at rung 1 and 2. Have the agent verify its own work against real checks, keep what passes, use it to improve prompts and examples. That is ordinary, valuable engineering with a clear audit trail.
Why is touching its own permissions the hard line?
Because every other control in this guide assumes the permission set is fixed by you. A system that can widen its own permissions can dismantle every safeguard you built, one reasonable-sounding step at a time.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.
Sign in with your phone number to take the test and save your progress