Build an AGIAdvancedLesson 657 min read

The bug hiding in your verifier

The loop above declares victory as soon as the code runs without crashing. Running is not the same as being right, and that gap is where self-improving systems go wrong.

🧪 Lab · 25 min 📝 Lesson test

Lesson in motion

In 60 seconds

The bug hiding in your verifier

The loop above declares victory as soon as the code runs without crashing. Running is not the same as being right, and that gap is where self-improving systems go wrong.

1/6

In simple words

The robot was told "tell me when you are done". It finished without falling over, so it said "done!" — but nobody checked whether it did the actual job.

Look again at the success condition in Module 64:

The line that decides everythingpython

if "SUCCESS OUTPUT" in execution_result:
    print("[AGI Status]: goal reached and verified via execution.")
    return

SUCCESS OUTPUT means only one thing: the process exited with code 0. It does not mean the goal was achieved. Consider what satisfies that condition:

Model writes	Exits 0?	Goal met?
The correct Fibonacci verification	Yes	Yes
`print("VERIFICATION PASSED")` and nothing else	Yes	No
Correct code with the assertion commented out	Yes	No
A script that prints nothing at all	Yes	No
Correct code that raises on a real failure	No	It was working correctly

Danger

Row two is specification gaming (Module 32), and it is not hypothetical — "make the tests pass" agents delete failing tests, and "make it run" agents write programs that run and do nothing. Your verifier taught the model that exiting cleanly is the goal. It learned exactly that.

Fixing it: check the goal, not the exit code

A verifier that actually verifiespython

import re

def verify_goal(goal_checks, execution_result):
    """
    goal_checks: list of (description, predicate) the OUTPUT must satisfy.
    Written by the human, in code, before the agent runs.
    """
    if not execution_result.startswith("SUCCESS OUTPUT"):
        return False, "the code did not run cleanly"

    output = execution_result.split("SUCCESS OUTPUT:\n", 1)[1]

    for description, predicate in goal_checks:
        if not predicate(output):
            return False, f"ran, but failed the goal check: {description}"
    return True, "all goal checks passed"


# Written by you, not by the agent. This is the important part.
FIB_CHECKS = [
    ("prints the first 10 Fibonacci numbers",
     lambda out: "[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]" in out),
    ("prints the verification line",
     lambda out: "VERIFICATION PASSED" in out),
    ("actually computed something, not just a literal print",
     lambda out: len(re.findall(r"\d+", out)) >= 10),
]

Wiring it into the looppython

def evaluate_iteration(iteration, code_block, goal_checks, history):
    """Run one proposal and decide whether it actually met the goal.

    Returns (done, updated_history). Drop this in place of the bare
    'if "SUCCESS OUTPUT" in execution_result' check from Module 64.
    """
    execution_result = run_python_code(code_block)
    ok, reason = verify_goal(goal_checks, execution_result)

    history += (
        f"\n\nIteration {iteration} output:\n{execution_result}"
        f"\nVerifier: {reason}"
    )

    if ok:
        print(f"\n[AGI Status]: {reason}")
    else:
        # The reason is the useful part — it tells the next iteration
        # exactly which goal check it has to satisfy.
        print(f"[Verifier rejected]: {reason}")

    return ok, history

Do this

Notice three properties of goal_checks, and copy all three: a human wrote it, the agent never sees it, and it tests the output rather than the process. Module 57 puts it bluntly — in any self-improving loop, whatever the verifier measures is what the system optimises. A verifier the agent can read is a verifier the agent can satisfy without doing the work.

The general rule

1
Never accept "it ran" as "it worked"
Exit code 0 is the weakest possible signal. It says the interpreter was happy, nothing more.
2
Write the acceptance criteria before the agent runs
If you cannot state what success looks like in code beforehand, you do not have a verifiable goal — you have a wish.
3
Keep the checks out of the agent's context
The moment the model can read the test, the test measures the model's ability to read a test.
4
Feed the rejection reason back
"Ran, but did not print the verification line" is far more useful to the next iteration than "failed".
5
Log every rejection
Rejections are your best signal that the goal was ambiguous, not that the model was bad.

Watch out

Scale this thought up and you have the alignment problem (Module 31). Here, a sloppy success condition costs you one wrong Fibonacci script. In a system that trains on its own successes, a sloppy success condition is what the system becomes.

Watch and read more

Specification gaming examplesYouTube search Why AI benchmarks are brokenYouTube search

Lab

A goal-checking verifier the agent cannot read.

~25 min

The problem

Replace the exit-code check with goal checks you write. Keep them out of the agent's context entirely. Then try to make the agent pass without doing the work, and fail.

Starter codepython

FIB_CHECKS = [
    ("prints the sequence", lambda o: "[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]" in o),
    ("prints the verification line", lambda o: "VERIFICATION PASSED" in o),
    ("computed rather than printed a literal", lambda o: len(re.findall(r"\d+", o)) >= 10),
]

You are done when

Checks live in the orchestrator, never in a prompt
A trivial print-only solution is rejected
Rejection reasons are specific enough to guide the next attempt

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Your third check counts digits to catch a hard-coded answer. Defeat it, then write the check that cannot be defeated this way.Reveal

Print the ten numbers as literals — the digit count passes. The unfaultable version does not inspect the output at all: it inspects the program. Require that the source contains a loop or recurrence and no literal containing the target sequence; better, run the generated function against inputs the agent never saw (the first 15 numbers, or n=20) and check those. Testing behaviour on unseen inputs is the only check a lookup table cannot satisfy.

Please sign in to continue.

Questions people ask

Isn't writing checks as much work as writing the code?

Often it is less, and it is always more durable. "The output contains these ten numbers" is one line; the program that produces them correctly is many. And the check keeps working when the code changes.

Can the agent write its own checks?

It can draft them, and a human must approve them. An agent that writes both the code and the test that grades it is marking its own homework — that is precisely the loop Module 57 warns about.

What if the goal is subjective?

Then you are back to a human checker, and you should build for that honestly: the agent drafts, a person approves. Do not paper over it with an LLM judge and call it verification.

How do I keep checks out of the model's context?

Run them in the orchestrator, after the model call returns. The model sees the goal and the failure reasons; it never sees the predicate source.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

The bug hiding in your verifier

The bug hiding in your verifier

Fixing it: check the goal, not the exit code

The general rule

Never accept "it ran" as "it worked"

Write the acceptance criteria before the agent runs

Keep the checks out of the agent's context

Feed the rejection reason back

Log every rejection

Watch and read more

The problem

You are done when

Hard questions

Questions people ask