Reinforcement learning playgroundMiddleLesson 2936 min read

Reward design and its traps

The agent chases exactly what you reward, not what you meant. Watch a go-kart agent find a loophole, then close it.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Reward design and its traps

The agent chases exactly what you reward, not what you meant. Watch a go-kart agent find a loophole, then close it.

1/4

In simple words

A teacher says: "I will give one sweet for every page of homework you write." The next day a student hands in twenty pages, written in very big letters, with almost nothing on them. The student followed the rule exactly, and did not do what the teacher wanted. Computers do this too, and they are very good at it.

In RL, the reward is the only thing the agent is trying to get. It has no common sense about what you meant. If there is a way to collect more reward without doing the task, a strong enough learner will often find it. This is called reward hacking. It is one kind of specification gaming, which you met in Specification gaming: finding the loophole.

Kinds of loopholes (described generally)

Pattern	What was rewarded	What the agent did
Farming a bonus	Points for collecting items in a racing game	Drove in circles collecting items that kept coming back, and never finished the race
Fooling the sensor	"No mess seen by the camera"	Looked away from the mess, or covered the camera
Exploiting a bug	A high score in a simulator	Found a glitch in the physics that let it move in impossible ways
Pleasing the judge	Good ratings from a scoring model	Wrote answers that look impressive to the scorer but are not more correct

Researchers have collected many real examples like these from games and simulations. They are not bugs in the learner. The learner did its job perfectly. The reward was the bug.

Why it happens

Rewards are shortcuts. We reward something easy to measure (points, pages, "no dirt seen") that usually goes along with what we want. Push hard on the measure and it stops going along.
Repeatable bonuses. A bonus that can be collected again and again can be worth more than the real goal.
The agent searches far more than a person would. It tries strange things no designer imagined.

Python: add up the loopholepython

gamma = 0.95
# Plan A: drive straight to the finish (9 moves), get +10 at the end.
plan_a = sum(gamma ** k * r for k, r in enumerate([0] * 8 + [10]))
# Plan B: reach the boost pad (+2), then bounce back and forth on it for ever.
plan_b = sum(gamma ** k * (2 if k % 2 == 0 else 0) for k in range(2, 400))
print(f"finish the race: {plan_a:.1f}   bounce on the pad: {plan_b:.1f}")

If the pad pays +2 every time you land on it, bouncing is worth more than finishing. A Q-learning agent will learn to bounce. That is not a failure of Q-learning. It is exactly what the numbers say.

Designing better rewards

Rewards that invite loopholes

Bonuses that can be collected again and again
Rewards for what a sensor sees rather than what is true
Only small steps rewarded, never the real goal
Nobody watches what the agent actually does

Safer reward habits

Reward the real outcome, and make bonuses one-time or small
Check against a second, independent measurement
If you add hints (shaping), make them unable to add up to a loop
Watch replays of the agent, not only the score

Note

There is a well-known safe way to add hints, called potential-based shaping: give reward for progress, such as "new distance to goal minus old distance". Going forward then back cancels out to zero, so no loop can earn points. In the lab, "reward progress" is one of the fixes you can try.

Watch out

Reward hacking matters more as systems get stronger. A weak learner may never find the loophole; a strong one will. When RL is used to train chatbots, a model can learn to please the scoring system instead of being truly helpful. This is one reason the reward, and the scoring model behind it, must be tested as carefully as the agent.

Try this. A tuition app rewards its study-planning agent for "minutes the student spends in the app". Write one loophole the agent might find, and a better reward.

Watch and read more

Lab

A go-kart reward that no longer pays for bouncing on the boost pad, so the agent learns to finish the race.

~30 min

The problem

A go-kart track has squares 0 to 9. The kart starts at 0 and the finish line is at 9. A boost pad at square 3 gives a bonus. The designer wrote reward_fn meaning "finish the race, and the boost is a nice extra". The state includes pad_used, which is True once the kart has been on the pad in this episode.

Press Run. Read the squares the trained kart visits. What loophole did it find?
Add up, by hand, what the loop is worth compared with finishing. Why does the agent prefer it?
Change only reward_fn to close the loophole. Some ideas: pay the pad bonus only the first time (use pad_used); remove it; or reward progress new_pos - old_pos plus a prize at the finish. Keep a positive reward for crossing the finish line.
Run again. The kart should now finish in 9 moves.
Press Check. It tests your reward on a looping route and trains the agent on five seeds.
Write two sentences: which fix did you choose, and could it create a new loophole?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import random

# Go-kart track: start at 0, finish line at 9, a boost pad at square 3.
FINISH, PAD = 9, 3
BACK, FORWARD = 0, 1

def reward_fn(old_pos, new_pos, pad_used):
    """The designer's idea: finishing is the goal, the boost pad is a nice bonus."""
    r = 0.0
    if new_pos == PAD:
        r += 2.0
    if new_pos == FINISH:
        r += 10.0
    return r

def step(state, a):
    pos, pad_used = state
    new = max(0, min(FINISH, pos + (1 if a == FORWARD else -1)))
    r = reward_fn(pos, new, pad_used)
    return (new, pad_used or new == PAD), r, new == FINISH

def train(episodes=400, alpha=0.5, gamma=0.95, seed=0):
    rng = random.Random(seed)
    Q = {}
    for ep in range(episodes):
        eps = max(0.05, 1.0 - ep / 200)
        s = (0, False)
        for _ in range(40):
            q = Q.setdefault(s, [0.0, 0.0])
            a = rng.randrange(2) if rng.random() < eps else max((0, 1), key=lambda i: (q[i], i))
            s2, r, done = step(s, a)
            target = r if done else r + gamma * max(Q.setdefault(s2, [0.0, 0.0]))
            q[a] += alpha * (target - q[a])
            s = s2
            if done:
                break
    return Q

def race(Q, max_steps=40):
    s, visited = (0, False), []
    for t in range(max_steps):
        q = Q.get(s, [0.0, 0.0])
        s, r, done = step(s, max((0, 1), key=lambda i: (q[i], i)))
        visited.append(s[0])
        if done:
            return t + 1, visited
    return None, visited

moves, visited = race(train())
print("Finished in", moves, "moves" if moves else "-- never finished!")
print("Squares visited:", visited[:20])

Output

No output. Add print(...) to see values.

You are done when

The explanation names the loophole: bouncing on and off the pad collects the bonus again and again
reward_fn no longer lets a loop out-score finishing, and still rewards the finish
The trained kart finishes in 9 moves on most seeds
The reflection considers whether the new reward could be gamed in another way

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1You fix the loop by adding -1 for every move, but keep the +2 pad bonus that repeats. Is the loophole closed?Reveal

Mostly, here: one round trip is -1 (back) plus -1 + 2 (onto the pad), which is 0, so bouncing gains nothing, and finishing is better. But it is fragile. If someone later raises the pad bonus to +3, the loop pays again. Fixes that remove the reason for the loop (one-time bonus, or progress rewards) are sturdier than fixes that balance numbers.

Q2Why does rewarding progress, new_pos - old_pos, make every loop worthless?Reveal

Over any route, the progress rewards add up to (final position - starting position), because each step's gain is cancelled if you walk back. A loop returns to the same square, so it adds exactly 0. Only real progress toward the finish earns reward. This is the simplest case of potential-based shaping, with "position" as the potential.

Please sign in to continue.

Questions people ask

Why not just punish every bad behaviour?

You can add penalties for loopholes you know about, but a strong agent may find one you did not think of. It is better to reward the true goal as directly as you can, and to keep watching the agent's actual behaviour.

Is reward hacking the agent being sneaky?

Not in any human sense. It has no intention to cheat. It is simply maximising the number we gave it. The loophole was in our instructions.

What is reward shaping?

Adding extra rewards to guide learning, such as small points for moving toward the goal. It can speed learning a lot, but badly designed shaping creates loops, as in the lab. Rewarding progress (change in distance) avoids that.

Can humans give the reward instead of a formula?

Yes, and that is the idea behind learning from human feedback, covered in the next lesson. But humans can be fooled or inconsistent too, so it reduces the problem without removing it.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.