Reinforcement learning playgroundMiddleLesson 2926 min read

Exploration versus exploitation

Explore too little and you miss the best prize. Explore too much and you play badly for ever. Learn to decay epsilon, and compare runs fairly.

✦ Ask the tutor 🧪 Lab · 30 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Exploration versus exploitation

Explore too little and you miss the best prize. Explore too much and you play badly for ever. Learn to decay epsilon, and compare runs fairly.

1/4

In simple words

In a mango orchard, there is a small guava tree right next to the gate, and a huge mango tree at the far end. If you always pick from the first tree that gave you fruit, you will eat guavas for ever and never find the mangoes. But if you wander randomly every single day, you will waste a lot of time. The smart plan: wander a lot at first, then settle down.

Every RL agent faces this trade-off. Exploitation means using what you already know to collect reward now. Exploration means trying something that might be worse, to learn whether it is better. You need both.

Three ways to set epsilon

Schedule	While learning	Final result
Never explore (epsilon = 0)	Grabs the first reward it finds	Often stuck on a small prize
Always explore a lot (epsilon = 1)	Mostly random: low reward all through training	The table may be good, but the agent played badly for the whole time
Decay: start high, end low	Explores widely early, then uses what it learned	Usually finds the big prize and plays well by the end

Epsilon decay

An epsilon schedule is a function that says what epsilon to use in each episode. Two common shapes:

Python: two decay schedulespython

def linear(ep, n, start=1.0, end=0.05, frac=0.6):
    # falls in a straight line over the first 60% of episodes, then stays at 'end'
    t = min(1.0, ep / (frac * n))
    return start + (end - start) * t

def exponential(ep, n, start=1.0, end=0.05, rate=0.99):
    # multiplies by 'rate' every episode, but never goes below 'end'
    return max(end, start * rate ** ep)

n = 500
for ep in (0, 50, 100, 200, 300, 499):
    print(f"episode {ep:3}: linear {linear(ep, n):.3f}   exponential {exponential(ep, n):.3f}")

Both start near 1 (almost all exploring) and end small (mostly using what was learned). Keeping a small floor, such as 0.05, means the agent never stops checking completely.

Why the small prize is a trap

The guava is close, so the agent finds it early and its Q value goes up. With little exploring, the greedy choice then always walks toward the guava. The mango's reward never gets a chance to spread back to the start, because the agent almost never walks that far. This is a local optimum: good enough to stop searching, but not the best.

Comparing runs fairly

RL results are noisy: the same code with a different random seed can learn quite differently. So never judge a setting from one run.

1
Fix the seeds
Run each setting with the same list of seeds, such as 0 to 4.
2
Change one thing
Only the schedule changes between runs, not the map or alpha.
3
Measure two things
The final greedy result, and the average reward during training.
4
Report all runs
"4 of 5 seeds found the mango" is honest. The best seed alone is not.

Note

There are smarter ways to explore than random actions: giving a bonus to rarely visited states, being optimistic about unknown actions, or rewarding the agent for surprises (sometimes called curiosity). They matter in big games where random wandering almost never finds the prize. The trade-off is the same.

Real example

Arjun picks a coaching centre. He could join the first one his friend mentions (exploit). Or he could attend free demo classes at many centres first (explore). He tries three demos in the first week, then commits. That is epsilon decay in real life.

Try this. In the lab, move the mango one square further away. Which schedules still find it? What do you have to change in your schedule to find it again?

Watch and read more

Lab

An epsilon decay schedule that finds the far mango tree and plays well, backed by a fair comparison over five seeds.

~30 min

The problem

An orchard path: guava tree G (+1, ends the episode) is 2 steps left of the start S, and the mango tree M (+10, ends the episode) is 6 steps right. The Q-learning agent is ready. You write the exploration schedule.

Press Run. The comparison shows "never explore", "always 0.3" and "always 1.0". Note which tree each ends up at, and the average reward during training.
Write epsilon_at(episode, n_episodes): start close to 1.0, go down as episodes pass, never go up, and end at 0.1 or less.
Run again. Your schedule should find the mango on most seeds and earn a higher average training reward than "always 1.0".
Press Check.
Write three sentences comparing the four schedules, using the numbers you saw.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

import random

# G = guava (+1), M = mango (+10). Both end the episode. Start at S.
PATH = "G.S.....M"
START, N = PATH.index("S"), len(PATH)
LEFT, RIGHT = 0, 1

def step(pos, a):
    pos = max(0, min(N - 1, pos + (-1 if a == LEFT else 1)))
    if PATH[pos] == "G":
        return pos, 1.0, True
    if PATH[pos] == "M":
        return pos, 10.0, True
    return pos, 0.0, False

def epsilon_at(episode, n_episodes):
    # TODO: start close to 1.0, fall as episodes pass, end at 0.1 or less.
    return 0.0

def train(schedule, n_episodes=500, alpha=0.5, gamma=0.9, seed=0):
    rng = random.Random(seed)
    Q = [[0.0, 0.0] for _ in range(N)]
    rewards = []
    for ep in range(n_episodes):
        eps = schedule(ep, n_episodes)
        s, total = START, 0.0
        for _ in range(50):
            if rng.random() < eps:
                a = rng.randrange(2)
            else:
                a = max(range(2), key=lambda i: Q[s][i])     # ties go LEFT
            s2, r, done = step(s, a)
            target = r if done else r + gamma * max(Q[s2])
            Q[s][a] += alpha * (target - Q[s][a])
            s, total = s2, total + r
            if done:
                break
        rewards.append(total)
    return Q, rewards

def greedy_end(Q):
    s = START
    for _ in range(50):
        s, r, done = step(s, max(range(2), key=lambda i: Q[s][i]))
        if done:
            return PATH[s]
    return "nowhere"

schedules = {
    "never explore": lambda e, n: 0.0,
    "always 0.3":    lambda e, n: 0.3,
    "always 1.0":    lambda e, n: 1.0,
    "yours":         epsilon_at,
}
for name, sched in schedules.items():
    ends, avgs = [], []
    for seed in range(5):
        Q, rewards = train(sched, seed=seed)
        ends.append(greedy_end(Q))
        avgs.append(sum(rewards) / len(rewards))
    print(f"{name:14} ends at {''.join(ends)}   average training reward {sum(avgs) / 5:.2f}")

Output

No output. Add print(...) to see values.

You are done when

epsilon_at starts near 1, never rises, and ends at 0.1 or less
Your schedule finds the mango on at least 4 of 5 seeds
Its average training reward beats "always 1.0" clearly
The comparison uses numbers from all five seeds, not the best one

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1"Always 1.0" also ends at the mango. So why is it worse than your schedule?Reveal

Because it plays randomly for all 500 episodes. Q-learning is off-policy, so the table still learns the best route, and the greedy route at the end is right. But during training the agent often walked into the guava or wandered, so its average reward was low. If an agent learns while it is being used (a robot at work, an app with real users), that poor behaviour during learning is a real cost.

Q2Why does "always 0.3" fail here, even though it explores 30% of the time?Reveal

To reach the mango by accident, the agent must go right six times from the start. When the greedy choice points left, each step right needs a random move (chance 0.3 x 0.5 = 0.15), and six in a row is very unlikely. So the mango is almost never found. Exploration that is random step by step is weak at reaching far-away rewards, which is why big games need smarter exploration or a schedule that starts near 1.

Please sign in to continue.

Questions people ask

Is there one best epsilon schedule?

No. It depends on the game, the number of episodes and how far away the good rewards are. That is why you compare schedules with several seeds on your own problem.

Why not keep epsilon at 1 during training and use the greedy policy only at the end?

For a tiny game that can work, because Q-learning learns from random play. But in bigger games, random play rarely reaches the interesting parts, so the agent must use what it has learned to get deeper into the game. And if learning happens while the agent is being used, random play has a real cost.

What is a random seed and why does it matter?

A seed fixes the sequence of "random" numbers, so a run can be repeated exactly. Different seeds give different luck. Testing several seeds shows whether a result is real or just lucky.

Does exploration matter outside games?

Yes. A recommendation system that only shows what already works may never learn about new items. A robot that never tries a new grip never finds a better one. But real exploration must stay within safe limits.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Exploration versus exploitation

Exploration versus exploitation

Three ways to set epsilon

Epsilon decay

Why the small prize is a trap

Comparing runs fairly

Fix the seeds

Change one thing

Measure two things

Report all runs

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask