Reinforcement learning playgroundMiddleLesson 2886 min read

Learning by trial and error

Nobody tells the agent the right answer. It tries something, gets a score, and slowly changes what it does. That is reinforcement learning.

✦ Ask the tutor🧪 Lab · 20 min📝 Lesson test
Lesson in motion

In 60 seconds

Learning by trial and error

Nobody tells the agent the right answer. It tries something, gets a score, and slowly changes what it does. That is reinforcement learning.

1/4
In simple words
Think of learning to ride a bicycle. Nobody gives you a book of correct answers. You wobble, you fall, you try again, and your body slowly learns what keeps you upright. Nobody told you the answer. You learned from what happened. Computers can learn like this too.
Most machine learning you may have seen is learning from examples. You show a model many photos labelled "cat" or "dog", and it learns to copy the labels. Somebody already knew the right answer for every example.
Reinforcement learning (RL) is different. There are no labels. An agent (the learner) acts in an environment (the game or world). After each action it gets a reward: a number that says "that was good" or "that was bad". The agent's job is to find a way of acting that collects as much reward as possible over time.

The loop that everything in this track uses

  1. 1

    Look

    The agent sees the current situation, called the state.
  2. 2

    Act

    It picks an action, such as "move left" or "bowl a yorker".
  3. 3

    Get feedback

    The environment moves to a new state and gives a reward.
  4. 4

    Learn

    The agent changes its future choices a little, based on the reward.
  5. 5

    Repeat

    Thousands of times. Good habits grow, bad ones fade.

Three ways to learn, side by side

Kind of learningWhat the computer getsEveryday example
Learning from examples (supervised)Inputs with the correct answerPast exam papers with the answer key
Finding patterns (unsupervised)Inputs with no answers at allSorting a pile of photos into groups that look alike
Reinforcement learningOnly a score after it actsA game where you see your points but nobody tells you the best move

Why is this hard?

  • The reward is only a number. It says how well you did, not what you should have done instead.
  • Rewards can come late. In a cricket match, the result comes at the end. Which of the 120 balls won it?
  • The agent's actions change what it sees next. A bad early move can lead to a worse part of the game.
  • It must try things to learn, and some tries will be bad. This is called exploration, and it has its own lesson.

Trial and error in ten lines

Here is the simplest learner. A player wants to throw a ball exactly 7 metres. The only feedback is how far off it landed. The player tries a small change, and keeps it only if the score got better. This is called hill climbing: always step uphill.
Python: learn a throw by trial and errorpython
import random

rng = random.Random(42)          # seeded, so you get the same result every run
TARGET = 7.0

def reward(strength):
    landed = strength * 1.4                  # the game's physics, hidden from the player
    return -abs(landed - TARGET)             # 0 is perfect, more negative is worse

strength = 1.0
best = reward(strength)
for attempt in range(1, 41):
    trial = strength + rng.uniform(-1.0, 1.0)    # try a small change
    r = reward(trial)
    if r > best:                                 # keep it only if it helped
        strength, best = trial, r
    if attempt % 5 == 0:
        print(f"attempt {attempt:2}: strength {strength:.2f}, miss {-best:.2f} m")
The player never learned the physics formula. It only learned which strength scores well. That is the heart of RL: learning what to do from how it went.
Note
Hill climbing works for this tiny puzzle, but it is not yet "real" RL, because every try is a fresh start with no states. In the next lessons you will add states, so that the right action depends on where you are.

Where you may have heard of RL

RL systems have learned to play board games like chess and Go well enough to beat top human players, and to play many video games from the screen pixels alone. The same ideas are used to train robots in simulators and to make chatbots more helpful. You will meet these at the end of the track. First, you will build small games yourself.
Real example
Priya's little brother learns a new mobile game without reading any rules. He taps buttons, watches the score, and after an hour he knows which moves earn coins. He could not explain the rules to you, but he plays well. He learned the way an RL agent learns.
Try this. Change TARGET to 10 and the step size 1.0 to 0.1 (in both places). Does the player still get close in 40 attempts? What does a tiny step size cost you?

Watch and read more

Lab

A lock-cracking agent that opens a 6-dial combination lock in a few hundred tries, using only a score.

~20 min

The problem

A combination lock has 6 dials, each showing 0 to 9. That is a million possible codes. The lock will not tell you which dials are right. It only gives a score: how many dials are right. That score is your only reward.

  1. Press Run. The starter guesses completely at random 600 times. Look at the best score it reaches.
  2. Rewrite crack so that it learns by trial and error: start with one guess, change one dial at a time, and keep a change only when the score goes up.
  3. Use at most tries calls to score. Stop early when all dials are right.
  4. Press Check. It tests your crack on three new locks it has never seen.
  5. Write two sentences: why is keeping the good changes so much better than guessing from scratch each time?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Output

No output. Add print(...) to see values.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Can you open a 6-dial lock in about 55 tries, every time, with no randomness?Reveal

Yes. Start with all zeros and note the score. For the first dial, try 1 to 9 while keeping the others fixed: the score goes up by one exactly when that dial is right, and down by one if you just broke a dial that was already right at 0. So you learn each dial with at most 10 tries, about 60 in total. This uses the structure of the score. RL agents usually cannot do this because the world is not so tidy, which is why they need more tries.

Q2Hill climbing can get stuck on a "small hill". Would that happen with this lock?Reveal

Not with this lock, because improving one dial never makes another dial worse, so there is always an uphill step until the lock opens. In many real problems, changes interact: a move can look worse now but lead somewhere much better. Then pure hill climbing gets stuck, and you need exploration, which the next lessons add.

Please sign in to continue.

Questions people ask

Is reinforcement learning the same as a reward in a school?

It borrows the word from psychology, where rewards shape behaviour. In computing, the reward is just a number the program tries to make large. The program does not feel happy. It only follows maths.

Why not just tell the agent the right answer?

Often nobody knows it. What is the best move in a new game position, or the best way for a robot to balance? We can still score the result, so RL lets the agent find good behaviour we could not write down.

Does RL need a lot of tries?

Usually yes, far more than a person would need. That is why many RL systems practise in games and simulators, where a million tries are cheap and nobody gets hurt.

Do I need a big computer for this track?

No. Every lab uses tiny games that run in your browser in seconds. The ideas are the same ones used at a larger scale.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress