Reinforcement learning playgroundMiddleLesson 2897 min read

States, actions and rewards

Every RL problem is described with the same few words. Learn them by building a tiny corridor game and scoring it with a discounted return.

✦ Ask the tutor 🧪 Lab · 20 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

States, actions and rewards

Every RL problem is described with the same few words. Learn them by building a tiny corridor game and scoring it with a discounted return.

1/5

In simple words

Think of a board game like snakes and ladders. Where your token stands is the state. Rolling and moving is the action. Climbing a ladder is a good reward, sliding down a snake is a bad one. The game ends when someone reaches 100. RL describes every game like this.

To teach an agent anything, we first describe the game in a fixed shape. The same shape works for a board game, a robot or a delivery app. These words are worth learning well, because every later lesson uses them.

Word	Meaning	In a snakes and ladders game
State	Everything the agent needs to know about the situation now	The square your token is on
Action	A choice the agent can make	(Here there is only one: roll. Our corridor game gives real choices.)
Reward	A number received after each action	+ for a ladder, - for a snake
Episode	One full game, from start to an ending	From square 1 to square 100
Terminal state	A state where the episode ends	Square 100
Policy	The agent's rule: which action to take in each state	"Always roll" (not much choice!)
Return	The total reward collected from now until the end	All the ladders minus all the snakes still ahead

The environment's one job: step

In code, an environment usually has a function called step. You give it the current state and an action. It gives back three things: the next state, the reward, and whether the game is done. Popular RL toolkits use this same idea, with a few extra details.

1
state = start
Put the agent at the start of the game.
2
action = policy(state)
The agent chooses.
3
next_state, reward, done = step(state, action)
The world answers.
4
If not done, state = next_state and go back to step 2
Collect rewards along the way.
5
When done, the episode is over
Add up the rewards to see how well it went.

Why future rewards count a little less: discounting

Would you prefer ₹100 today or ₹100 in five years? Most people pick today. RL agents are built the same way. A reward that comes k steps in the future is multiplied by gamma to the power k, where gamma (the discount factor) is a number between 0 and 1, such as 0.9.

The discounted returntext

return = r0 + gamma * r1 + gamma^2 * r2 + gamma^3 * r3 + ...

gamma = 0     only the very next reward matters (very short-sighted)
gamma = 0.9   rewards far away still count, but less
gamma = 1     every reward counts the same (only safe if episodes always end)

Discounting does two useful things. It makes the agent prefer reaching a goal sooner, and it keeps the total finite even in very long games.

Python: same rewards, different gammapython

def discounted(rewards, gamma):
    total = 0.0
    for k, r in enumerate(rewards):
        total += (gamma ** k) * r
    return total

short_path = [-1, -1, 10]              # two steps, then the prize
long_path = [-1, -1, -1, -1, -1, 10]   # five steps, then the same prize

for gamma in (0.5, 0.9, 1.0):
    print(f"gamma={gamma}: short={discounted(short_path, gamma):6.2f}  long={discounted(long_path, gamma):6.2f}")

With any of these values the short path scores higher, because it wastes fewer steps and gets the prize earlier. The -1 per step is a common trick: it tells the agent "hurry up".

Choosing a good state

A state should hold everything needed to decide well, and not much more. If a Ludo agent's state leaves out where the other players' tokens are, it cannot learn to avoid being captured. But if the state includes useless details, like the colour of the table, there are far more states to learn about and learning slows down.

A poor state

Leaves out something that changes what you should do
Includes the time of day for a game where time does not matter
Is so detailed that no two situations ever look the same

A good state

Holds what the best action depends on
Leaves out details that never change the best action
Lets similar situations be treated alike

Note

When the state holds everything needed, and the future depends only on the current state and action, the problem is called a Markov decision process (MDP). You do not need the maths now. It is simply the name for the shape you are learning here.

Try this. Describe a game of carrom as states, actions and rewards. What would you put in the state? What is one reward that would teach a bad habit?

Watch and read more

Lab

A working corridor game with correct rewards and endings, plus a discounted-return function you can trust.

~20 min

The problem

You are the player P in a corridor of 7 squares. Walk left into the muddy puddle at square 0 and you lose (reward -10, game over). Walk right to the hot samosa at square 6 and you win (reward +10, game over). Every other move costs -1.

Press Run. The starter's step moves the player but never gives a reward and never ends the game. Look at the strange output.
Fix step: return -10 and done=True in the puddle, +10 and done=True at the samosa, and -1 with done=False anywhere else.
Fix discounted_return so that it really uses gamma.
Run again. "Always right" should earn rewards -1, -1, +10. Check the discounted return by hand with gamma = 0.9.
Press Check.
Extra: write your own policy that goes left from square 1 and right everywhere else. Is it ever better than always right? Why not?

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

# A corridor game. You are P at square 3.
#   square:  0   1   2   3   4   5   6
#            ~   .   .   P   .   .   S
# ~ = muddy puddle (reward -10, game over)
# S = hot samosa   (reward +10, game over)
# any other move costs -1
START, PUDDLE, SAMOSA = 3, 0, 6

def step(state, action):
    """Return (next_state, reward, done)."""
    next_state = state - 1 if action == "left" else state + 1
    reward = 0      # TODO: -10 in the puddle, +10 at the samosa, -1 otherwise
    done = False    # TODO: True in the puddle or at the samosa
    return next_state, reward, done

def discounted_return(rewards, gamma):
    # TODO: rewards[0] + gamma * rewards[1] + gamma**2 * rewards[2] + ...
    return sum(rewards)

def play(policy, max_steps=10):
    state, rewards = START, []
    for _ in range(max_steps):
        state, reward, done = step(state, policy(state))
        rewards.append(reward)
        if done:
            break
    return rewards

def always_right(state):
    return "right"

def always_left(state):
    return "left"

for name, pol in [("always right", always_right), ("always left", always_left)]:
    rw = play(pol)
    print(f"{name:13} rewards={rw}  return(gamma=0.9)={discounted_return(rw, 0.9):.2f}")

Output

No output. Add print(...) to see values.

You are done when

step returns -1 for ordinary moves, and ends the game with -10 or +10 at the two ends
discounted_return multiplies each reward by gamma to the power of its position
Always right scores 6.2 with gamma 0.9, always left scores less
The extra question explains that stepping left from square 1 lands in the puddle, so it is never better

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Suppose the puddle gave only -2 instead of -10. From square 1, which is better with gamma = 0.9: step into the puddle, or walk to the samosa?Reveal

Into the puddle: return -2. Walking right from square 1 takes 5 moves: four moves at -1 and then +10, so -1 - 0.9 - 0.81 - 0.729 + 0.6561 x 10, which is about 3.12. Walking right is still better. But if the step cost were -3, the walk would be worse than -2, and a clever agent would jump in the puddle on purpose. Reward numbers decide behaviour.

Q2Your friend adds a "time left" counter to the state, even though the game has no time limit. What does that change?Reveal

Nothing useful: the best action does not depend on it, but now every square appears with many different counter values. A table-based learner would have to learn each combination separately, which takes more tries. If the game did have a time limit, then time left would belong in the state, because the best action could change when time is short.

Please sign in to continue.

Questions people ask

Can a reward be negative?

Yes. Negative rewards act as penalties: falling in a puddle, crashing, or just using up a step. The agent tries to make the total as high as possible, so it learns to avoid them.

Who decides the rewards?

The person who builds the environment. This is a big responsibility: the agent will chase exactly what you reward, not what you meant. Lesson 6 is all about this.

What if the game never ends?

Some tasks go on forever, like keeping a room at a steady temperature. With a discount factor below 1, the return still adds up to a finite number, so learning still works.

Is the policy the same as the agent?

The policy is the agent's decision rule. The agent also has a way of learning, which changes the policy over time. At the end of training, the policy is what you keep and use.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

States, actions and rewards

States, actions and rewards

The environment's one job: step

state = start

action = policy(state)

next_state, reward, done = step(state, action)

If not done, state = next_state and go back to step 2

When done, the episode is over

Why future rewards count a little less: discounting

Choosing a good state

Watch and read more

The problem

Practice lab

You are done when

Hard questions

Questions people ask