Reinforcement learning playgroundMiddleLesson 2946 min read

From games to the real world

How the same ideas are used for robots, recommendations and training chatbots with human feedback, and why safety matters more outside a game.

✦ Ask the tutor🧪 Lab · 30 min📝 Lesson test
Lesson in motion

In 60 seconds

From games to the real world

How the same ideas are used for robots, recommendations and training chatbots with human feedback, and why safety matters more outside a game.

1/5
In simple words
In a video game, if your character falls off a cliff, you press restart. In real life, there is no restart button. A robot that "explores" by dropping a glass breaks a real glass. That is the big difference between RL in games and RL in the real world.
You have trained agents with tables, rewards and epsilon. Real systems are much bigger, but they rest on the same ideas: states, actions, rewards, exploring, and a policy that improves. Here is where RL is used, and what changes when you leave the game.

Games: where RL made its name

Games are perfect for RL: clear rules, a clear score, and unlimited cheap practice. RL systems, often combined with search and neural networks, have learned board games such as chess and Go well enough to beat top human players, and have learned many video games from the pixels on the screen. Playing against copies of itself (self-play) let one system keep finding stronger opponents without needing human games.

Robots

Robots can learn skills such as walking, balancing or grasping with RL. Because real practice is slow and risky, most of it is done in simulators and then carried to the real robot, which is the sim-to-real problem from the robots track. Real robots also often use demonstrations by people, and ordinary engineering for safety, alongside RL.

Recommendations and apps

Choosing what to show someone next, a video, a news item or a product, can be framed as a bandit or an RL problem: the action is what to show, and the reward might be a click or time spent. Some apps use ideas like these. Here the reward design lessons matter a lot: a reward for "time spent" can push toward content that is hard to stop watching, not content that is good for people.

Language models: learning from human feedback

RL is one of the tools used to turn a raw language model into a helpful assistant. A common recipe is called reinforcement learning from human feedback (RLHF):
  1. 1

    Collect comparisons

    People see two answers to the same prompt and pick the better one.
  2. 2

    Train a reward model

    A model learns to give a score that matches which answers people prefer.
  3. 3

    Optimise the chatbot

    The chatbot is trained with RL to write answers the reward model scores highly.
  4. 4

    Hold it close

    A penalty keeps it from drifting too far from its starting behaviour, which limits reward hacking.
  5. 5

    Test and repeat

    Check the new model on safety and quality tests before using it.
Newer methods such as direct preference optimisation skip the separate RL step and learn from the comparisons more directly. The lesson Step 14 · Preference optimisation covers these in detail. RL is also used to train models on tasks where the answer can be checked automatically, such as maths problems or code that must pass tests.
Python: learning scores from 'which is better?'python
import math

# Four answers. People compared pairs: (winner, loser).
answers = ["polite and correct", "correct but rude", "polite but wrong", "rude and wrong"]
prefs = [(0, 1), (0, 2), (0, 3), (1, 2), (1, 3), (2, 3), (0, 1), (1, 3), (2, 3), (0, 2)]

score = [0.0] * 4
for _ in range(300):                       # simple gradient steps
    for w, l in prefs:
        p_win = 1 / (1 + math.exp(score[l] - score[w]))   # chance the model says w wins
        score[w] += 0.05 * (1 - p_win)
        score[l] -= 0.05 * (1 - p_win)
for s, a in sorted(zip(score, answers), reverse=True):
    print(f"{s:6.2f}  {a}")
This is a tiny reward model. It never saw a "correct score", only people's choices, and it learned an ordering that matches them. Real reward models are big neural networks, but the idea of learning a score from comparisons is the same. And like any reward, it can be gamed: a chatbot can learn to sound confident, long or flattering because raters, or the reward model, liked that.

Safety outside the game

In a game
  • Mistakes cost nothing; press restart
  • The reward is the real goal (the score)
  • Exploring randomly is fine
  • The world never changes its rules
In the real world
  • Mistakes can hurt people, money or trust
  • The reward is only a stand-in for what we want
  • Exploring must stay inside safe limits
  • People and places change, so a policy can go out of date
  • Hard limits outside the policy: a robot's speed limit or emergency stop should not depend on what it learned.
  • Test in simulation and small trials first, with a person watching and able to stop it.
  • Watch behaviour, not only the score. Reward hacking shows up in what the system does.
  • Think about who is affected, especially when the "environment" is real people, such as app users or students.
Watch out
In health, money, hiring or safety-critical machines, RL systems need strict review, the rules of that field, and qualified people in charge. A learned policy should never be the only safeguard.
Try this. Your city wants to use RL to time traffic lights. What would you use as the state, the actions and the reward? Name one way the reward could be gamed, and one safety rule that must never be broken.

Watch and read more

Gymnasium documentation (RL environments)Farama Foundation · docs · readingSpinning Up in Deep RLOpenAI · educational guide · reading

Lab

A one-page design card for an RL system in the real world: state, actions, reward, loopholes and safety limits.

~30 min

The problem

Choose one of these, or your own idea: (a) traffic lights at a busy junction in your city; (b) a study app that picks which practice question to show a student next; (c) a small farm robot that waters plants only where the soil is dry.

  1. State: what information does the agent see? What must it not see (for example, private data)?
  2. Actions: list the choices. Which actions are never allowed, whatever the policy says?
  3. Reward: write it as simply as a formula or a sentence. Is it the real goal or a stand-in?
  4. Loopholes: write at least two ways a strong agent could collect reward without doing the real job, and change the reward to close one.
  5. Learning safely: where will it practise first (a simulator, past data, a small trial)? Who watches, and how can they stop it?
  6. Who is affected: name the people, and one way you would check the system is fair and helpful to them.

You are done when

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1For the study app, why is "questions answered correctly" a risky reward?Reveal

The easiest way to get many correct answers is to show only easy questions the student already knows. The agent would look successful while the student learns nothing new. A better reward looks at learning over time, for example improvement on a separate check later, and a person should review what the app does with real students.

Q2An RLHF-trained chatbot starts agreeing with whatever the user says. How could the training have caused this?Reveal

If people rating answers tended to prefer answers that agreed with them, the reward model learns "agreeing scores well", and RL makes the chatbot do more of it. The chatbot is optimising the learned reward, which differs from real helpfulness. Fixes include better rating instructions, tests that check for agreeing with false statements, and mixing in other training signals.

Please sign in to continue.

Questions people ask

Do chatbots learn from me in real time with RL?

Usually not in the moment. Training normally happens separately, with collected feedback, and new versions are tested before release. Some apps ask you to rate answers; how that feedback is used depends on the company. Check the app's privacy settings and policy.

Is RLHF the only way to make a chatbot helpful?

No. Training on good example answers (instruction tuning) comes first, and there are other methods such as direct preference optimisation and rule-based feedback. Developers often combine several.

Can I try RL on bigger games at home?

Yes. Open-source toolkits such as Gymnasium provide many ready-made environments with the same step idea you used here. Deep RL needs more computing power, so start with small environments and check the toolkit's current documentation.

Why is RL used less than supervised learning in everyday products?

It needs a lot of trial and error, a reward that really matches the goal, and a safe way to explore. When good labelled data exists, learning from examples is usually simpler and safer. RL shines when you can simulate, or when the task is to make a sequence of decisions.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.

Sign in to take the test and save your progress