Go deeper · How machines learn

How does a machine learn from rewards?

By trying actions and keeping track of which ones lead to rewards. In reinforcement learning, nobody gives the learner the right answer for each step. It acts, gets a score back, and slowly shifts toward the actions that earned more. The hard part is deciding when to try something new and when to stick with what already works.

Grades 6+15 minutesAverages onlyOptional: the update rule

What you will be able to do

Name the agent, environment, actions, reward and policy in an example, explain the trade-off between exploring and exploiting, and say why an AI chatbot agent is usually not learning from rewards while you use it.

Before you start

Read How can examples become a rule? first. How AI works introduces reinforcement learning as one of three ways to learn; this page goes deeper.

1 · You have already seen this

Learning a game by playing it

Many people have started a new video game without reading the rules.

You press buttons and watch what happens. Picking up a coin raises your score, and falling off a ledge costs you. After a while you know which moves pay off on each screen, even though nobody gave you a list of right answers.

Reinforcement learning trains a computer program in a similar way. The program acts, gets back a number that says how well it did, and adjusts what it does next time. This page shows how that works, lets you try it with three snack machines, and shows where it goes wrong.

2 · What happens, step by step

From one try to a better rule

Reinforcement learning in five steps. The next section goes deeper into steps 1 and 2, and the one after it into step 4. After the lab, a worked example goes deeper into steps 3 and 4.

How an agent learns

Five steps

  1. 1 · Take an actionThe learner, called the agent, makes one choice, such as jump or move left in a game. It chooses with its policy, its current rule for picking an action.
  2. 2 · Get a rewardThe world it acts on changes and sends back a number that a person designed, such as +10 for a coin or −100 for falling.
  3. 3 · Update the numbersThe agent nudges its numbers toward what just happened, so actions that scored well become more likely next time.
  4. 4 · Sometimes try something elseMost of the time the agent picks the best action so far. Now and then it tries another one, so it keeps learning about the others.
  5. 5 · Repeat many timesThe loop runs thousands or millions of times. Over those rounds the policy changes so the agent collects more reward in total.

Going deeper · steps 1 and 2

Act, get a score, adjust

Reinforcement learning (RL) is about a learner inside a situation, not a learner reading a table of labeled examples.

In supervised learning, every training example comes with the right answer. In reinforcement learning there is no answer key. The learner, called the agent, takes an action in an environment: a game, a simulator, a room. The environment changes and sends back a reward, a number that says how good the result was. Then the agent acts again.

Over many rounds the agent builds a policy: its rule for choosing an action in each situation. Learning means changing the policy so that it collects more reward in total. Rewards can arrive late. In a board game the only clear score might be a win or a loss at the very end, so the agent has to work out which earlier moves deserve the credit.

A reward is just a number that a person designed. The agent does not feel pleased. It adjusts numbers inside itself so that high-scoring actions become more likely next time.

The reinforcement-learning loop An agent box on the left and an environment box on the right. An arrow labeled action goes from the agent to the environment. An arrow labeled reward and new situation comes back from the environment to the agent. Inside the agent is its policy, the rule it uses to choose actions, which it updates after each reward. AGENT the learner Policy its rule for choosing ENVIRONMENT game, simulator, room, snack machines action reward + new situation then the agent adjusts its policy and acts again
The loop repeats thousands or millions of times. Practice is the agent’s training data.
Five words, two made-up examples
WordMeaningIn a video gameIn a robot vacuum (practicing in a simulator)
AgentThe learner that chooses actionsThe program pressing the buttonsThe vacuum’s controller
EnvironmentEverything the agent acts onThe game worldA simulated room with furniture
ActionOne choice the agent can makeJump, move left, waitDrive forward, turn, stop
RewardA number sent back after an action+10 for a coin, −100 for falling+1 per clean tile, −5 for a bump
PolicyThe agent’s rule for picking an action in each situationWhich button to press on each screenWhich way to drive from each spot

Going deeper · step 4

Try something new, or stick with the best so far?

Every RL agent faces this dilemma. So do you, every time you choose where to eat lunch.

Predict first

Three snack machines stand in a hallway. You try each one once: A gives you a snack, B and C give nothing. Should you use only A from now on?

Show the answer

Not yet. One try each is very little evidence. B might pay out most of the time and just have been unlucky once. If you only ever use A, you will never find out. Exploiting means using the best option so far; exploring means trying others to learn more. A good strategy does mostly the first and some of the second.

A common simple strategy is called epsilon-greedy. Epsilon (ε) is a Greek letter that often stands for a small number. Most of the time the agent is greedy: it picks the machine with the best average so far. With a small probability ε, say on 10% of pulls, it picks a machine at random instead. That bit of exploring costs a few snacks, but it protects the agent from getting stuck on a lucky first guess. The lab lets you watch both happen.

3 · Try it

Watch an agent learn by trying

Pick an explore setting and a day of luck, then let the agent pull.

Lab

Which snack machine pays best?

Three made-up machines each pay a snack with a hidden chance. Choose how often the agent explores, let it pull, and watch its estimates and its total. Every “day” replays the same luck, so you can compare settings fairly.

Simulated in your browser with made-up machines and chances. Nothing is learned from you or stored.

Explore (ε)
Luck
Let the agent pull
Or pull one yourself

Last pulls, newest first:

    Three challenges

    1. Day 1, ε = 0%: press “100 pulls.” Which machine is the agent stuck on, and why? Reset and try ε = 10%.
    2. Day 2: compare ε = 0% with ε = 30%. When does exploring cost snacks instead of saving them?
    3. Set ε = 100%. Why do the estimates get close to the real chances while the total stays near half the pulls?

    The agent tries each machine once, then pulls 97 more times. With probability ε it picks a machine at random (exploring); otherwise it picks the machine with the best average so far (exploiting). The real chances are A 30%, B 70%, C 50%, but the agent does not know them.

    Which snack machine pays best? 100 pulls, made-up machines
    Dayε (explore)Snacks in 100 pullsPulls of A / B / CAgent’s estimates A / B / C
    Day 10%2998 / 1 / 10.30 / 0.00 / 0.00
    Day 110%6526 / 70 / 40.46 / 0.73 / 0.50
    Day 130%6529 / 60 / 110.45 / 0.75 / 0.64
    Day 1100%5837 / 32 / 310.43 / 0.75 / 0.58
    Day 20%771 / 98 / 10.00 / 0.79 / 0.00
    Day 210%785 / 90 / 50.60 / 0.81 / 0.40
    Day 230%7411 / 79 / 100.36 / 0.81 / 0.60
    Day 2100%6332 / 30 / 380.38 / 0.93 / 0.61
    Day 30%443 / 1 / 960.33 / 0.00 / 0.45
    Day 310%706 / 85 / 90.33 / 0.74 / 0.56
    Day 330%6412 / 73 / 150.33 / 0.73 / 0.47
    Day 3100%4841 / 33 / 260.34 / 0.67 / 0.46

    Read Day 1 with ε = 0%: the first pull of A paid and the first pulls of B and C did not, so the never-exploring agent used A 98 times and collected only 29 snacks.

    Keeping score

    Estimates, updates and the price of curiosity

    The agent in the lab keeps one number per machine: its average reward so far.

    Worked example

    How the agent keeps score, and what exploring costs

    1. The resultsThe agent pulls machine B five times: snack, nothing, nothing, snack, snack. Write a snack as 1 and nothing as 0: 1, 0, 0, 1, 1.
    2. Estimate after each pullSnacks so far ÷ pulls so far: 1 ÷ 1 = 1.00, then 1 ÷ 2 = 0.50, then 1 ÷ 3 ≈ 0.33, then 2 ÷ 4 = 0.50, then 3 ÷ 5 = 0.60. Early estimates jump around; they settle as pulls add up.
    3. The update rule (optional)You do not need to store every result. New estimate = old estimate + (reward − old estimate) ÷ number of pulls. For pull 5: 0.50 + (1 − 0.50) ÷ 5 = 0.50 + 0.10 = 0.60, the same answer. Many RL methods use updates of this shape: nudge the estimate toward what just happened.
    4. Price of exploringIn the lab the real chances are A 30%, B 70%, C 50%. Once the agent has found B, ε = 10% means 90% of pulls use B (0.9 × 0.70 = 0.63 snacks per pull) and 10% use a random machine (0.1 × (0.30 + 0.70 + 0.50) ÷ 3 = 0.05). Total: 0.68 snacks per pull, against 0.70 for an agent that magically knew the answer.
    5. Price of never exploringAn agent stuck on A earns 0.30 snacks per pull. Over 100 pulls that is about 30 snacks instead of about 68. A little exploring is cheap insurance.

    The machines and chances are made up. Real agents face the same trade-off with many more actions and situations.

    Where it works

    Games, robots and AlphaGo

    RL shines when an agent can practice an enormous number of times and the score is clear.

    Games are ideal: the rules never change and the score is exact. In 2015, researchers at DeepMind reported a system that learned to play 49 Atari video games using only the screen pixels and the game score as input.

    AlphaGo combined several approaches. It first learned from records of human expert games (supervised learning), then improved by playing against itself with a reward for winning (reinforcement learning), and chose each move with a tree search that looks ahead. In a 2016 paper, its team reported the first win by a computer program against a professional Go player on a full-size board without a handicap. The search part is explained in How does a computer plan without learning?

    Robots often practice in a simulator first, because real trial and error is slow and can break things. Skills learned in simulation do not always carry over to the real world, where floors are slippery and sensors are noisy, so engineers test carefully on real hardware.

    RL is a poor fit when tries are expensive or dangerous, or when nobody can write down a reward that matches what people actually want. That last problem is bigger than it sounds.

    RL and today’s AI agents

    Is a chatbot agent learning from rewards?

    Usually not while you use it. The word “agent” means something different here.

    An AI agent built on a large language model (LLM) is a pretrained model placed in a loop with tools. It reads a goal, writes a step such as a web search or another tool call, gets the result back as text, and decides on the next step. The model’s parameters stay frozen while it runs. Results land in its context window, the text it is working with, not in its parameters. When the task ends, the model itself has not changed.

    RL does show up earlier, during training. In reinforcement learning from human feedback (RLHF), people rank pairs of answers, a second model learns to predict those rankings, and the language model is adjusted toward answers that would score higher. Newer reasoning models are also trained with rewards for answers that can be checked, such as math problems with known results. How transformers work shows these training stages.

    Same word, “agent,” two different things
    A game or robot RL agent, while trainingAn LLM agent while you use it
    Where rewards come fromThe environment: a score after actions, or a win or loss at the endMostly from earlier training stages, such as RLHF or reasoning training, not from your session
    Is it learning as it acts?Yes: rewards update its policyUsually no: the model’s parameters stay fixed
    Where new information goesInto the numbers of its policyInto its context window, as text it can read
    How it picks the next actionIts learned policyThe model predicts the next text, including tool calls, inside a loop that developers wrote
    Next timeIt acts with whatever it learnedIt starts from the same model, unless the app saves notes or a new version is trained

    What about the thumbs-up button? A company can collect that feedback and use it to train a later version of its model. That is a new training run, as on Training vs using, not the agent learning in the middle of your conversation.

    A big limitation

    Careful what you reward

    An agent chases the reward it was given, not the goal its designer had in mind.

    When the reward and the real goal differ, an agent can find a loophole. Researchers call this reward hacking or specification gaming. A 2020 DeepMind article collected dozens of examples. In a boat-racing game, an agent got points for hitting targets along the course, so it learned to circle and hit the same targets over and over instead of finishing the race. In a simulated block task, an agent was rewarded for how high the bottom of a red block was, so it flipped the block upside down instead of stacking it on a blue one.

    The agent is not cheating on purpose; it is doing exactly what the numbers say. The fix is a better-specified goal, plus testing and human oversight, which is part of what people mean by alignment.

    Play it

    Careful What You Wish For

    In this ARQIV game a very literal genie does exactly what you said, not what you meant. Writing a reward is a lot like writing a wish.

    Play the game →

    4 · Common mix-ups

    What people get wrong about rewards

    Four ideas that trip people up, and what to think instead.

    • Mix-up: The agent wants the reward.

      Better: A reward is a number the designer chose. The agent changes its internal numbers to collect more of it, with no feelings and no idea what the designer meant.

    • Mix-up: A chatbot agent learns from every task it does for you.

      Better: Its model is usually frozen at run time. An app may save notes about you, but the model’s parameters change only when a new version is trained.

    • Mix-up: Exploring is a waste of time.

      Better: A little exploring costs a little reward. Never exploring can trap the agent on a bad early guess, as Day 1 in the lab shows.

    • Mix-up: Reinforcement learning needs no data.

      Better: The agent’s own experience is its data, and it usually needs a lot: often far more tries than a person would.

    5 · Check yourself

    Three quick questions

    Answer in your head or on paper before you open each one.

    1. A robot dog learns to walk in a simulator. It gets +1 for each meter it moves forward and −10 each time it falls. Name the agent, the environment, two actions, the reward and the policy.

      Check your answer

      Agent: the dog’s controller. Environment: the simulated ground and physics. Actions: for example, move a leg, shift weight. Reward: +1 per meter, −10 per fall. Policy: its rule for which movement to make from each position, which RL gradually improves.

    2. In the lab, an agent with ε = 0% can spend almost all its pulls on machine A, the worst one. How can that happen?

      Check your answer

      It tries each machine once. If A’s first pull pays and B’s and C’s do not, A has the best average (1.00 against 0). A greedy agent always picks the best average, so it never tries B or C again, and their estimates stay at 0 forever. Its policy was shaped by one unlucky round.

    3. Your friend says, “I gave the travel agent bot a thumbs-up, so it learned to book trains better.” What is more likely true?

      Check your answer

      The model inside the bot did not change during the conversation; its parameters were frozen. The thumbs-up may be stored and used later to train a new version, for example with RLHF, or the app may save a note. Either way, the learning happens in a separate training step, not in the moment.

    For younger learners

    A grades 3–5 version

    No percentages, just tally marks and a choice each turn.

    20 minutes · teacher-led

    Three paper bags

    1. Fill three bags with 10 tokens each: 3, 7 and 5 of them are stars. Do not say which bag is which.
    2. Each turn, a team picks a bag, draws one token, scores a point for a star and puts the token back.
    3. Tally each bag’s stars on the board. After 20 turns, ask: which bag is best, and how sure are you?

    Talk about it

    Why try the other bags?

    Teams that only used their first lucky bag often miss the best one. Good answers say that trying other bags sometimes costs a point but helps you find out more. That is exploring.

    Honest limits

    What this page leaves out

    What this page leaves out

    The lab has only one situation, so its policy is just “which machine.” Most RL problems have many situations, and the agent must learn what to do in each and how today’s action affects later rewards. That needs ideas this page skips: value functions, Q-learning, discounting future rewards, policy gradients and learning with a model of the world. Sutton and Barto’s free textbook covers them.

    Sources

    Where this comes from

    1. Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd edition (2018), free online. Chapter 2 covers multi-armed bandits, epsilon-greedy and incremental averages.
    2. Volodymyr Mnih et al. (2015), “Human-level control through deep reinforcement learning,” Nature: one agent learning 49 Atari games from pixels and score.
    3. David Silver et al. (2016), “Mastering the game of Go with deep neural networks and tree search,” Nature: AlphaGo’s supervised learning, self-play reinforcement learning and tree search.
    4. Long Ouyang et al. (2022), “Training language models to follow instructions with human feedback”: the RLHF recipe of human rankings, a reward model and reinforcement learning.
    5. DeepSeek-AI (2025), “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning”: training reasoning with rewards for checkable answers.
    6. Victoria Krakovna et al., DeepMind (2020), “Specification gaming: the flip side of AI ingenuity”: the boat-race and block-flipping examples.

    AI words on this page

    Reinforcement learning, Reward and policy, RLHF, AI agent, Reasoning model, Supervised learning, Parameters, Context window, Tool call, Search and planning, Alignment

    All definitions live in one place: AI words in plain language →

    Review status

    Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.