Go deeper · How machines learn
How does a machine learn from rewards?
By trying actions and keeping track of which ones lead to rewards. In reinforcement learning, nobody gives the learner the right answer for each step. It acts, gets a score back, and slowly shifts toward the actions that earned more. The hard part is deciding when to try something new and when to stick with what already works.
What you will be able to do
Name the agent, environment, actions, reward and policy in an example, explain the trade-off between exploring and exploiting, and say why an AI chatbot agent is usually not learning from rewards while you use it.
Before you start
Read How can examples become a rule? first. How AI works introduces reinforcement learning as one of three ways to learn; this page goes deeper.
1 · You have already seen this
Learning a game by playing it
Many people have started a new video game without reading the rules.
You press buttons and watch what happens. Picking up a coin raises your score, and falling off a ledge costs you. After a while you know which moves pay off on each screen, even though nobody gave you a list of right answers.
Reinforcement learning trains a computer program in a similar way. The program acts, gets back a number that says how well it did, and adjusts what it does next time. This page shows how that works, lets you try it with three snack machines, and shows where it goes wrong.
2 · What happens, step by step
From one try to a better rule
Reinforcement learning in five steps. The next section goes deeper into steps 1 and 2, and the one after it into step 4. After the lab, a worked example goes deeper into steps 3 and 4.
How an agent learns
Five steps
- 1 · Take an actionThe learner, called the agent, makes one choice, such as jump or move left in a game. It chooses with its policy, its current rule for picking an action.
- 2 · Get a rewardThe world it acts on changes and sends back a number that a person designed, such as +10 for a coin or −100 for falling.
- 3 · Update the numbersThe agent nudges its numbers toward what just happened, so actions that scored well become more likely next time.
- 4 · Sometimes try something elseMost of the time the agent picks the best action so far. Now and then it tries another one, so it keeps learning about the others.
- 5 · Repeat many timesThe loop runs thousands or millions of times. Over those rounds the policy changes so the agent collects more reward in total.
Going deeper · steps 1 and 2
Act, get a score, adjust
Reinforcement learning (RL) is about a learner inside a situation, not a learner reading a table of labeled examples.
In supervised learning, every training example comes with the right answer. In reinforcement learning there is no answer key. The learner, called the agent, takes an action in an environment: a game, a simulator, a room. The environment changes and sends back a reward, a number that says how good the result was. Then the agent acts again.
Over many rounds the agent builds a policy: its rule for choosing an action in each situation. Learning means changing the policy so that it collects more reward in total. Rewards can arrive late. In a board game the only clear score might be a win or a loss at the very end, so the agent has to work out which earlier moves deserve the credit.
A reward is just a number that a person designed. The agent does not feel pleased. It adjusts numbers inside itself so that high-scoring actions become more likely next time.
| Word | Meaning | In a video game | In a robot vacuum (practicing in a simulator) |
|---|---|---|---|
| Agent | The learner that chooses actions | The program pressing the buttons | The vacuum’s controller |
| Environment | Everything the agent acts on | The game world | A simulated room with furniture |
| Action | One choice the agent can make | Jump, move left, wait | Drive forward, turn, stop |
| Reward | A number sent back after an action | +10 for a coin, −100 for falling | +1 per clean tile, −5 for a bump |
| Policy | The agent’s rule for picking an action in each situation | Which button to press on each screen | Which way to drive from each spot |
Going deeper · step 4
Try something new, or stick with the best so far?
Every RL agent faces this dilemma. So do you, every time you choose where to eat lunch.
Predict first
Three snack machines stand in a hallway. You try each one once: A gives you a snack, B and C give nothing. Should you use only A from now on?
Show the answer
Not yet. One try each is very little evidence. B might pay out most of the time and just have been unlucky once. If you only ever use A, you will never find out. Exploiting means using the best option so far; exploring means trying others to learn more. A good strategy does mostly the first and some of the second.
A common simple strategy is called epsilon-greedy. Epsilon (ε) is a Greek letter that often stands for a small number. Most of the time the agent is greedy: it picks the machine with the best average so far. With a small probability ε, say on 10% of pulls, it picks a machine at random instead. That bit of exploring costs a few snacks, but it protects the agent from getting stuck on a lucky first guess. The lab lets you watch both happen.
3 · Try it
Watch an agent learn by trying
Pick an explore setting and a day of luck, then let the agent pull.
Lab
Which snack machine pays best?
Three made-up machines each pay a snack with a hidden chance. Choose how often the agent explores, let it pull, and watch its estimates and its total. Every “day” replays the same luck, so you can compare settings fairly.
Simulated in your browser with made-up machines and chances. Nothing is learned from you or stored.
Last pulls, newest first:
Three challenges
- Day 1, ε = 0%: press “100 pulls.” Which machine is the agent stuck on, and why? Reset and try ε = 10%.
- Day 2: compare ε = 0% with ε = 30%. When does exploring cost snacks instead of saving them?
- Set ε = 100%. Why do the estimates get close to the real chances while the total stays near half the pulls?
The agent tries each machine once, then pulls 97 more times. With probability ε it picks a machine at random (exploring); otherwise it picks the machine with the best average so far (exploiting). The real chances are A 30%, B 70%, C 50%, but the agent does not know them.
| Day | ε (explore) | Snacks in 100 pulls | Pulls of A / B / C | Agent’s estimates A / B / C |
|---|---|---|---|---|
| Day 1 | 0% | 29 | 98 / 1 / 1 | 0.30 / 0.00 / 0.00 |
| Day 1 | 10% | 65 | 26 / 70 / 4 | 0.46 / 0.73 / 0.50 |
| Day 1 | 30% | 65 | 29 / 60 / 11 | 0.45 / 0.75 / 0.64 |
| Day 1 | 100% | 58 | 37 / 32 / 31 | 0.43 / 0.75 / 0.58 |
| Day 2 | 0% | 77 | 1 / 98 / 1 | 0.00 / 0.79 / 0.00 |
| Day 2 | 10% | 78 | 5 / 90 / 5 | 0.60 / 0.81 / 0.40 |
| Day 2 | 30% | 74 | 11 / 79 / 10 | 0.36 / 0.81 / 0.60 |
| Day 2 | 100% | 63 | 32 / 30 / 38 | 0.38 / 0.93 / 0.61 |
| Day 3 | 0% | 44 | 3 / 1 / 96 | 0.33 / 0.00 / 0.45 |
| Day 3 | 10% | 70 | 6 / 85 / 9 | 0.33 / 0.74 / 0.56 |
| Day 3 | 30% | 64 | 12 / 73 / 15 | 0.33 / 0.73 / 0.47 |
| Day 3 | 100% | 48 | 41 / 33 / 26 | 0.34 / 0.67 / 0.46 |
Read Day 1 with ε = 0%: the first pull of A paid and the first pulls of B and C did not, so the never-exploring agent used A 98 times and collected only 29 snacks.
Keeping score
Estimates, updates and the price of curiosity
The agent in the lab keeps one number per machine: its average reward so far.
Worked example
How the agent keeps score, and what exploring costs
- The resultsThe agent pulls machine B five times: snack, nothing, nothing, snack, snack. Write a snack as 1 and nothing as 0: 1, 0, 0, 1, 1.
- Estimate after each pullSnacks so far ÷ pulls so far: 1 ÷ 1 = 1.00, then 1 ÷ 2 = 0.50, then 1 ÷ 3 ≈ 0.33, then 2 ÷ 4 = 0.50, then 3 ÷ 5 = 0.60. Early estimates jump around; they settle as pulls add up.
- The update rule (optional)You do not need to store every result. New estimate = old estimate + (reward − old estimate) ÷ number of pulls. For pull 5: 0.50 + (1 − 0.50) ÷ 5 = 0.50 + 0.10 = 0.60, the same answer. Many RL methods use updates of this shape: nudge the estimate toward what just happened.
- Price of exploringIn the lab the real chances are A 30%, B 70%, C 50%. Once the agent has found B, ε = 10% means 90% of pulls use B (0.9 × 0.70 = 0.63 snacks per pull) and 10% use a random machine (0.1 × (0.30 + 0.70 + 0.50) ÷ 3 = 0.05). Total: 0.68 snacks per pull, against 0.70 for an agent that magically knew the answer.
- Price of never exploringAn agent stuck on A earns 0.30 snacks per pull. Over 100 pulls that is about 30 snacks instead of about 68. A little exploring is cheap insurance.
The machines and chances are made up. Real agents face the same trade-off with many more actions and situations.
Where it works
Games, robots and AlphaGo
RL shines when an agent can practice an enormous number of times and the score is clear.
Games are ideal: the rules never change and the score is exact. In 2015, researchers at DeepMind reported a system that learned to play 49 Atari video games using only the screen pixels and the game score as input.
AlphaGo combined several approaches. It first learned from records of human expert games (supervised learning), then improved by playing against itself with a reward for winning (reinforcement learning), and chose each move with a tree search that looks ahead. In a 2016 paper, its team reported the first win by a computer program against a professional Go player on a full-size board without a handicap. The search part is explained in How does a computer plan without learning?
Robots often practice in a simulator first, because real trial and error is slow and can break things. Skills learned in simulation do not always carry over to the real world, where floors are slippery and sensors are noisy, so engineers test carefully on real hardware.
RL is a poor fit when tries are expensive or dangerous, or when nobody can write down a reward that matches what people actually want. That last problem is bigger than it sounds.
RL and today’s AI agents
Is a chatbot agent learning from rewards?
Usually not while you use it. The word “agent” means something different here.
An AI agent built on a large language model (LLM) is a pretrained model placed in a loop with tools. It reads a goal, writes a step such as a web search or another tool call, gets the result back as text, and decides on the next step. The model’s parameters stay frozen while it runs. Results land in its context window, the text it is working with, not in its parameters. When the task ends, the model itself has not changed.
RL does show up earlier, during training. In reinforcement learning from human feedback (RLHF), people rank pairs of answers, a second model learns to predict those rankings, and the language model is adjusted toward answers that would score higher. Newer reasoning models are also trained with rewards for answers that can be checked, such as math problems with known results. How transformers work shows these training stages.
| A game or robot RL agent, while training | An LLM agent while you use it | |
|---|---|---|
| Where rewards come from | The environment: a score after actions, or a win or loss at the end | Mostly from earlier training stages, such as RLHF or reasoning training, not from your session |
| Is it learning as it acts? | Yes: rewards update its policy | Usually no: the model’s parameters stay fixed |
| Where new information goes | Into the numbers of its policy | Into its context window, as text it can read |
| How it picks the next action | Its learned policy | The model predicts the next text, including tool calls, inside a loop that developers wrote |
| Next time | It acts with whatever it learned | It starts from the same model, unless the app saves notes or a new version is trained |
What about the thumbs-up button? A company can collect that feedback and use it to train a later version of its model. That is a new training run, as on Training vs using, not the agent learning in the middle of your conversation.
A big limitation
Careful what you reward
An agent chases the reward it was given, not the goal its designer had in mind.
When the reward and the real goal differ, an agent can find a loophole. Researchers call this reward hacking or specification gaming. A 2020 DeepMind article collected dozens of examples. In a boat-racing game, an agent got points for hitting targets along the course, so it learned to circle and hit the same targets over and over instead of finishing the race. In a simulated block task, an agent was rewarded for how high the bottom of a red block was, so it flipped the block upside down instead of stacking it on a blue one.
The agent is not cheating on purpose; it is doing exactly what the numbers say. The fix is a better-specified goal, plus testing and human oversight, which is part of what people mean by alignment.
Play it
Careful What You Wish For
In this ARQIV game a very literal genie does exactly what you said, not what you meant. Writing a reward is a lot like writing a wish.
4 · Common mix-ups
What people get wrong about rewards
Four ideas that trip people up, and what to think instead.
Mix-up: The agent wants the reward.
Better: A reward is a number the designer chose. The agent changes its internal numbers to collect more of it, with no feelings and no idea what the designer meant.
Mix-up: A chatbot agent learns from every task it does for you.
Better: Its model is usually frozen at run time. An app may save notes about you, but the model’s parameters change only when a new version is trained.
Mix-up: Exploring is a waste of time.
Better: A little exploring costs a little reward. Never exploring can trap the agent on a bad early guess, as Day 1 in the lab shows.
Mix-up: Reinforcement learning needs no data.
Better: The agent’s own experience is its data, and it usually needs a lot: often far more tries than a person would.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
A robot dog learns to walk in a simulator. It gets +1 for each meter it moves forward and −10 each time it falls. Name the agent, the environment, two actions, the reward and the policy.
Check your answer
Agent: the dog’s controller. Environment: the simulated ground and physics. Actions: for example, move a leg, shift weight. Reward: +1 per meter, −10 per fall. Policy: its rule for which movement to make from each position, which RL gradually improves.
In the lab, an agent with ε = 0% can spend almost all its pulls on machine A, the worst one. How can that happen?
Check your answer
It tries each machine once. If A’s first pull pays and B’s and C’s do not, A has the best average (1.00 against 0). A greedy agent always picks the best average, so it never tries B or C again, and their estimates stay at 0 forever. Its policy was shaped by one unlucky round.
Your friend says, “I gave the travel agent bot a thumbs-up, so it learned to book trains better.” What is more likely true?
Check your answer
The model inside the bot did not change during the conversation; its parameters were frozen. The thumbs-up may be stored and used later to train a new version, for example with RLHF, or the app may save a note. Either way, the learning happens in a separate training step, not in the moment.
For younger learners
A grades 3–5 version
No percentages, just tally marks and a choice each turn.
20 minutes · teacher-led
Three paper bags
- Fill three bags with 10 tokens each: 3, 7 and 5 of them are stars. Do not say which bag is which.
- Each turn, a team picks a bag, draws one token, scores a point for a star and puts the token back.
- Tally each bag’s stars on the board. After 20 turns, ask: which bag is best, and how sure are you?
Talk about it
Why try the other bags?
Teams that only used their first lucky bag often miss the best one. Good answers say that trying other bags sometimes costs a point but helps you find out more. That is exploring.
Honest limits
What this page leaves out
What this page leaves out
The lab has only one situation, so its policy is just “which machine.” Most RL problems have many situations, and the agent must learn what to do in each and how today’s action affects later rewards. That needs ideas this page skips: value functions, Q-learning, discounting future rewards, policy gradients and learning with a model of the world. Sutton and Barto’s free textbook covers them.
Where to go next
From rewards to plans
Next in this branch
How does a computer plan without learning?
Search trees and heuristics: the other half of AlphaGo.
AI at work
What changes when an AI can take actions?
Tools, permissions and loops in LLM agents.
Back to basics
How AI works
Three ways to learn, and the training loop behind neural networks.
RL for chatbots
Inside a large language model
RLHF and reward models, DPO, GRPO and RLVR: how rewards turn a text predictor into a reasoning assistant.
Sources
Where this comes from
- Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An Introduction, 2nd edition (2018), free online. Chapter 2 covers multi-armed bandits, epsilon-greedy and incremental averages.
- Volodymyr Mnih et al. (2015), “Human-level control through deep reinforcement learning,” Nature: one agent learning 49 Atari games from pixels and score.
- David Silver et al. (2016), “Mastering the game of Go with deep neural networks and tree search,” Nature: AlphaGo’s supervised learning, self-play reinforcement learning and tree search.
- Long Ouyang et al. (2022), “Training language models to follow instructions with human feedback”: the RLHF recipe of human rankings, a reward model and reinforcement learning.
- DeepSeek-AI (2025), “DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning”: training reasoning with rewards for checkable answers.
- Victoria Krakovna et al., DeepMind (2020), “Specification gaming: the flip side of AI ingenuity”: the boat-race and block-flipping examples.
AI words on this page
Reinforcement learning, Reward and policy, RLHF, AI agent, Reasoning model, Supervised learning, Parameters, Context window, Tool call, Search and planning, Alignment
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.