AI foundations · 05

Learning now, or using what it learned?

Training is when a model’s internal numbers, its parameters, are adjusted to fit many examples. Inference is when the finished model, with those numbers frozen, turns a new input into an output. When you ask a chatbot a question, you are almost always doing inference: the model answers, and its parameters stay exactly as they were.

Grades 6+20 minutesArithmetic and squaresOptional gradient math for grades 9+

What you will be able to do

Say whether a situation is training, inference or retrieval, and describe one training step in four moves: predict, measure the loss, find the gradient, update.

Before you start

Stops 01 and 02 help. You should be comfortable with negative numbers and squaring a number. Tensors (stop 04) are useful but not needed.

1 · You have already seen this

The answer comes fast because the learning is done

Ask a chatbot a question and it starts answering right away.

That answer comes from a model that was trained earlier. When you ask something new, the app only runs the finished model, and the model stays the same however many questions it answers: its internal numbers stay exactly as they were. Photo apps that label your pictures work the same way.

These are two separate jobs. Training adjusts a model’s numbers, its parameters, to fit many examples. Inference uses the finished model, with those numbers frozen, to turn a new input into an output. The rest of this page shows how that works, lets you train a small model yourself, and shows where it goes wrong.

2 · What happens, step by step

Build it once, use it many times

Training and inference are separate jobs, often done on different computers at different times.

Training · learning now

Examples → training → saved model

  1. ExamplesInputs paired with the right answers: weeks 1 to 4 and the heights measured.
  2. Training loopPredict, measure the error (the loss), nudge the parameters, repeat many times.
  3. Saved modelThe final parameters are written to a file. Here that is just two numbers, w and b.

Inference · using what it learned

New input + saved model → output

  1. New inputA case the model never saw, such as week 5.
  2. Saved model, frozenThe same w and b, loaded from the file. Nothing changes them.
  3. OutputA prediction, such as about 11 cm. Run it a million times and w and b stay the same.

A model is like a recipe with adjustable numbers in it, called parameters. During training, a computer tries the recipe on examples where the right answer is known, measures how wrong it was, and adjusts the numbers a little. It repeats this many times, then saves the numbers.

During inference, the saved recipe is used as it is. A new input goes in, an output comes out, and the recipe does not change. A large language model may be trained for weeks or months on thousands of specialized chips; after that it can answer millions of questions a day without its parameters changing at all.

Think of a bakery. The head baker refines a cake recipe over many trial batches (training), then prints the final recipe card that every shift bakes from (inference). Nobody rewrites the card each time a customer orders a cake.

Going deeper · the training loop

Four moves, repeated

Here is the smallest model that still trains the way big neural networks do: a straight line with two parameters.

Imagine a made-up bean plant, measured once a week: 3 cm after week 1, 6 cm after week 2, 7 cm after week 3 and 9 cm after week 4. The model guesses height from the week with a line: height = w × week + b. The parameter w is the slope, the centimeters added per week, and b is where the line starts. Training means finding good values for w and b.

In neural networks, a number like b is called a bias. That is just a learned offset; it has nothing to do with unfairness. Each step of training makes the same four moves:

  1. PredictRun the model on the training examples with the current parameters.
  2. Measure the lossTurn “how wrong” into one number. Here: square each error, then average. That is the mean squared error (MSE). Squaring makes every error positive and punishes big misses most.
  3. Find the gradientFor each parameter, work out how steeply the loss rises if that parameter increases. Those slopes together are the gradient.
  4. UpdateMove each parameter a small step the opposite way, downhill. The learning rate sets how big the step is.

Worked example

One training step with real numbers

  1. Startw = 1, b = 0. Predictions for weeks 1 to 4: 1, 2, 3, 4 cm.
  2. ErrorsPrediction minus real height: 1 − 3 = −2, 2 − 6 = −4, 3 − 7 = −4, 4 − 9 = −5. Every guess is too low.
  3. Loss (MSE)Squares: 4, 16, 16, 25. Sum 61. Mean: 61 ÷ 4 = 15.25.
  4. GradientFor w, multiply each error by its week and add: −2 − 8 − 12 − 20 = −42. Then 2 × (−42) ÷ 4 = −21. For b, just add the errors: −15. Then 2 × (−15) ÷ 4 = −7.5. Both are negative, so making w or b bigger would lower the loss. (Why 2 ÷ 4? See the optional math below.)
  5. UpdateLearning rate 0.05. New w = 1 − 0.05 × (−21) = 2.05. New b = 0 − 0.05 × (−7.5) = 0.375.
  6. CheckNew predictions 2.425, 4.475, 6.525, 8.575. Errors −0.575, −1.525, −0.475, −0.425. Squares add to 0.330625 + 2.325625 + 0.225625 + 0.180625 = 3.0625, so the loss is 3.0625 ÷ 4 ≈ 0.77. One step cut the loss from 15.25 to 0.77.

Made-up measurements for a made-up plant. The best possible line for these four points is w = 1.9, b = 1.5, with loss 0.175. No straight line goes through all four points, so the loss can never reach 0.

Where the gradient formulas come from (optional, grades 9+)

With n examples, the loss is

L(w, b) = (1/n) · Σ (w·xi + b − yi)²

Take the derivative of each squared error with the chain rule: the outside gives 2 × (error), the inside gives xi for w and 1 for b. So

∂L/∂w = (2/n) · Σ (errori · xi) and ∂L/∂b = (2/n) · Σ errori

Gradient descent then updates both at once, with learning rate η:

w ← w − η · ∂L/∂w and b ← b − η · ∂L/∂b

For a network with millions of parameters nobody writes these by hand: backpropagation applies the same chain rule, layer by layer, automatically.

Backpropagation and the optimizer do different jobs

For a line, you can write the gradient by hand. A neural network can have billions of parameters arranged in layers, and each one affects the loss through every layer after it. Backpropagation is the method that computes all those gradients efficiently, working backward from the loss through the layers with the chain rule from calculus. It only calculates. It changes nothing.

The optimizer then uses the gradients to update the parameters. Plain gradient descent, as in the worked example, is the simplest optimizer; popular variants such as Adam adjust each parameter’s step using recent history. In PyTorch code you can see the split: loss.backward() computes the gradients and optimizer.step() applies the update. On How AI works this is walking downhill in fog: backpropagation feels the slope under your feet, the optimizer takes the step.

Big models take each step after a small batch of examples, not the whole dataset; one pass through all the training data is an epoch.

3 · Try it

Train a line, then freeze it

Fit the line by hand first. Then let gradient descent take steps, try a bigger learning rate, and finally freeze the model and use it.

Predict first

In the worked example the learning rate was 0.05. If you make it 0.2, four times bigger, will one step from w = 1, b = 0 lower the loss more, or less?

Show the answer

Neither: the loss goes up. The step is so big that it jumps past the best line to w = 5.2, b = 1.5, and the loss rises from 15.25 to 81.85. The next step overshoots even further (loss about 447). Try it in the lab, then press Reset. A moderately bigger rate, 0.1, still works and ends up a little faster than 0.05.

Lab

Train, then use

Move w and b to lower the loss, or press “Training step” to let gradient descent make one update and show its arithmetic. Change the learning rate to see small, good and too-big steps. Then press “Freeze and use” to predict new weeks without changing w or b.

Simulated in your browser with four made-up measurements. Plain arithmetic, no AI service.

Chart: the four measurements as dots, the model’s current line, and the errors as vertical dashed lines. The table next to it lists the same numbers.Adjust the model by hand

Challenge: get the loss below 0.2 by hand.

Predictions and errors for the four measurements
Week xReal yPredictedErrorError²
Mean of the squared errors (loss)
Learning rate (step size)

The lab needs JavaScript. Here are the numbers it would show, starting from w = 1, b = 0 with the four made-up measurements (week 1: 3 cm, week 2: 6 cm, week 3: 7 cm, week 4: 9 cm).

Training numbers for the bean-plant line (made-up data)
SituationwbLoss (MSE)What it shows
Start1015.25Every prediction is too low.
After one step, learning rate 0.052.050.3750.766One careful step cuts the loss by about 95%.
After 10 steps, learning rate 0.052.2130.5800.316Progress slows as the slope flattens.
After one step, learning rate 0.25.21.581.85Too big a step overshoots: the loss goes up.
After two steps, learning rate 0.2−4.7−1.8446.67Each step overshoots further: training blows up.
Best possible line1.91.50.175No straight line goes through all four points.

Inference with the best line, frozen: week 5 → 1.9 × 5 + 1.5 = 11 cm; week 6 → 12.9 cm; week 50 → 96.5 cm. Making predictions never changes w or b. The week-50 answer is silly because the line only ever saw weeks 1 to 4.

Everyday AI

Does a chatbot learn from you while you chat?

Usually not in the training sense. Several different mechanisms can make it look as if it did.

Predict first

You tell a chat app that your favorite color is green. A week later, in a brand-new chat, it mentions green. Was the model retrained on your message?

Show the answer

Probably not. The likeliest explanation is saved memory: the app stored a note about you and quietly added it to the input of the new chat. The model’s parameters did not change. Apps with a memory feature usually let you see and delete what they saved.

Five things that can look like “the AI learned from me”
What happensHow it worksDo the model’s parameters change?How long it lasts
ContextYour earlier messages in the conversation are sent to the model again with each new message, as part of its input (its context window).NoUntil the conversation ends or grows too long to fit
Saved memorySome apps save notes about you (“likes green, learning Spanish”) in a database and add them to the input of later chats.NoUntil you or the app deletes the note
RetrievalThe system searches documents or the web and pastes the passages it finds into the input. See Retrieval and RAG.NoJust for that answer
Later trainingDepending on the company’s policy and your settings, saved conversations may become data for a future training run, which produces a new model version.Yes, but in a separate training run, not during your chatUntil that version is replaced
Online learning exceptionSome systems do update their parameters continually as new data arrives, for example some spam filters and recommendation feeds that adjust as people click.Yes, in small, frequent updatesOngoing

Other learners

Not every model learns with gradients

Gradient descent trains neural networks and many linear models. Other machine learning methods learn differently, but they still have a training phase and a using phase.

Decision tree

Training picks questions

Training builds a series of yes-or-no questions, such as “Is the fruit heavier than 150 g?” At each branch it tries candidate questions and keeps the one that best separates the examples, by counting. No gradients and no learning rate. Inference: follow the questions down to an answer.

k-nearest neighbors

Training mostly stores

Training is mostly saving the examples. Inference does the work: find the k stored examples closest to the new input and let them vote, or average their values. Here using the model is the slow part, and adding a new example changes later answers without any gradient steps.

More on these on Do you always need a neural network?

In the real world

An application, a limit and a link

Application

Weather forecasts

Some newer forecasting models are trained once on decades of past weather, then run (inference) several times a day on the latest observations. Training takes weeks; each forecast takes minutes. See AI in weather.

Limitation

Frozen means stuck in the past

A frozen model knows only what its training data showed. The bean line predicts 96.5 cm at week 50, because a straight line is all it knows; nothing in weeks 1 to 4 said that plants stop growing. Chatbots likewise know nothing after their training data was collected unless it is put into their input.

Connection

Train, then test, with a class

In Train the Machine, students first train a creature classifier, then test it on new creatures: the same two phases. How AI works shows the loop inside a neural network.

4 · Common mix-ups

Learning and using are easy to blur

Four ideas that trip people up, and what to think instead.

  • Mix-up: Every chat message retrains the model.

    Better: Chat messages become part of the input (the context). Parameters change only in a training run, which for big models happens separately and much later, if at all.

  • Mix-up: Backpropagation is how a network learns.

    Better: Backpropagation only calculates gradients. The optimizer uses them to change the parameters. Training needs both, plus data and a loss.

  • Mix-up: “Inference” means the AI is reasoning or drawing conclusions.

    Better: In machine learning, inference just means running a trained model on an input. (In statistics the word means drawing conclusions from data, which is different again.)

  • Mix-up: A lower training loss always means a better model.

    Better: Loss on the training examples can keep falling while the model gets worse on new data (overfitting). Test on examples it never trained on; see Evaluating AI.

5 · Check yourself

Training, inference or retrieval?

Label each scenario before you open the answer. Some have more than one label.

  1. A photo app labels your new picture “dog” half a second after you take it.

    Check your answer

    Inference. A model trained earlier runs on your new photo. Its parameters do not change, however many photos you label.

  2. A lab runs thousands of chips for weeks, repeatedly adjusting a model’s parameters to lower its loss on billions of text examples.

    Check your answer

    Training. Parameters change, guided by a loss and its gradients. The result is a saved model that can then be used for inference.

  3. A school help bot searches the student handbook, finds the paragraph on late homework, and answers using it.

    Check your answer

    Retrieval and inference. The search step retrieves a passage; then the model runs (inference) with that passage in its input. No training happens: when the handbook changes, the bot learns about it through retrieval, not by changing parameters. See Retrieval and RAG.

For younger learners

A grades 3–5 version

No loss or gradients: the difference between working out a rule and using it.

15 minutes · small groups

The sealed rule card

  1. Show number pairs: 2 → 5, 3 → 7, 4 → 9. Groups guess the rule and test it on the pairs, changing it until it works (training).
  2. Each group writes its final rule on a card and seals it in an envelope.
  3. Call out new numbers, such as 10 and 0. Groups may answer only by using the card (inference). The rule “double it and add 1” gives 21 and 1.
  4. Ask: if your card gives a wrong answer, may you change it during the quiz? (Not without going back to training.)

Exit question

Learning or using?

“When were you learning the rule, and when were you only using it?” Good answers name both moments: “We learned it while we tested the pairs. We used it when we answered from the card.”

Honest limits

What this page leaves out

What this page leaves out

Mini-batches and epochs in detail, optimizers such as Adam, how to decide when to stop training (validation data), the later training stages that turn a language model into an assistant (see how assistants are trained), and how models are deployed, monitored and updated. Real neural networks also have many layers, nonlinear functions and far more data than one line.