Go deeper · Advanced track

What happens after a model is trained?

It goes to work, and the work does not stop. A team saves exactly which data, code and settings made each version, puts one version into use, and keeps checking it on new cases. When a new release breaks, they roll back to an older version. When the world changes, they retrain. Doing all of this carefully is called MLOps.

Grades 9+30 minutesPercentagesOptional advanced page

This page is part of the optional advanced track. You do not need it for the main route.

What you will be able to do

Name the four parts of a model version, tell data drift from concept drift on a weekly dashboard, choose an alert threshold, and decide whether a problem needs a rollback or retraining.

Before you start

Read first: Can you build and test a model yourself? (this page continues its library model), Good at what, compared with what? (accuracy and baselines) and How does AI predict tomorrow and spot the unusual? (drift and false alarms).

1 · You have already seen this

The filter that has to keep up

Most people have had a scam message slip past a spam filter that used to catch messages like it.

A spam filter learned from messages that people marked as scam or not scam. Scammers keep changing what they write, so a filter that did well last year can let new kinds of scam through this year. The people who run it have to keep checking how it does and train it again when it falls behind.

Looking after a model once it is in real use is called MLOps. This page follows a school library’s model that predicts which loans will come back late, starting on the day it is switched on. It shows how that work goes, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From switching it on to the next version

Looking after a model in four steps. The next section goes deeper into step 2, and the two after it into steps 1 and 3. After the lab, a worked example and a section on responding go deeper into step 4.

How a model is looked after

Four steps

  1. 1 · Save the versionRecord the four parts that made the model: the exact data, the code, the trained model and its settings. Give that combination a name, such as v1.
  2. 2 · Put it to workSwitch the model on so it makes real predictions. The library model sends an early reminder when a student borrows 4 or more books at once.
  3. 3 · Check it every weekLook at this week’s inputs, and score the predictions once the right answers arrive. For a loan, that is only after its due date.
  4. 4 · Act on an alertWhen a number crosses the line you set, find out what changed before doing anything.

Going deeper · step 2

The model went live. Now what?

We pick up the library model from the build-a-model page on the day it starts sending real reminders.

On the build-a-model page, a school library trained a one-question model: if a student borrows 4 or more books at once, predict the loan will come back late and send a friendly reminder early. It beat the baseline on 8 test loans. Now the library switches it on. Putting a trained model into real use is called deployment.

From then on the model predicts loans nobody has seen yet. Its parameters do not change by themselves, but the students, the timetable and the library’s rules do. Each week some loans fall due, and the library learns which predictions were right.

Careful teams record every version, watch it after release and plan for failure. This practice is called MLOps (machine learning operations). A 2015 paper by Google engineers argued that the model is only a small part of a real machine learning system: most of the work, and the risk, sits in the data, code and checks around it.

The life of a deployed model A loop. Train and test, then save the version (data, code, model and settings), then deploy it. A deployed model is monitored every week. If an alert fires, a bad new release is fixed by rolling back to a saved version, and a changed world is fixed by retraining on new labeled data, which starts the loop again. With no alert, the team keeps watching. Train and test beat the baseline Save the version data, code, model, settings Deploy real predictions Monitor inputs every week, results once labels arrive Alert? a number crossed your line bad release: roll back world changed: retrain No alert: keep watching. Every new version gets a new name and an updated model card.
The loop never really ends. Rollback fixes a bad release; retraining fixes a changed world.

Going deeper · step 1

Write down everything that made it

If you cannot rebuild a model, you cannot fix it, compare it or go back to it.

Four things that make up one model version
Part of a versionLibrary exampleWhy record it
DataThe exact 16 training loans and 8 test loans, saved as a fixed copyNew rows make a different model. You need to know which rows made this one.
CodeThe script that turns checkout records into features and trains the modelOne changed line can change every prediction.
ModelThe trained result: “4 or more books → late.” For a neural network, a file of parametersThis is what you deploy, and what you roll back to.
SettingsHyperparameters, the random seed, the versions of the software librariesSame data and code with different settings can give a different model.

Rebuilding the same model from those records is called reproducibility. Many training methods use random numbers, for example to shuffle rows or to pick a neural network’s starting weights. A random seed is the number that starts that random sequence. Fix it, and the “random” choices come out the same on every run. Even then, different software versions or hardware can cause small differences, so teams record those too.

Each saved combination gets a name, such as v1 and v2. That list of versions is what makes going back possible.

Predict first

Two teammates train “the same” model on the same day and get different test scores. Name three things that could differ.

Show the answer

Any of these: a different copy of the data (one took it before new loans were added), a different random seed (so rows were shuffled or split differently), a different version of the code or of the machine learning library, or different settings such as how deep a tree may grow. Recording all four parts of a version turns a mystery into a quick check.

Going deeper · step 3

Keep checking on new cases

A test score describes the past. Monitoring tells you about this week.

Monitoring means checking a deployed model regularly. The library’s dashboard has two checks. The input check: what share of this week’s loans had 4 or more books? In training it was 6 of 16, or 37.5%. The results check: of the loans now due, how many did the model get right? That needs labels, which arrive late: you only know a loan was late after its due date.

Two kinds of change can show up. Data drift means the inputs change: during a class project, many more students borrow stacks of books. If stacks are still usually late, the old rule still works. Concept drift means that link changes: if the library shortens its loan period, loans of 2 or 3 books start coming back late too. The inputs look the same, but the right answers changed, and only the results check can see it.

An alert is a rule that calls a person when a number crosses a line, such as “accuracy below 75%.” It is the trade-off from the forecasting page and the health screening lab: too strict, and normal wobble causes false alarms; too loose, and real problems run for weeks.

Predict first

In weeks 5 and 6, during a project fortnight, 26 of 44 and 25 of 42 loans had 4+ books, far above the 37.5% in training. Version 1 was right on 37 of 44 (84.1%) and 34 of 42 (81.0%). Should the team retrain?

Show the answer

Not yet. This is data drift: the inputs changed, but stacks of books were still usually late, so the rule still worked. Accuracy stayed close to the 83–88% of weeks 1–4. Retraining on those two weeks would learn the same rule anyway. The right move is to look closer, note it, and keep watching.

3 · Try it

Run the weekly dashboard

Ten made-up weeks. Each week the loans that fell due are scored. You set the alert line and decide what to do.

Lab

Keep, roll back or retrain?

Weeks 1–3 have already happened. A teammate released version 2 at the start of week 3. After each week, pick one action; it takes effect the next week. Try different alert lines and watch which alerts were real.

Simulated in your browser with made-up weekly numbers. “Retrain” picks the rule with the fewest mistakes on the last two weeks.

Weekly dashboard (made-up loans)
WeekVersionAccuracyAlertRightLate loans caught4+ book loans

“Shifted” marks weeks where the share of 4+ book loans is more than 15 points away from the training share of 37.5%.

Version record

    The lab is a weekly dashboard. Without it, use this table of all ten made-up weeks. Training share of 4+ book loans: 6 of 16 (37.5%).

    Ten made-up weeks, scored after each due date
    Week4+ book loansv1 right (4+ books → late)v2 right (bug: always on time)Retrained rule (2+ books → late)
    Week 115 of 40 (37.5%)35 of 40 (87.5%)26 of 40 (65%)24 of 40 (60%)
    Week 213 of 36 (36.1%)30 of 36 (83.3%)23 of 36 (63.9%)21 of 36 (58.3%)
    Week 316 of 42 (38.1%)37 of 42 (88.1%)27 of 42 (64.3%)27 of 42 (64.3%)
    Week 414 of 40 (35%)34 of 40 (85%)26 of 40 (65%)24 of 40 (60%)
    Week 526 of 44 (59.1%)37 of 44 (84.1%)21 of 44 (47.7%)29 of 44 (65.9%)
    Week 625 of 42 (59.5%)34 of 42 (81%)19 of 42 (45.2%)28 of 42 (66.7%)
    Week 714 of 38 (36.8%)34 of 38 (89.5%)24 of 38 (63.2%)25 of 38 (65.8%)
    Week 815 of 42 (35.7%)28 of 42 (66.7%)17 of 42 (40.5%)33 of 42 (78.6%)
    Week 914 of 40 (35%)25 of 40 (62.5%)15 of 40 (37.5%)33 of 40 (82.5%)
    Week 1016 of 44 (36.4%)28 of 44 (63.6%)16 of 44 (36.4%)38 of 44 (86.4%)

    Version 2 goes live in week 3 and scores like “always on time,” catching no late loans: a bad release, so roll back. Weeks 5 and 6 have far more 4+ book loans (data drift), but version 1 still scores above 80%: no action needed. From week 8 version 1 drops to the low 60s while the inputs look normal (concept drift). Rolling back cannot help, because version 1 is the one running. Retraining on weeks 8 and 9 picks “2 or more books → late,” which scores 38 of 44 in week 10. Best plan: roll back after week 3 and retrain after week 9, 322 of 408 loans right (78.9%). Keeping version 2 to the end: 230 of 408 (56.4%).

    Worked example

    Spotting concept drift, every number shown

    Weeks 7 to 10 of the dashboard, assuming version 1 is running.

    Worked example

    Concept drift, week by week

    1. BeforeWeek 7: version 1 is right on 34 of 38 loans = 89.5%. Inputs: 14 of 38 loans had 4+ books = 36.8%, close to the training share of 6 ÷ 16 = 37.5%.
    2. The dropWeek 8: 28 of 42 = 66.7%. Week 9: 25 of 40 = 62.5%. Inputs: 15 of 42 = 35.7% and 14 of 40 = 35.0%, still normal. Results fell, inputs did not move: concept drift.
    3. Find the missesIn week 8 version 1 caught 13 of 25 late loans. Of the 15 loans with 2–3 books, 10 came back late. In weeks 1–4 that group was late only 7 times out of 54 (13.0%).
    4. Roll back?No. Version 1 is already running. Going back to it cannot fix a change in the world.
    5. RetrainOn weeks 8 and 9 together (42 + 40 = 82 loans), try four rules. Always on time: 32 right. 4+ books → late: 53. 2+ books → late: 66. Always late: 50. The new rule wins: 66 ÷ 82 = 80.5%, against 53 ÷ 82 = 64.6% for the old one.
    6. Week 10Version 3 (2+ books → late) is right on 38 of 44 = 86.4%. Version 1 would have scored 28 of 44 = 63.6%.

    All loans and weeks are made up for this page. A real team would also test version 3 on held-out recent loans before releasing it.

    Responding

    Roll back or retrain?

    They fix different problems, so the first job after an alert is to find out what changed.

    Three kinds of trouble and what usually helps
    What changedLibrary exampleWhich check sees itUsual response
    A new release has a bugVersion 2 reads a field that arrives empty in the live systemResults drop right after the releaseRoll back to the last good version, then find the bug
    Data drift: the inputsProject fortnight: far more loans of 4+ booksThe input checkLook closer. Retrain only if results drop too
    Concept drift: the link from inputs to answersA shorter loan period: 2–3 book loans start coming back lateOnly the results checkRetrain on labeled loans from after the change, then test before release

    A rollback puts a saved earlier version back into use. It is fast and safe when the new release is the problem, and it only works because the old version was kept. It cannot fix a changed world: if the world moved away from version 1, going back to version 1 changes nothing.

    Retraining builds a new version from newer labeled data. It needs enough examples from after the change, and the same honest test as the first version before release. Retraining with buggy code just makes a new buggy version.

    Every new version also gets an updated model card (proposed by researchers in 2019; you wrote one on the build-a-model page): what changed, which data it learned from, its test results and its known limits.

    In real life. Spam filters must keep up with new kinds of scam messages, so they are checked and retrained again and again. A limit: results checks need labels, and some arrive slowly: a weather forecast is checked tomorrow, but a prediction about who finishes a course waits months. A connection: on Learning now, or using what it learned? you saw that a deployed model does not learn by itself. This page is what people do about that.

    Optional

    Going further

    For older students: two habits real teams add.

    Feedback loops

    The model changes its own data

    A student who gets a reminder may bring the book back on time. Then the reminder changes the label the model is later scored and retrained on. The 2015 “hidden technical debt” paper calls this a hidden feedback loop. One defense is to keep a small comparison group of loans whose reminders do not depend on the model.

    Careful releases

    Small first, then everyone

    Instead of switching every loan to a new version at once, teams often send a small share of cases to it first, or run it quietly next to the old one and compare. Google’s crash course also recommends checking data in training and in live use with the same rules, which would have caught version 2’s empty field.

    4 · Common mix-ups

    What people get wrong about drift

    Four ideas that trip people up, and what to think instead.

    • Mix-up: A deployed model wears out, like a machine part.

      Better: Its parameters stay exactly the same. The world around it moves. Nothing inside the model broke, so the answer to drift is to keep monitoring it.

    • Mix-up: Rolling back fixes drift.

      Better: Rollback fixes a bad release. If the world changed, the older version is out of date too.

    • Mix-up: If the inputs look normal, the model is fine.

      Better: Concept drift changes the right answers without changing the inputs. Only checks against real outcomes catch it.

    • Mix-up: Retraining always helps.

      Better: Retraining on data from before a change learns the old world again, and retraining with buggy code makes a new buggy version.

    5 · Check yourself

    Three quick questions

    Answer in your head or on paper before you open each one.

    1. A plant-disease photo checker gets a new phone camera that makes every photo brighter. Accuracy on checked photos stays the same. Data drift or concept drift, and should the team retrain?

      Check your answer

      Data drift: the inputs changed, the link between a leaf’s look and its disease did not. With results unchanged, there is no need to retrain yet. Keep watching, and check the brighter photos as their own group in case one kind of disease gets harder to spot.

    2. Version 4 goes live on Monday. That week accuracy falls from about 85% to 60%, while the inputs look normal. What is your first move?

      Check your answer

      Roll back to version 3, the last good version. A sudden drop right after a release points to the release, and rollback is the fast, safe fix. Then compare version 4 with its record to find the cause, fix it, and test again before the next release.

    3. Why can a team not roll back to version 1 if they never saved its data, code, settings and trained model?

      Check your answer

      There is nothing to go back to. Without the trained model they would have to rebuild it, and without the data, code and settings (including the random seed) they cannot be sure the rebuilt model is the same one that worked. Recording versions is what makes rollback and reproducibility possible.

    For younger learners

    A grades 6–8 version

    Same story on paper, no percentages needed.

    30 minutes · groups of three

    Paper dashboard

    1. Print the ten-week table from the lab’s text version and fold back the last two columns.
    2. Each group draws an alert line on a bar chart, then reveals version 1’s score one week at a time and adds a bar.
    3. When a bar drops below the line, the group decides: a normal wobble, a change in the kind of loans, or a change in the right answers? Then they unfold the last column (2+ books → late) to see whether a new rule would help.

    Exit question

    What changed?

    Good answers say whether the kind of loans changed (more stacks of books) or the right answers changed (shorter loan period), and why only the second needs a new rule.

    Honest limits

    What this page leaves out

    What this page leaves out

    Real MLOps tools and pipelines, automatic retraining schedules, data checks with written rules (schemas), checking groups of users separately for fairness, cost, the privacy of logged data, and security. Ten made-up weeks are much tidier than a real dashboard: real numbers wobble more, labels can be wrong, and the cause of a change is rarely announced.