Go deeper · Advanced track
What happens after a model is trained?
It goes to work, and the work does not stop. A team saves exactly which data, code and settings made each version, puts one version into use, and keeps checking it on new cases. When a new release breaks, they roll back to an older version. When the world changes, they retrain. Doing all of this carefully is called MLOps.
This page is part of the optional advanced track. You do not need it for the main route.
What you will be able to do
Name the four parts of a model version, tell data drift from concept drift on a weekly dashboard, choose an alert threshold, and decide whether a problem needs a rollback or retraining.
Before you start
Read first: Can you build and test a model yourself? (this page continues its library model), Good at what, compared with what? (accuracy and baselines) and How does AI predict tomorrow and spot the unusual? (drift and false alarms).
1 · You have already seen this
The filter that has to keep up
Most people have had a scam message slip past a spam filter that used to catch messages like it.
A spam filter learned from messages that people marked as scam or not scam. Scammers keep changing what they write, so a filter that did well last year can let new kinds of scam through this year. The people who run it have to keep checking how it does and train it again when it falls behind.
Looking after a model once it is in real use is called MLOps. This page follows a school library’s model that predicts which loans will come back late, starting on the day it is switched on. It shows how that work goes, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From switching it on to the next version
Looking after a model in four steps. The next section goes deeper into step 2, and the two after it into steps 1 and 3. After the lab, a worked example and a section on responding go deeper into step 4.
How a model is looked after
Four steps
- 1 · Save the versionRecord the four parts that made the model: the exact data, the code, the trained model and its settings. Give that combination a name, such as v1.
- 2 · Put it to workSwitch the model on so it makes real predictions. The library model sends an early reminder when a student borrows 4 or more books at once.
- 3 · Check it every weekLook at this week’s inputs, and score the predictions once the right answers arrive. For a loan, that is only after its due date.
- 4 · Act on an alertWhen a number crosses the line you set, find out what changed before doing anything.
Going deeper · step 2
The model went live. Now what?
We pick up the library model from the build-a-model page on the day it starts sending real reminders.
On the build-a-model page, a school library trained a one-question model: if a student borrows 4 or more books at once, predict the loan will come back late and send a friendly reminder early. It beat the baseline on 8 test loans. Now the library switches it on. Putting a trained model into real use is called deployment.
From then on the model predicts loans nobody has seen yet. Its parameters do not change by themselves, but the students, the timetable and the library’s rules do. Each week some loans fall due, and the library learns which predictions were right.
Careful teams record every version, watch it after release and plan for failure. This practice is called MLOps (machine learning operations). A 2015 paper by Google engineers argued that the model is only a small part of a real machine learning system: most of the work, and the risk, sits in the data, code and checks around it.
Going deeper · step 1
Write down everything that made it
If you cannot rebuild a model, you cannot fix it, compare it or go back to it.
| Part of a version | Library example | Why record it |
|---|---|---|
| Data | The exact 16 training loans and 8 test loans, saved as a fixed copy | New rows make a different model. You need to know which rows made this one. |
| Code | The script that turns checkout records into features and trains the model | One changed line can change every prediction. |
| Model | The trained result: “4 or more books → late.” For a neural network, a file of parameters | This is what you deploy, and what you roll back to. |
| Settings | Hyperparameters, the random seed, the versions of the software libraries | Same data and code with different settings can give a different model. |
Rebuilding the same model from those records is called reproducibility. Many training methods use random numbers, for example to shuffle rows or to pick a neural network’s starting weights. A random seed is the number that starts that random sequence. Fix it, and the “random” choices come out the same on every run. Even then, different software versions or hardware can cause small differences, so teams record those too.
Each saved combination gets a name, such as v1 and v2. That list of versions is what makes going back possible.
Predict first
Two teammates train “the same” model on the same day and get different test scores. Name three things that could differ.
Show the answer
Any of these: a different copy of the data (one took it before new loans were added), a different random seed (so rows were shuffled or split differently), a different version of the code or of the machine learning library, or different settings such as how deep a tree may grow. Recording all four parts of a version turns a mystery into a quick check.
Going deeper · step 3
Keep checking on new cases
A test score describes the past. Monitoring tells you about this week.
Monitoring means checking a deployed model regularly. The library’s dashboard has two checks. The input check: what share of this week’s loans had 4 or more books? In training it was 6 of 16, or 37.5%. The results check: of the loans now due, how many did the model get right? That needs labels, which arrive late: you only know a loan was late after its due date.
Two kinds of change can show up. Data drift means the inputs change: during a class project, many more students borrow stacks of books. If stacks are still usually late, the old rule still works. Concept drift means that link changes: if the library shortens its loan period, loans of 2 or 3 books start coming back late too. The inputs look the same, but the right answers changed, and only the results check can see it.
An alert is a rule that calls a person when a number crosses a line, such as “accuracy below 75%.” It is the trade-off from the forecasting page and the health screening lab: too strict, and normal wobble causes false alarms; too loose, and real problems run for weeks.
Predict first
In weeks 5 and 6, during a project fortnight, 26 of 44 and 25 of 42 loans had 4+ books, far above the 37.5% in training. Version 1 was right on 37 of 44 (84.1%) and 34 of 42 (81.0%). Should the team retrain?
Show the answer
Not yet. This is data drift: the inputs changed, but stacks of books were still usually late, so the rule still worked. Accuracy stayed close to the 83–88% of weeks 1–4. Retraining on those two weeks would learn the same rule anyway. The right move is to look closer, note it, and keep watching.
3 · Try it
Run the weekly dashboard
Ten made-up weeks. Each week the loans that fell due are scored. You set the alert line and decide what to do.
Lab
Keep, roll back or retrain?
Weeks 1–3 have already happened. A teammate released version 2 at the start of week 3. After each week, pick one action; it takes effect the next week. Try different alert lines and watch which alerts were real.
Simulated in your browser with made-up weekly numbers. “Retrain” picks the rule with the fewest mistakes on the last two weeks.
| Week | Version | Accuracy | Alert | Right | Late loans caught | 4+ book loans |
|---|
“Shifted” marks weeks where the share of 4+ book loans is more than 15 points away from the training share of 37.5%.
Version record
The lab is a weekly dashboard. Without it, use this table of all ten made-up weeks. Training share of 4+ book loans: 6 of 16 (37.5%).
| Week | 4+ book loans | v1 right (4+ books → late) | v2 right (bug: always on time) | Retrained rule (2+ books → late) |
|---|---|---|---|---|
| Week 1 | 15 of 40 (37.5%) | 35 of 40 (87.5%) | 26 of 40 (65%) | 24 of 40 (60%) |
| Week 2 | 13 of 36 (36.1%) | 30 of 36 (83.3%) | 23 of 36 (63.9%) | 21 of 36 (58.3%) |
| Week 3 | 16 of 42 (38.1%) | 37 of 42 (88.1%) | 27 of 42 (64.3%) | 27 of 42 (64.3%) |
| Week 4 | 14 of 40 (35%) | 34 of 40 (85%) | 26 of 40 (65%) | 24 of 40 (60%) |
| Week 5 | 26 of 44 (59.1%) | 37 of 44 (84.1%) | 21 of 44 (47.7%) | 29 of 44 (65.9%) |
| Week 6 | 25 of 42 (59.5%) | 34 of 42 (81%) | 19 of 42 (45.2%) | 28 of 42 (66.7%) |
| Week 7 | 14 of 38 (36.8%) | 34 of 38 (89.5%) | 24 of 38 (63.2%) | 25 of 38 (65.8%) |
| Week 8 | 15 of 42 (35.7%) | 28 of 42 (66.7%) | 17 of 42 (40.5%) | 33 of 42 (78.6%) |
| Week 9 | 14 of 40 (35%) | 25 of 40 (62.5%) | 15 of 40 (37.5%) | 33 of 40 (82.5%) |
| Week 10 | 16 of 44 (36.4%) | 28 of 44 (63.6%) | 16 of 44 (36.4%) | 38 of 44 (86.4%) |
Version 2 goes live in week 3 and scores like “always on time,” catching no late loans: a bad release, so roll back. Weeks 5 and 6 have far more 4+ book loans (data drift), but version 1 still scores above 80%: no action needed. From week 8 version 1 drops to the low 60s while the inputs look normal (concept drift). Rolling back cannot help, because version 1 is the one running. Retraining on weeks 8 and 9 picks “2 or more books → late,” which scores 38 of 44 in week 10. Best plan: roll back after week 3 and retrain after week 9, 322 of 408 loans right (78.9%). Keeping version 2 to the end: 230 of 408 (56.4%).
Worked example
Spotting concept drift, every number shown
Weeks 7 to 10 of the dashboard, assuming version 1 is running.
Worked example
Concept drift, week by week
- BeforeWeek 7: version 1 is right on 34 of 38 loans = 89.5%. Inputs: 14 of 38 loans had 4+ books = 36.8%, close to the training share of 6 ÷ 16 = 37.5%.
- The dropWeek 8: 28 of 42 = 66.7%. Week 9: 25 of 40 = 62.5%. Inputs: 15 of 42 = 35.7% and 14 of 40 = 35.0%, still normal. Results fell, inputs did not move: concept drift.
- Find the missesIn week 8 version 1 caught 13 of 25 late loans. Of the 15 loans with 2–3 books, 10 came back late. In weeks 1–4 that group was late only 7 times out of 54 (13.0%).
- Roll back?No. Version 1 is already running. Going back to it cannot fix a change in the world.
- RetrainOn weeks 8 and 9 together (42 + 40 = 82 loans), try four rules. Always on time: 32 right. 4+ books → late: 53. 2+ books → late: 66. Always late: 50. The new rule wins: 66 ÷ 82 = 80.5%, against 53 ÷ 82 = 64.6% for the old one.
- Week 10Version 3 (2+ books → late) is right on 38 of 44 = 86.4%. Version 1 would have scored 28 of 44 = 63.6%.
All loans and weeks are made up for this page. A real team would also test version 3 on held-out recent loans before releasing it.
Responding
Roll back or retrain?
They fix different problems, so the first job after an alert is to find out what changed.
| What changed | Library example | Which check sees it | Usual response |
|---|---|---|---|
| A new release has a bug | Version 2 reads a field that arrives empty in the live system | Results drop right after the release | Roll back to the last good version, then find the bug |
| Data drift: the inputs | Project fortnight: far more loans of 4+ books | The input check | Look closer. Retrain only if results drop too |
| Concept drift: the link from inputs to answers | A shorter loan period: 2–3 book loans start coming back late | Only the results check | Retrain on labeled loans from after the change, then test before release |
A rollback puts a saved earlier version back into use. It is fast and safe when the new release is the problem, and it only works because the old version was kept. It cannot fix a changed world: if the world moved away from version 1, going back to version 1 changes nothing.
Retraining builds a new version from newer labeled data. It needs enough examples from after the change, and the same honest test as the first version before release. Retraining with buggy code just makes a new buggy version.
Every new version also gets an updated model card (proposed by researchers in 2019; you wrote one on the build-a-model page): what changed, which data it learned from, its test results and its known limits.
In real life. Spam filters must keep up with new kinds of scam messages, so they are checked and retrained again and again. A limit: results checks need labels, and some arrive slowly: a weather forecast is checked tomorrow, but a prediction about who finishes a course waits months. A connection: on Learning now, or using what it learned? you saw that a deployed model does not learn by itself. This page is what people do about that.
Optional
Going further
For older students: two habits real teams add.
Feedback loops
The model changes its own data
A student who gets a reminder may bring the book back on time. Then the reminder changes the label the model is later scored and retrained on. The 2015 “hidden technical debt” paper calls this a hidden feedback loop. One defense is to keep a small comparison group of loans whose reminders do not depend on the model.
Careful releases
Small first, then everyone
Instead of switching every loan to a new version at once, teams often send a small share of cases to it first, or run it quietly next to the old one and compare. Google’s crash course also recommends checking data in training and in live use with the same rules, which would have caught version 2’s empty field.
4 · Common mix-ups
What people get wrong about drift
Four ideas that trip people up, and what to think instead.
Mix-up: A deployed model wears out, like a machine part.
Better: Its parameters stay exactly the same. The world around it moves. Nothing inside the model broke, so the answer to drift is to keep monitoring it.
Mix-up: Rolling back fixes drift.
Better: Rollback fixes a bad release. If the world changed, the older version is out of date too.
Mix-up: If the inputs look normal, the model is fine.
Better: Concept drift changes the right answers without changing the inputs. Only checks against real outcomes catch it.
Mix-up: Retraining always helps.
Better: Retraining on data from before a change learns the old world again, and retraining with buggy code makes a new buggy version.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
A plant-disease photo checker gets a new phone camera that makes every photo brighter. Accuracy on checked photos stays the same. Data drift or concept drift, and should the team retrain?
Check your answer
Data drift: the inputs changed, the link between a leaf’s look and its disease did not. With results unchanged, there is no need to retrain yet. Keep watching, and check the brighter photos as their own group in case one kind of disease gets harder to spot.
Version 4 goes live on Monday. That week accuracy falls from about 85% to 60%, while the inputs look normal. What is your first move?
Check your answer
Roll back to version 3, the last good version. A sudden drop right after a release points to the release, and rollback is the fast, safe fix. Then compare version 4 with its record to find the cause, fix it, and test again before the next release.
Why can a team not roll back to version 1 if they never saved its data, code, settings and trained model?
Check your answer
There is nothing to go back to. Without the trained model they would have to rebuild it, and without the data, code and settings (including the random seed) they cannot be sure the rebuilt model is the same one that worked. Recording versions is what makes rollback and reproducibility possible.
For younger learners
A grades 6–8 version
Same story on paper, no percentages needed.
30 minutes · groups of three
Paper dashboard
- Print the ten-week table from the lab’s text version and fold back the last two columns.
- Each group draws an alert line on a bar chart, then reveals version 1’s score one week at a time and adds a bar.
- When a bar drops below the line, the group decides: a normal wobble, a change in the kind of loans, or a change in the right answers? Then they unfold the last column (2+ books → late) to see whether a new rule would help.
Exit question
What changed?
Good answers say whether the kind of loans changed (more stacks of books) or the right answers changed (shorter loan period), and why only the second needs a new rule.
Honest limits
What this page leaves out
What this page leaves out
Real MLOps tools and pipelines, automatic retraining schedules, data checks with written rules (schemas), checking groups of users separately for fairness, cost, the privacy of logged data, and security. Ten made-up weeks are much tidier than a real dashboard: real numbers wobble more, labels can be wrong, and the cause of a change is rarely announced.
Where to go next
Keep connecting
Revisit
How does AI predict tomorrow?
Time order, naive baselines and alarms that are tuned between misses and false alarms.
Related
Who checks that an AI system is fair and safe?
Purpose, affected people, evidence, limits and human oversight.
Revisit
Can you build and test a model yourself?
The library model before deployment: baseline, split, test and model card.
LLMOps
AI in production
CI/CD for prompts and models, version pinning, tracing, evals and access control for language-model apps.
Sources
Where this comes from
- Google for Developers, Machine Learning Crash Course, “Production ML systems”: static and dynamic training, deployment testing and monitoring.
- Google for Developers, “Deployment testing”: reproducible training with fixed seeds and version control, and checking new versions against older ones.
- Google for Developers, “Monitoring pipelines”: data checks and training-serving skew.
- D. Sculley and others (2015), “Hidden Technical Debt in Machine Learning Systems,” NeurIPS 2015: feedback loops, data dependencies and changes in the external world.
- Margaret Mitchell and others (2019), “Model Cards for Model Reporting.”
AI words on this page
MLOps, Reproducibility, Model, Parameters, Hyperparameter, Inference, Distribution shift (data drift), Model drift (concept drift), Baseline, Test set, False positive, Evaluation, Human oversight
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.