Go deeper · Use, question, build
Can you build and test a model yourself?
Yes. A useful first project needs no neural network and no big computer: a clear question, a small labeled table, a baseline to beat, a fair split, one simple model, one honest test and a short write-up. The write-up and the honest test matter as much as the model.
What you will be able to do
Run a complete small machine learning project in seven steps, explain why the model must beat a baseline on data it never saw, spot leakage and test-set peeking, and write a mini model card with honest limits.
Before you start
Features, labels and splits from What did we teach it by accident?, baselines from Good at what, compared with what?, and decision trees from Do you always need a neural network?
1 · You have already seen this
Guessing who will turn up late
Most people can guess which friend will arrive late before anyone has turned up.
That guess comes from memory. You think back over past meetups, notice what the late arrivals had in common, and expect the same next time. The next meetup then shows whether your guess holds up.
Building a model writes that habit down step by step. You pick a question whose answer was recorded, decide what the model may look at, keep some examples aside, train it, test it once and write down what you found. This page follows one small project, a school library guessing which loans will come back late. It shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
Seven steps, one small project
Professional projects use bigger data and fancier models, but the order of decisions is the same.
Imagine a school library that wants to know which loans are likely to come back late, so it can send a friendly reminder a few days early. That is a prediction question with a yes or no answer: a classification task.
Before any model, the library needs records where the answer is known. Each past loan is one row. The columns the model may look at are features, such as how many books were borrowed at once. The answer it should learn to predict, late or on time, is the label.
Then come the steps most beginners skip. A baseline says what score you get for free. A split locks some rows away as a test set, so the final score shows how the model does on loans it never saw. And a short model card tells the next person what the model is for and where it fails.
| Step | What you decide | The trap to avoid |
|---|---|---|
| 1 · Question | A yes or no (or number) question, and the label that answers it in your records | A question nobody recorded the answer to |
| 2 · Baseline | The score of the simplest rule, such as “always the most common answer” | Celebrating 62.5% when “always on time” also scores 62.5% |
| 3 · Data and features | Which columns the model may look at, known at the moment you predict | Leakage: a column that secretly contains the answer |
| 4 · Split | Which rows train the model and which are locked away for the final test | Peeking at the test rows to make choices |
| 5 · Train | The model family, kept as simple as possible at first | A complex model before a simple one has been tried |
| 6 · Test | One honest comparison with the baseline on the locked rows | Testing again and again until a number looks good |
| 7 · Document | A short model card: purpose, data, results, limits | Leaving out who could be harmed by mistakes |
Going deeper · steps 3 and 4
Twenty-four made-up loans
Every loan below is invented for this page. The split was made once, before anything else: loans #1–16 train the model and loans #17–24 are the locked test set.
Predict first
Before you train anything: the baseline always says “on time.” Of the 8 test loans, 3 came back late. What accuracy does the baseline get on the test loans, and how many late loans does it catch?
Show the answer
It is right on the 5 on-time loans: 5 ÷ 8 = 62.5%. It catches none of the 3 late loans, which is the whole point of the project. To beat this baseline on accuracy, a model has to score above 62.5%. Usefulness is a separate question: count the late loans it catches and the unnecessary reminders it sends, and weigh what each kind of mistake costs.
Show all 24 loans as a table
| Loan | Pages | Books borrowed at once | Due in exam week | Reminder emails sent | Came back | Set |
|---|---|---|---|---|---|---|
| #1 | 180 | 1 | no | 0 | on time | training |
| #2 | 240 | 2 | no | 0 | on time | training |
| #3 | 520 | 1 | yes | 0 | on time | training |
| #4 | 150 | 3 | no | 0 | on time | training |
| #5 | 310 | 2 | yes | 0 | on time | training |
| #6 | 600 | 2 | no | 0 | on time | training |
| #7 | 210 | 1 | no | 0 | on time | training |
| #8 | 280 | 4 | no | 0 | on time | training |
| #9 | 360 | 3 | no | 0 | on time | training |
| #10 | 130 | 2 | no | 0 | on time | training |
| #11 | 450 | 5 | yes | 2 | late | training |
| #12 | 380 | 4 | yes | 1 | late | training |
| #13 | 560 | 6 | no | 1 | late | training |
| #14 | 200 | 5 | no | 2 | late | training |
| #15 | 470 | 2 | yes | 1 | late | training |
| #16 | 300 | 4 | no | 1 | late | training |
| #17 | 260 | 2 | no | 0 | on time | test |
| #18 | 540 | 1 | no | 0 | on time | test |
| #19 | 190 | 4 | yes | 1 | late | test |
| #20 | 410 | 5 | no | 2 | late | test |
| #21 | 230 | 3 | yes | 0 | on time | test |
| #22 | 330 | 1 | yes | 1 | late | test |
| #23 | 170 | 1 | no | 0 | on time | test |
| #24 | 390 | 2 | no | 0 | on time | test |
3 · Try it
Build it step by step
Work through the steps in order. Change your mind at any point and see what changes downstream.
Lab
A guided model project
Pick a question, set the baseline, choose features, keep the split, train a one-question model, test it once and write the model card.
Simulated in your browser: a one-question model is trained on 16 made-up loans when you press Train.
1 · Ask a question that has a label
The library’s records list each loan’s due date and return date, and nothing else about the readers. Which question can you train a model for?
2 · Set the baseline
The simplest possible model always predicts the most common answer among the training loans.
3 · Choose features
Which columns may the model look at? Pick one or more.
4 · Keep the split
Loans #1–16 are for training. Loans #17–24 (5 on time, 3 late) were chosen once, at random, and are locked until step 6.
5 · Train
The model is a decision stump: it tries every cut-off on every chosen feature and keeps the single yes or no question with the fewest mistakes on the training loans.
6 · Test once
Unlock the 8 test loans and compare the model with the baseline.
7 · Document and reflect
Write the purpose and at least one limit. The rest of the card fills in from your choices.
Your model card
The lab walks through all seven steps with the 24 made-up loans below. Without it, do the same on paper: count the training labels for the baseline, try each feature’s best yes or no question on loans #1–16, then check loans #17–24 once. This is what you should find:
| Feature the model may use | Question it learns | Training loans right | Test loans right | Late test loans caught |
|---|---|---|---|---|
| None: the baseline | Always say on time. | 10 of 16 | 5 of 8 | 0 of 3 |
| Pages | Does the book have at least 370 pages? Yes: late. No: on time. | 12 of 16 | 4 of 8 | 1 of 3 |
| Books borrowed at once | Were 4 or more books borrowed at once? Yes: late. No: on time. | 14 of 16 | 7 of 8 | 2 of 3 |
| Due in exam week | Is it due in exam week? Yes: late. No: on time. | 11 of 16 | 6 of 8 | 2 of 3 |
| Reminder emails sent | Was at least 1 reminder email sent? Yes: late. No: on time. | 16 of 16 | 8 of 8 | 3 of 3 |
Reminder emails are only sent after a book is already late, so a model that uses them is reading the answer. That is data leakage, and the perfect score is not real. With several features allowed, the model picks the one with the fewest training mistakes: books borrowed at once, unless reminder emails are allowed.
Why the test loans stay locked. If you use them to choose features, cut-offs or models, they stop being new to the model, and the final score becomes too optimistic. When you need to compare options, split a third group, a validation set, off the training rows, or use cross-validation. Touch the test set once, at the end.
Worked example
One feature, every number shown
Here are steps 2, 5 and 6 by hand, using only “books borrowed at once.”
Worked example
Books borrowed at once, from baseline to test
- BaselineThe 16 training loans: 10 on time, 6 late. The baseline always says on time. Training: 10 ÷ 16 = 62.5%. Test: 5 ÷ 8 = 62.5%, and it catches 0 of the 3 late test loans.
- Try cut-offsBetween 2 and 3 books: 4 training mistakes. Between 3 and 4: 2 mistakes. Between 4 and 5: 3. Between 5 and 6: 5.
- Pick the bestRule: 4 or more books → late, otherwise on time. Training: 14 ÷ 16 = 87.5%. The 2 misses: loan #8 (4 books, on time) and loan #15 (2 books, late).
- Test onceLoans #19 (4 books) and #20 (5 books) are flagged late, and both were. Loan #22 (1 book) is called on time but came back late. 7 ÷ 8 = 87.5%, catching 2 of 3 late loans with no false alarms.
- CompareModel 7 of 8, baseline 5 of 8: the model is right on 2 more test loans. But each test loan is worth 12.5 percentage points, so this is a promising sign, not proof.
All 24 loans are invented for this page. The lab above computes the same numbers in your browser.
Predict first
Pages looked useful on the training loans: “at least 370 pages means late” gets 12 of 16 right, better than the baseline’s 10. How do you think it does on the 8 test loans?
Show the answer
Worse than the baseline: 4 of 8 (50%), against 5 of 8. It calls two long books late that were not (#18 and #24) and misses two shorter books that were late (#19 and #22). With only 16 training loans, a cut-off can fit chance patterns. This is why the test set, not the training score, gets the last word.
Optional
The same project in Python
For older students who want to run it for real. The data is the same made-up table, typed into the code.
The script uses scikit-learn, a free Python library for classical machine learning. DummyClassifier is the baseline, a DecisionTreeClassifier with max_depth=1 is the one-question model, and the split is fixed by row number so you get the same answer every time. Changing it and re-running until the test number goes up is exactly the habit step 4 warns about.
Show the Python version (scikit-learn, optional)
# Will a library book come back late? A tiny made-up dataset (24 loans).
# Run it with Python 3 after installing scikit-learn: pip install scikit-learn
from sklearn.dummy import DummyClassifier
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.metrics import accuracy_score, recall_score
FEATURES = ["pages", "books_at_once", "exam_week"]
# Each row: pages, books borrowed at once, due in exam week (1 = yes), late (1 = yes).
# "Reminder emails sent" is left out on purpose: it is only known after a book is late.
LOANS = [
[180, 1, 0, 0], # loan 1
[240, 2, 0, 0], # loan 2
[520, 1, 1, 0], # loan 3
[150, 3, 0, 0], # loan 4
[310, 2, 1, 0], # loan 5
[600, 2, 0, 0], # loan 6
[210, 1, 0, 0], # loan 7
[280, 4, 0, 0], # loan 8
[360, 3, 0, 0], # loan 9
[130, 2, 0, 0], # loan 10
[450, 5, 1, 1], # loan 11
[380, 4, 1, 1], # loan 12
[560, 6, 0, 1], # loan 13
[200, 5, 0, 1], # loan 14
[470, 2, 1, 1], # loan 15
[300, 4, 0, 1], # loan 16
[260, 2, 0, 0], # loan 17
[540, 1, 0, 0], # loan 18
[190, 4, 1, 1], # loan 19
[410, 5, 0, 1], # loan 20
[230, 3, 1, 0], # loan 21
[330, 1, 1, 1], # loan 22
[170, 1, 0, 0], # loan 23
[390, 2, 0, 0], # loan 24
]
X = [row[:3] for row in LOANS]
y = [row[3] for row in LOANS]
# Step 4: a fixed split. Loans 17-24 are the locked test set.
X_train, y_train = X[:16], y[:16]
X_test, y_test = X[16:], y[16:]
# Step 2: the baseline always predicts the most common training label.
baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
# Step 5: a decision stump, a tree that asks only one question.
stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X_train, y_train)
print(export_text(stump, feature_names=FEATURES))
# Step 6: test once, on loans the models never saw.
for name, model in [("baseline", baseline), ("stump", stump)]:
pred = model.predict(X_test)
acc = accuracy_score(y_test, pred)
caught = recall_score(y_test, pred, zero_division=0)
print(f"{name}: test accuracy {acc:.3f}, late loans caught {caught:.3f}")
What it prints: the stump’s rule is “books_at_once <= 3.50” (3 or fewer books: on time; 4 or more: late). Baseline: test accuracy 0.625, late loans caught 0.000. Stump: test accuracy 0.875, late loans caught 0.667. scikit-learn picks its question with a slightly different score (Gini impurity) than the lab’s “fewest mistakes,” but on this data both choose the same question. Checked with scikit-learn 1.9.
Document and reflect
The model card is part of the model
A model nobody can check is a model nobody should trust.
What goes on the card
Four short headings
- Purpose: what the model is for, who uses it, and what it should not be used for.
- Data: where the rows came from, how many, which version, and how they were split.
- Results: the test score next to the baseline, including the errors that matter, such as late loans missed.
- Limits: who could be affected by mistakes, what the data does not cover, and what to do when it fails.
Researchers proposed “model cards” in 2019 as a standard short report for trained models. Real cards are longer, but the habit starts here.
Reflect
Should this model exist?
Even a good score leaves questions. A reminder email is harmless, but the same model used to refuse loans to students would punish people for borrowing a lot, which is what libraries want. The purpose decides what mistakes cost.
A real project would also need far more loans, a check that it works equally well for different groups of readers, and a plan for when borrowing habits change, such as a new exam timetable. Read more in Who checks that an AI system is fair and safe?
4 · Common mix-ups
Where beginner projects go wrong
Four common beliefs, each with what to think instead.
Mix-up: A high training score means the model works.
Better: Only the locked test set shows how it does on new rows. Pages scored 12 of 16 in training and 4 of 8 in testing.
Mix-up: If accuracy is above 50%, the model learned something.
Better: Compare with the baseline, not with 50%. “Always on time” already scores 62.5% here while catching no late loans.
Mix-up: More features always help.
Better: A feature that is only known after the fact, like reminder emails, makes the score perfect and the model useless. Every feature must be available at the moment you predict.
Mix-up: Documentation is paperwork for after the real work.
Better: The model card records what was tested and what was not. Without it, the next person cannot tell a careful model from a lucky one.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
Your model scores 70% on the test set. Why can’t you tell yet whether it is good?
Check your answer
You need the baseline. If “always predict the most common answer” scores 75%, your model is less accurate than that baseline. But accuracy is not the same as usefulness: you also need to know which mistakes each one makes and what they cost: here the baseline scores 62.5% while missing every late loan.
A classmate tests the model, sees 6 of 8, swaps the feature, tests again, gets 7 of 8 and reports 87.5%. What went wrong?
Check your answer
They used the test set to choose the feature, so it became a validation set. The 87.5% is now optimistic. They should compare features on a validation set split off the training loans, then test the final choice once on untouched rows.
Why is “reminder emails sent” not allowed as a feature, even though it predicts lateness perfectly?
Check your answer
Reminders are sent only after a book is overdue, so the column is a consequence of the label. At checkout, when the prediction is needed, it does not exist yet. Using it is data leakage.
For younger learners
A grades 6–8 version
The same project on paper, no percentages needed.
40 minutes · pairs
Be the model
- Print the table of 24 loans and cut off loans #17–24 into a sealed envelope.
- Count late and on-time loans among #1–16. What would “always guess the most common answer” get right?
- Each pair writes one yes or no question about one column and counts how many of the 16 loans it gets right.
- Open the envelope once and score the question on the 8 hidden loans.
Exit question
Why was the envelope sealed?
Good answers say the hidden loans are a fair test, because nobody could fit their question to them. Ask too: why is “reminder emails sent” a cheat?
Warm-up idea: Train the Machine shows a model learning a misleading pattern.
Honest limits
What this page leaves out
What this page leaves out
Cleaning messy data, handling missing values, features made from text or dates, choosing among many models with cross-validation, deploying a model and monitoring it after launch. Twenty-four rows is far too few for a real decision: here, one test loan moves accuracy by 12.5 percentage points. Next on the builder path: sturdier splits, a repeatable pipeline, then deployment, monitoring and a plan to roll back.
Where to go next
Keep building
Revisit · 06
Good at what, compared with what?
Confusion matrices, precision and recall: which late loans did the model miss?
Revisit · 05
Learning now, or using what it learned?
Training versus using a model, and why a deployed model does not keep learning by itself.
Next in this branch
How does AI change jobs?
The people who label data, test models and keep them running.
Sources
Where this comes from
- Google for Developers, Machine Learning Crash Course, “Dividing the original dataset”: training, validation and test sets.
- scikit-learn User Guide, “Common pitfalls and recommended practices”: data leakage and inconsistent preprocessing.
- scikit-learn,
DummyClassifier: baselines such as “most frequent” for comparing real models. - scikit-learn User Guide, “Decision Trees”: how trees choose split questions.
- Margaret Mitchell and others (2019), “Model Cards for Model Reporting.”
- Timnit Gebru and others, “Datasheets for Datasets” (first posted 2018).
AI words on this page
Model, Feature, Label, Classification, Baseline, Training data, Test set, Validation set, Cross-validation, Data leakage, Decision tree, Overfitting, Precision and recall, Dataset card, Evaluation
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.