Go deeper · Use, question, build

Can you build and test a model yourself?

Yes. A useful first project needs no neural network and no big computer: a clear question, a small labeled table, a baseline to beat, a fair split, one simple model, one honest test and a short write-up. The write-up and the honest test matter as much as the model.

Grades 9+35 minutesCounting and percentagesOptional: PythonPaper version for grades 6–8

What you will be able to do

Run a complete small machine learning project in seven steps, explain why the model must beat a baseline on data it never saw, spot leakage and test-set peeking, and write a mini model card with honest limits.

Before you start

Features, labels and splits from What did we teach it by accident?, baselines from Good at what, compared with what?, and decision trees from Do you always need a neural network?

1 · You have already seen this

Guessing who will turn up late

Most people can guess which friend will arrive late before anyone has turned up.

That guess comes from memory. You think back over past meetups, notice what the late arrivals had in common, and expect the same next time. The next meetup then shows whether your guess holds up.

Building a model writes that habit down step by step. You pick a question whose answer was recorded, decide what the model may look at, keep some examples aside, train it, test it once and write down what you found. This page follows one small project, a school library guessing which loans will come back late. It shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

Seven steps, one small project

Professional projects use bigger data and fancier models, but the order of decisions is the same.

Imagine a school library that wants to know which loans are likely to come back late, so it can send a friendly reminder a few days early. That is a prediction question with a yes or no answer: a classification task.

Before any model, the library needs records where the answer is known. Each past loan is one row. The columns the model may look at are features, such as how many books were borrowed at once. The answer it should learn to predict, late or on time, is the label.

Then come the steps most beginners skip. A baseline says what score you get for free. A split locks some rows away as a test set, so the final score shows how the model does on loans it never saw. And a short model card tells the next person what the model is for and where it fails.

Seven steps of a small project
StepWhat you decideThe trap to avoid
1 · QuestionA yes or no (or number) question, and the label that answers it in your recordsA question nobody recorded the answer to
2 · BaselineThe score of the simplest rule, such as “always the most common answer”Celebrating 62.5% when “always on time” also scores 62.5%
3 · Data and featuresWhich columns the model may look at, known at the moment you predictLeakage: a column that secretly contains the answer
4 · SplitWhich rows train the model and which are locked away for the final testPeeking at the test rows to make choices
5 · TrainThe model family, kept as simple as possible at firstA complex model before a simple one has been tried
6 · TestOne honest comparison with the baseline on the locked rowsTesting again and again until a number looks good
7 · DocumentA short model card: purpose, data, results, limitsLeaving out who could be harmed by mistakes

Going deeper · steps 3 and 4

Twenty-four made-up loans

Every loan below is invented for this page. The split was made once, before anything else: loans #1–16 train the model and loans #17–24 are the locked test set.

Predict first

Before you train anything: the baseline always says “on time.” Of the 8 test loans, 3 came back late. What accuracy does the baseline get on the test loans, and how many late loans does it catch?

Show the answer

It is right on the 5 on-time loans: 5 ÷ 8 = 62.5%. It catches none of the 3 late loans, which is the whole point of the project. To beat this baseline on accuracy, a model has to score above 62.5%. Usefulness is a separate question: count the late loans it catches and the unnecessary reminders it sends, and weigh what each kind of mistake costs.

Show all 24 loans as a table
24 made-up library loans
LoanPagesBooks borrowed at onceDue in exam weekReminder emails sentCame backSet
#11801no0on timetraining
#22402no0on timetraining
#35201yes0on timetraining
#41503no0on timetraining
#53102yes0on timetraining
#66002no0on timetraining
#72101no0on timetraining
#82804no0on timetraining
#93603no0on timetraining
#101302no0on timetraining
#114505yes2latetraining
#123804yes1latetraining
#135606no1latetraining
#142005no2latetraining
#154702yes1latetraining
#163004no1latetraining
#172602no0on timetest
#185401no0on timetest
#191904yes1latetest
#204105no2latetest
#212303yes0on timetest
#223301yes1latetest
#231701no0on timetest
#243902no0on timetest

3 · Try it

Build it step by step

Work through the steps in order. Change your mind at any point and see what changes downstream.

Lab

A guided model project

Pick a question, set the baseline, choose features, keep the split, train a one-question model, test it once and write the model card.

Simulated in your browser: a one-question model is trained on 16 made-up loans when you press Train.

1 · Ask a question that has a label

The library’s records list each loan’s due date and return date, and nothing else about the readers. Which question can you train a model for?

2 · Set the baseline

The simplest possible model always predicts the most common answer among the training loans.

3 · Choose features

Which columns may the model look at? Pick one or more.

4 · Keep the split

Loans #1–16 are for training. Loans #17–24 (5 on time, 3 late) were chosen once, at random, and are locked until step 6.

5 · Train

The model is a decision stump: it tries every cut-off on every chosen feature and keeps the single yes or no question with the fewest mistakes on the training loans.

6 · Test once

Unlock the 8 test loans and compare the model with the baseline.

7 · Document and reflect

Write the purpose and at least one limit. The rest of the card fills in from your choices.

Your model card

The lab walks through all seven steps with the 24 made-up loans below. Without it, do the same on paper: count the training labels for the baseline, try each feature’s best yes or no question on loans #1–16, then check loans #17–24 once. This is what you should find:

Baseline and one-question models, one feature at a time
Feature the model may useQuestion it learnsTraining loans rightTest loans rightLate test loans caught
None: the baselineAlways say on time.10 of 165 of 80 of 3
PagesDoes the book have at least 370 pages? Yes: late. No: on time.12 of 164 of 81 of 3
Books borrowed at onceWere 4 or more books borrowed at once? Yes: late. No: on time.14 of 167 of 82 of 3
Due in exam weekIs it due in exam week? Yes: late. No: on time.11 of 166 of 82 of 3
Reminder emails sentWas at least 1 reminder email sent? Yes: late. No: on time.16 of 168 of 83 of 3

Reminder emails are only sent after a book is already late, so a model that uses them is reading the answer. That is data leakage, and the perfect score is not real. With several features allowed, the model picks the one with the fewest training mistakes: books borrowed at once, unless reminder emails are allowed.

Why the test loans stay locked. If you use them to choose features, cut-offs or models, they stop being new to the model, and the final score becomes too optimistic. When you need to compare options, split a third group, a validation set, off the training rows, or use cross-validation. Touch the test set once, at the end.

Worked example

One feature, every number shown

Here are steps 2, 5 and 6 by hand, using only “books borrowed at once.”

Worked example

Books borrowed at once, from baseline to test

  1. BaselineThe 16 training loans: 10 on time, 6 late. The baseline always says on time. Training: 10 ÷ 16 = 62.5%. Test: 5 ÷ 8 = 62.5%, and it catches 0 of the 3 late test loans.
  2. Try cut-offsBetween 2 and 3 books: 4 training mistakes. Between 3 and 4: 2 mistakes. Between 4 and 5: 3. Between 5 and 6: 5.
  3. Pick the bestRule: 4 or more books → late, otherwise on time. Training: 14 ÷ 16 = 87.5%. The 2 misses: loan #8 (4 books, on time) and loan #15 (2 books, late).
  4. Test onceLoans #19 (4 books) and #20 (5 books) are flagged late, and both were. Loan #22 (1 book) is called on time but came back late. 7 ÷ 8 = 87.5%, catching 2 of 3 late loans with no false alarms.
  5. CompareModel 7 of 8, baseline 5 of 8: the model is right on 2 more test loans. But each test loan is worth 12.5 percentage points, so this is a promising sign, not proof.

All 24 loans are invented for this page. The lab above computes the same numbers in your browser.

Predict first

Pages looked useful on the training loans: “at least 370 pages means late” gets 12 of 16 right, better than the baseline’s 10. How do you think it does on the 8 test loans?

Show the answer

Worse than the baseline: 4 of 8 (50%), against 5 of 8. It calls two long books late that were not (#18 and #24) and misses two shorter books that were late (#19 and #22). With only 16 training loans, a cut-off can fit chance patterns. This is why the test set, not the training score, gets the last word.

Optional

The same project in Python

For older students who want to run it for real. The data is the same made-up table, typed into the code.

The script uses scikit-learn, a free Python library for classical machine learning. DummyClassifier is the baseline, a DecisionTreeClassifier with max_depth=1 is the one-question model, and the split is fixed by row number so you get the same answer every time. Changing it and re-running until the test number goes up is exactly the habit step 4 warns about.

Show the Python version (scikit-learn, optional)
# Will a library book come back late? A tiny made-up dataset (24 loans).
# Run it with Python 3 after installing scikit-learn:  pip install scikit-learn
from sklearn.dummy import DummyClassifier
from sklearn.tree import DecisionTreeClassifier, export_text
from sklearn.metrics import accuracy_score, recall_score

FEATURES = ["pages", "books_at_once", "exam_week"]
# Each row: pages, books borrowed at once, due in exam week (1 = yes), late (1 = yes).
# "Reminder emails sent" is left out on purpose: it is only known after a book is late.
LOANS = [
    [180, 1, 0, 0],  # loan 1
    [240, 2, 0, 0],  # loan 2
    [520, 1, 1, 0],  # loan 3
    [150, 3, 0, 0],  # loan 4
    [310, 2, 1, 0],  # loan 5
    [600, 2, 0, 0],  # loan 6
    [210, 1, 0, 0],  # loan 7
    [280, 4, 0, 0],  # loan 8
    [360, 3, 0, 0],  # loan 9
    [130, 2, 0, 0],  # loan 10
    [450, 5, 1, 1],  # loan 11
    [380, 4, 1, 1],  # loan 12
    [560, 6, 0, 1],  # loan 13
    [200, 5, 0, 1],  # loan 14
    [470, 2, 1, 1],  # loan 15
    [300, 4, 0, 1],  # loan 16
    [260, 2, 0, 0],  # loan 17
    [540, 1, 0, 0],  # loan 18
    [190, 4, 1, 1],  # loan 19
    [410, 5, 0, 1],  # loan 20
    [230, 3, 1, 0],  # loan 21
    [330, 1, 1, 1],  # loan 22
    [170, 1, 0, 0],  # loan 23
    [390, 2, 0, 0],  # loan 24
]
X = [row[:3] for row in LOANS]
y = [row[3] for row in LOANS]

# Step 4: a fixed split. Loans 17-24 are the locked test set.
X_train, y_train = X[:16], y[:16]
X_test, y_test = X[16:], y[16:]

# Step 2: the baseline always predicts the most common training label.
baseline = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)

# Step 5: a decision stump, a tree that asks only one question.
stump = DecisionTreeClassifier(max_depth=1, random_state=0).fit(X_train, y_train)
print(export_text(stump, feature_names=FEATURES))

# Step 6: test once, on loans the models never saw.
for name, model in [("baseline", baseline), ("stump", stump)]:
    pred = model.predict(X_test)
    acc = accuracy_score(y_test, pred)
    caught = recall_score(y_test, pred, zero_division=0)
    print(f"{name}: test accuracy {acc:.3f}, late loans caught {caught:.3f}")

What it prints: the stump’s rule is “books_at_once <= 3.50” (3 or fewer books: on time; 4 or more: late). Baseline: test accuracy 0.625, late loans caught 0.000. Stump: test accuracy 0.875, late loans caught 0.667. scikit-learn picks its question with a slightly different score (Gini impurity) than the lab’s “fewest mistakes,” but on this data both choose the same question. Checked with scikit-learn 1.9.

Document and reflect

The model card is part of the model

A model nobody can check is a model nobody should trust.

What goes on the card

Four short headings

  • Purpose: what the model is for, who uses it, and what it should not be used for.
  • Data: where the rows came from, how many, which version, and how they were split.
  • Results: the test score next to the baseline, including the errors that matter, such as late loans missed.
  • Limits: who could be affected by mistakes, what the data does not cover, and what to do when it fails.

Researchers proposed “model cards” in 2019 as a standard short report for trained models. Real cards are longer, but the habit starts here.

Reflect

Should this model exist?

Even a good score leaves questions. A reminder email is harmless, but the same model used to refuse loans to students would punish people for borrowing a lot, which is what libraries want. The purpose decides what mistakes cost.

A real project would also need far more loans, a check that it works equally well for different groups of readers, and a plan for when borrowing habits change, such as a new exam timetable. Read more in Who checks that an AI system is fair and safe?

4 · Common mix-ups

Where beginner projects go wrong

Four common beliefs, each with what to think instead.

  • Mix-up: A high training score means the model works.

    Better: Only the locked test set shows how it does on new rows. Pages scored 12 of 16 in training and 4 of 8 in testing.

  • Mix-up: If accuracy is above 50%, the model learned something.

    Better: Compare with the baseline, not with 50%. “Always on time” already scores 62.5% here while catching no late loans.

  • Mix-up: More features always help.

    Better: A feature that is only known after the fact, like reminder emails, makes the score perfect and the model useless. Every feature must be available at the moment you predict.

  • Mix-up: Documentation is paperwork for after the real work.

    Better: The model card records what was tested and what was not. Without it, the next person cannot tell a careful model from a lucky one.

5 · Check yourself

Three quick questions

Answer in your head or on paper before you open each one.

  1. Your model scores 70% on the test set. Why can’t you tell yet whether it is good?

    Check your answer

    You need the baseline. If “always predict the most common answer” scores 75%, your model is less accurate than that baseline. But accuracy is not the same as usefulness: you also need to know which mistakes each one makes and what they cost: here the baseline scores 62.5% while missing every late loan.

  2. A classmate tests the model, sees 6 of 8, swaps the feature, tests again, gets 7 of 8 and reports 87.5%. What went wrong?

    Check your answer

    They used the test set to choose the feature, so it became a validation set. The 87.5% is now optimistic. They should compare features on a validation set split off the training loans, then test the final choice once on untouched rows.

  3. Why is “reminder emails sent” not allowed as a feature, even though it predicts lateness perfectly?

    Check your answer

    Reminders are sent only after a book is overdue, so the column is a consequence of the label. At checkout, when the prediction is needed, it does not exist yet. Using it is data leakage.

For younger learners

A grades 6–8 version

The same project on paper, no percentages needed.

40 minutes · pairs

Be the model

  1. Print the table of 24 loans and cut off loans #17–24 into a sealed envelope.
  2. Count late and on-time loans among #1–16. What would “always guess the most common answer” get right?
  3. Each pair writes one yes or no question about one column and counts how many of the 16 loans it gets right.
  4. Open the envelope once and score the question on the 8 hidden loans.

Exit question

Why was the envelope sealed?

Good answers say the hidden loans are a fair test, because nobody could fit their question to them. Ask too: why is “reminder emails sent” a cheat?

Warm-up idea: Train the Machine shows a model learning a misleading pattern.

Honest limits

What this page leaves out

What this page leaves out

Cleaning messy data, handling missing values, features made from text or dates, choosing among many models with cross-validation, deploying a model and monitoring it after launch. Twenty-four rows is far too few for a real decision: here, one test loan moves accuracy by 12.5 percentage points. Next on the builder path: sturdier splits, a repeatable pipeline, then deployment, monitoring and a plan to roll back.