AI foundations · 03
What did we teach it by accident?
Whatever pattern in the examples best predicts the labels, even one you never meant to teach: a snowy background, the camera that took the photo, or a column that already contains the answer. A model cannot tell which patterns you intended. Good data work means choosing examples on purpose, keeping an honest test aside, and writing down where the data came from.
What you will be able to do
Spot a misleading feature, a missing group and a leak between training and testing; justify how a dataset should be split; and fill in a short dataset card.
Before you start
How can examples become a rule?: features, labels, and why a rule is tested on examples it has never seen.
1 · You have already seen this
A test is only fair if the questions are new
A score only tells you something when the test was fair.
If a teacher hands out the exam questions a week early, the whole class can score full marks. The marks then say little about who understands the subject, and a new set of questions would show that. A score is only worth something when the test asks about things the students have not already seen.
A model is in a similar spot. It learns from a dataset, a collection of examples that people gathered, labeled and split, and it picks up whatever clue separates the answers in those examples, even the wrong one. Its test score means something only if that data was chosen and set aside with care. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From examples to a score you can trust
Building a dataset in five steps. The three sections after this one go deeper into steps 1, 2 and 3, the lab lets you try step 4, and the section after the lab covers step 5.
How a dataset is built
Five steps
- 1 · Collect examplesSomeone decides who, where, when and with what device. The model only knows that part of the world, and it will use any clue that separates the answers in it, even the background of a photo.
- 2 · Attach the labelsA person or a recording process writes the answer for each example, such as “goat” or “sheep.” The model treats those labels as the right answers, mistakes included.
- 3 · Set some asideBefore any learning starts, split the examples into a training set, a validation set and a test set. The test set stays locked until one final check at the end.
- 4 · Check every columnFor each input, ask: will I know this at the moment I need the prediction? A column that is only filled in afterwards can make a model look perfect in testing and fail in use.
- 5 · Write it downA dataset card records where the data came from, how it was labeled, its known gaps and what it may be used for, so the next person knows what it cannot do.
Going deeper · step 1
The model learned the wrong thing
A model is not trying to learn what you meant. It finds whatever separates the labels in the examples you gave it.
In Train the Machine, every Zib in the first training set happens to be blue. A rule like “blue means Zib” fits all 12 training creatures, then fails on many new ones. Nothing in the data said color was the wrong clue. Only more varied examples reveal it.
Researchers have shown the same thing with real photos. In a 2016 study, a team deliberately trained a classifier on wolf and husky photos chosen so that the wolves stood in snow and the huskies did not. An explanation tool, which highlights the parts of a photo a model relied on, showed it was mostly using the snow. The model was not cheating. Snow really did predict the label in that data.
To a model, a photo is just numbers for pixel colors (see how a picture becomes numbers). If most of the pixels in every goat photo are white snow, “a lot of white” is a perfectly good rule for the training set. The question is how to find out which rule it learned.
Predict first
A goat-or-sheep model scores 100% on test photos taken on the same trips as its training photos. Does that show it can recognize goats?
Show the answer
No. If the test photos share the same shortcut (goats in snow, sheep on grass), a model that learned “white means goat” also scores 100%. A test can only catch a shortcut if it includes examples where the shortcut gives the wrong answer. Try it in the lab below.
Lab
Goats, sheep or snow?
Choose the extra set of photos that would actually test whether the model learned the animals. Try the others too and see why they do not.
Simulated in your browser with made-up photo cards. No AI model runs here.
Training photos (8). Every goat photo came from a winter trip. Every sheep photo came from the home farm in summer.
The model scores 8 of 8 on these, and 10 of 10 on test photos from the same trips. Which extra set of photos would show whether it learned goats, or snow?
Training photos: 4 goats, all on snow, and 4 sheep, all on grass. The model scores 8 of 8 on them. Which extra set would show whether it learned the animals or the snow?
- 10 more goats from the same snowy trip. Perfect, but it proves nothing. Every goat is still standing in snow, so “goat” and “snow” give exactly the same answer. More of the same data cannot tell them apart.
- 10 sheep from a different farm, also on grass. Perfect again, and checking a new farm is a good habit. But the sheep are still on green grass, so this set cannot separate “sheep” from “grass.” It does not test the snow idea.
- 5 goats on summer grass and 5 sheep in snow. This is the set that tests it. Animal and background now disagree, so the two explanations predict different answers, and every sheep in snow became a goat, and every goat on grass became a sheep. It learned the weather, not the animals. (This simulated model looks only at background color, which is exactly what a real model can end up doing when the data lets it.) The fix is in the data: collect both animals on both backgrounds, retrain, and test again on new photos.
- The 8 training photos again, flipped left to right. Perfect, on photos it has already seen. Flipping a photo does not change its background, and a model should never be judged on its own training examples.
Going deeper · steps 1 and 2
Collection, features, labels and gaps
Every column and every row is the result of a choice someone made. Those choices decide what the model can learn.
| Word | What it means | In the goat and sheep photos |
|---|---|---|
| Collection | How the examples were gathered: who, where, when, with what device | One farmer’s phone: goats on a winter trip, sheep at home in summer |
| Example (row) | One item in the dataset | One photo |
| Feature | An input the model can use | The pixel colors, or a summary such as “mostly white” |
| Label | The answer someone attached to each example | “goat” or “sheep,” typed by the farmer |
| Missing value | A blank where a value should be | Six photos have no date (made up) |
| Sample | The part of the world you collected, standing in for the rest | Two places, two seasons, daylight only |
The sample is the part of the world you happened to collect, and the model only knows that part. Photos taken only in daylight teach nothing about dusk. A model trained on one farm’s animals may struggle on another farm. When the world the model meets later differs from its sample, people call it a distribution shift.
Labels are made by people or by a recording process, and both can be wrong or inconsistent. If one person labels every photo, that person’s mistakes become the “right answers.”
Missing values are rarely random. If a weather sensor tends to fail in hard frost, its blanks fall on the coldest days, and quietly deleting those rows deletes the coldest days from the data. You can drop rows, fill blanks with a typical value, or add a column that says “this was missing.” Each choice changes what the model sees.
Going deeper · step 3
Training, validation and test
Before any learning starts, set some examples aside. Like practice problems, a practice quiz and a final exam with questions you have not seen.
| Set | What it is for | Rule of thumb |
|---|---|---|
| Training set | The model learns its parameters from these examples. | The largest part, for example 70%. |
| Validation set | You compare models and settings on these, such as which columns to use, and pick one. | For example 15%. Each time you use it to choose, it becomes a little less fair. |
| Test set | One final check on examples that nothing was tuned on. | For example 15%. Look once, at the end, and report that number. |
If you keep comparing models on the validation set, you slowly fit your choices to it. That is why the test set stays locked until the end. The percentages above are only a starting point. With very little data, people often rotate which part is held out (called cross-validation).
Preprocessing counts as training. Filling blanks with an average, or rescaling a column, uses numbers computed from data. Compute them from the training set only, then apply those same numbers to the validation and test sets. In the library loans below, the 12 training loans average about 288 pages; if you also included the 8 test loans, the average would be about 294. The difference is small, but the test loans would no longer be unseen. The scikit-learn guide lists this as a common pitfall and recommends pipelines that fit each step on training data only.
Going further · grades 9+
When a random split is not fair
A random split assumes every row is a separate, independent example. Two common situations break that.
The same subject appears many times. Suppose a photo set has 20 photos each of 15 goats. A random split puts photos of the same goat in both training and test, so the model can score well by recognizing individual goats it has already seen, by a scar or a collar, not goats in general. A grouped split keeps every photo of one animal on the same side. The same goes for several recordings of one speaker or many readings from one sensor.
Time matters. If the model will predict next month, test it like next month. A random split of a year of library loans trains on June and tests on March, so the model learns from the future: for example, that a book became popular after a film came out. A time-based split trains on earlier loans and tests on later ones. In scikit-learn these are GroupKFold and TimeSeriesSplit.
3 · Try it
Perfect in testing, useless in use
Data leakage means training with information that will not be available when the model is actually used. The classic sign is a score that looks too good.
Lab
The library’s late-book predictor
Train with all three columns and test. Then use the model at checkout, when the books have not come back yet. Then switch off the column that leaked and try again.
Simulated in your browser: a one-rule learner on 28 made-up loans. No AI model runs here.
The library wants to send a friendly reminder for loans likely to come back late. Choose which columns the model may use (highlighted means switched on), then train it.
See all 28 made-up loans
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| A1 | 120 | 0 | €0.00 | no |
| A2 | 340 | 1 | €0.00 | no |
| A3 | 410 | 3 | €1.50 | yes |
| A4 | 95 | 0 | €0.00 | no |
| A5 | 280 | 2 | €0.50 | yes |
| A6 | 510 | 0 | €0.00 | no |
| A7 | 220 | 2 | €0.00 | no |
| A8 | 150 | 4 | €2.00 | yes |
| A9 | 300 | 1 | €0.00 | no |
| A10 | 460 | 1 | €1.00 | yes |
| A11 | 180 | 0 | €0.00 | no |
| A12 | 390 | 3 | €3.00 | yes |
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| B1 | 200 | 0 | €0.00 | no |
| B2 | 450 | 3 | €2.00 | yes |
| B3 | 130 | 2 | €0.00 | no |
| B4 | 380 | 1 | €1.00 | yes |
| B5 | 260 | 0 | €0.00 | no |
| B6 | 170 | 2 | €0.50 | yes |
| B7 | 520 | 0 | €0.00 | no |
| B8 | 310 | 3 | €1.50 | yes |
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| C1 | 240 | 0 | €0.00 (not known yet) | no |
| C2 | 430 | 3 | €0.00 (not known yet) | yes |
| C3 | 160 | 2 | €0.00 (not known yet) | yes |
| C4 | 350 | 0 | €0.00 (not known yet) | no |
| C5 | 110 | 2 | €0.00 (not known yet) | no |
| C6 | 480 | 2 | €0.00 (not known yet) | no |
| C7 | 290 | 4 | €0.00 (not known yet) | yes |
| C8 | 200 | 0 | €0.00 (not known yet) | no |
A one-rule learner tries every rule of the form “column ≥ value → late” on 12 made-up training loans and keeps the best. The tables show all the data.
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| A1 | 120 | 0 | €0.00 | no |
| A2 | 340 | 1 | €0.00 | no |
| A3 | 410 | 3 | €1.50 | yes |
| A4 | 95 | 0 | €0.00 | no |
| A5 | 280 | 2 | €0.50 | yes |
| A6 | 510 | 0 | €0.00 | no |
| A7 | 220 | 2 | €0.00 | no |
| A8 | 150 | 4 | €2.00 | yes |
| A9 | 300 | 1 | €0.00 | no |
| A10 | 460 | 1 | €1.00 | yes |
| A11 | 180 | 0 | €0.00 | no |
| A12 | 390 | 3 | €3.00 | yes |
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| B1 | 200 | 0 | €0.00 | no |
| B2 | 450 | 3 | €2.00 | yes |
| B3 | 130 | 2 | €0.00 | no |
| B4 | 380 | 1 | €1.00 | yes |
| B5 | 260 | 0 | €0.00 | no |
| B6 | 170 | 2 | €0.50 | yes |
| B7 | 520 | 0 | €0.00 | no |
| B8 | 310 | 3 | €1.50 | yes |
| Loan | Pages | Past late returns | Fine charged | Late? |
|---|---|---|---|---|
| C1 | 240 | 0 | €0.00 (not known yet) | no |
| C2 | 430 | 3 | €0.00 (not known yet) | yes |
| C3 | 160 | 2 | €0.00 (not known yet) | yes |
| C4 | 350 | 0 | €0.00 (not known yet) | no |
| C5 | 110 | 2 | €0.00 (not known yet) | no |
| C6 | 480 | 2 | €0.00 (not known yet) | no |
| C7 | 290 | 4 | €0.00 (not known yet) | yes |
| C8 | 200 | 0 | €0.00 (not known yet) | no |
- All columns allowed: it picks “fine charged ≥ €0.50 → late.” Training 12 of 12, test 8 of 8 (100%). At checkout every fine is €0, so in use it says “on time” for everything: 5 of 8 right (63%), 0 of 3 late books caught.
- Fine charged switched off: it picks “past late returns ≥ 2 → late.” Training 10 of 12, test 6 of 8 (75%), in use 6 of 8 (75%), 3 of 3 late books caught with 2 false alarms.
- Pages only: “pages ≥ 390 → late.” Training 9 of 12, test 4 of 8 (50%), in use 5 of 8, 1 of 3 late books caught.
Worked example
Where the 100% went
- Rule learnedFine charged ≥ €0.50 → late. It is right on every training loan: 12 ÷ 12 = 100%.
- TestOn 8 older loans kept aside, which also have their fines filled in: 8 ÷ 8 = 100%.
- In useAt checkout every fine is €0, so it says “on time” for all 8 new loans. 5 really were on time and 3 came back late: 5 ÷ 8 = 62.5%.
- BaselineA rule that always says “on time,” with no model at all: also 5 ÷ 8 = 62.5%.
- Late books found0 of 3. The one job it had, it cannot do.
- Without the leakPast late returns ≥ 2 → late. Test 6 ÷ 8 = 75%, in use 6 ÷ 8 = 75%, and all 3 late books found (with 2 false alarms).
All loans are made up. A fine is only charged after a late return, so the column records the answer itself.
To catch a leak, ask of every input column: will I know this at the moment I need the prediction? A fine, a refund issued after a complaint, a “closed” date, or tomorrow’s weather reading all fail that question. A model without leaks may score lower in testing, but that score is a fair preview of how it will do in use.
Write it down
A dataset card, and why privacy starts here
A dataset card is a short document that travels with a dataset, so the next person knows what it is for and what it cannot do.
The idea comes from proposals such as Datasheets for Datasets (Gebru and colleagues), which ask dataset makers to answer questions about motivation, collection, labeling and intended uses. A classroom version can be one table with seven fields.
Privacy and fairness are decided at collection, long before any model exists:
What you record. The library did not need names, only a code to keep one borrower’s loans together. Who is in the sample. If one group borrows most, the model learns mostly about them. What a column stands for. “Past late returns” predicts well, but it is a fact about a person, and a reminder that follows them around can feel like a penalty. Allowed use. Data collected to run a library should not quietly be reused to rank students.
Classroom datasets should be made up, or public and non-sensitive.
| Field | School library loans 2025–26 (a made-up example) |
|---|---|
| Purpose | Train a model that suggests which loans should get a friendly reminder before the due date. Not for deciding who may borrow. |
| Origin | One school library’s loan system, exported by the librarian. September 2025 to June 2026. |
| Collection method | Every loan recorded automatically at checkout and return: 1,240 loans. Names and student numbers removed; each borrower replaced by a random code, so loans by the same person can be kept together when splitting. |
| Label process | “Late” means returned after the due date, read from the return date. 37 books never came back; they have no label and were left out. |
| Known gaps | No data from July and August, when the library is closed. Grades 1 and 2 borrow on one class card, so they look like one very busy borrower. The “fine charged” column was removed because it is only known after a book is returned. |
| Allowed use | Training and testing the reminder model inside the school. Only summary counts may be shared outside it. |
| Intended limits | Not for judging or ranking students. Not tested on other schools or on summer reading. Check again each school year. |
In the real world
An application, a limit, a connection
An application: the weed-spotting camera on the farming page is only as good as its photos. Ask what changes with lighting, soil or season. A limitation: no dataset covers everything, so a dataset card’s “known gaps” field is a list of places the model has never been tested. A connection: the same habits decide whether a score means anything in evaluating AI, a later stop, and in fair classroom tests of chatbots.
4 · Common mix-ups
Three ideas that are easy to get backwards
Each one comes with what to think instead.
Mix-up: More data always fixes it.
Better: More of the same data repeats the same shortcut. In the lab, 10 more snowy goats scored 10 of 10 and proved nothing. What helps is the right variety of examples. More of the same adds little.
Mix-up: A 100% test score means the model is great.
Better: Often it means a leak. Check every input: will it be known at the moment of prediction?
Mix-up: Removing names makes a dataset private.
Better: Other columns can still point to a person, especially rare combinations. And privacy is also about how the data is used, not only what it contains.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
A class takes 20 photos each of 15 potted plants, labeled “needs water” or “fine,” to build a model for other people’s plants. How should they split the photos into training and test sets, and why?
Check your answer
By plant, not by photo: put all 20 photos of a plant on the same side, for example 11 plants for training and 4 for testing. A random split would put photos of the same pot in both sets, and the model could score well by recognizing those plants. The goal is plants it has never seen, so the test must contain only unseen plants.
A rain predictor scores 98% on its test set. One input column is “rain gauge reading at 9 a.m. tomorrow.” What went wrong?
Check your answer
That is data leakage: the column holds information from the future, which will not exist when the forecast is made. Remove it, retrain, and expect a lower but honest score. A time-based split (train on earlier days, test on later ones) is also the fair way to test a forecast.
A bird club collects 2,000 bird photos, all taken in summer, in daylight, at one lake, and all labeled by one member. Name two ways this dataset could mislead a model.
Check your answer
Any two of these: the sample misses winter plumage, dawn and dusk light, and birds of other places, so the model may fail there; the background (water, reeds) may become a shortcut for water birds; and the labels carry one person’s mistakes with no second check. A dataset card would list these as known gaps.
Honest limits
What this page leaves out
What this page leaves out
How to measure results in detail (accuracy, precision, recall), how much data is “enough,” data augmentation, labeling tools, and the law: privacy rules differ by country and by school, so check yours before collecting anything about people. Evaluating AI picks up the measuring part.
For younger learners
A grades 3–5 version
No splits or leaks by name: one picture-card sort and one good question.
20 minutes · teacher-led
What did the robot notice?
- Make eight picture cards: four goats on white snow and four sheep on green grass.
- One student is the “robot.” They study the cards, then guess new cards with their eyes half-closed, so they see mostly colors.
- Show a sheep on snow. What does the robot say? Why?
- Ask the class which new cards would catch the robot’s mistake.
Exit question
What did we teach it by accident?
Good answers name the shortcut (“white means goat”) and say how to fix it (“show it goats on grass and sheep in snow too”).
Next activity: Train the Machine, where better data fixes a bad model.
Where to go next
From data to numbers
Sources
Where this comes from
- Google for Developers, Machine Learning Crash Course, “Dividing the original dataset”: training, validation and test sets.
- scikit-learn User Guide, “Common pitfalls and recommended practices”: data leakage and fitting preprocessing on training data only.
- scikit-learn User Guide, “Cross-validation”: grouped splits (GroupKFold) and time-series splits (TimeSeriesSplit).
- Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin, “Why Should I Trust You?” (2016): the wolf-and-husky classifier that relied on snow.
- Timnit Gebru and colleagues, “Datasheets for Datasets”: questions to document a dataset’s motivation, collection, labeling and uses.
- NIST, AI Risk Management Framework: purpose, affected people, data and limits as parts of responsible AI.
AI words on this page
Training data, Feature, Label, Validation set, Test set, Cross-validation, Data leakage, Distribution shift, Generalization, Baseline, Base rate, Dataset card, Bias, Correlation vs causation
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.