AI foundations · 03

What did we teach it by accident?

Whatever pattern in the examples best predicts the labels, even one you never meant to teach: a snowy background, the camera that took the photo, or a column that already contains the answer. A model cannot tell which patterns you intended. Good data work means choosing examples on purpose, keeping an honest test aside, and writing down where the data came from.

Grades 6+20 minutesCounting and percentagesExtension for grades 9+

What you will be able to do

Spot a misleading feature, a missing group and a leak between training and testing; justify how a dataset should be split; and fill in a short dataset card.

Before you start

How can examples become a rule?: features, labels, and why a rule is tested on examples it has never seen.

1 · You have already seen this

A test is only fair if the questions are new

A score only tells you something when the test was fair.

If a teacher hands out the exam questions a week early, the whole class can score full marks. The marks then say little about who understands the subject, and a new set of questions would show that. A score is only worth something when the test asks about things the students have not already seen.

A model is in a similar spot. It learns from a dataset, a collection of examples that people gathered, labeled and split, and it picks up whatever clue separates the answers in those examples, even the wrong one. Its test score means something only if that data was chosen and set aside with care. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From examples to a score you can trust

Building a dataset in five steps. The three sections after this one go deeper into steps 1, 2 and 3, the lab lets you try step 4, and the section after the lab covers step 5.

How a dataset is built

Five steps

  1. 1 · Collect examplesSomeone decides who, where, when and with what device. The model only knows that part of the world, and it will use any clue that separates the answers in it, even the background of a photo.
  2. 2 · Attach the labelsA person or a recording process writes the answer for each example, such as “goat” or “sheep.” The model treats those labels as the right answers, mistakes included.
  3. 3 · Set some asideBefore any learning starts, split the examples into a training set, a validation set and a test set. The test set stays locked until one final check at the end.
  4. 4 · Check every columnFor each input, ask: will I know this at the moment I need the prediction? A column that is only filled in afterwards can make a model look perfect in testing and fail in use.
  5. 5 · Write it downA dataset card records where the data came from, how it was labeled, its known gaps and what it may be used for, so the next person knows what it cannot do.

Going deeper · step 1

The model learned the wrong thing

A model is not trying to learn what you meant. It finds whatever separates the labels in the examples you gave it.

In Train the Machine, every Zib in the first training set happens to be blue. A rule like “blue means Zib” fits all 12 training creatures, then fails on many new ones. Nothing in the data said color was the wrong clue. Only more varied examples reveal it.

Researchers have shown the same thing with real photos. In a 2016 study, a team deliberately trained a classifier on wolf and husky photos chosen so that the wolves stood in snow and the huskies did not. An explanation tool, which highlights the parts of a photo a model relied on, showed it was mostly using the snow. The model was not cheating. Snow really did predict the label in that data.

To a model, a photo is just numbers for pixel colors (see how a picture becomes numbers). If most of the pixels in every goat photo are white snow, “a lot of white” is a perfectly good rule for the training set. The question is how to find out which rule it learned.

Predict first

A goat-or-sheep model scores 100% on test photos taken on the same trips as its training photos. Does that show it can recognize goats?

Show the answer

No. If the test photos share the same shortcut (goats in snow, sheep on grass), a model that learned “white means goat” also scores 100%. A test can only catch a shortcut if it includes examples where the shortcut gives the wrong answer. Try it in the lab below.

Lab

Goats, sheep or snow?

Choose the extra set of photos that would actually test whether the model learned the animals. Try the others too and see why they do not.

Simulated in your browser with made-up photo cards. No AI model runs here.

Training photos (8). Every goat photo came from a winter trip. Every sheep photo came from the home farm in summer.

Goaton snow
Goaton snow
Goaton snow
Goaton snow
Sheepon grass
Sheepon grass
Sheepon grass
Sheepon grass

The model scores 8 of 8 on these, and 10 of 10 on test photos from the same trips. Which extra set of photos would show whether it learned goats, or snow?

Pick one set. The model will label every photo in it.

Training photos: 4 goats, all on snow, and 4 sheep, all on grass. The model scores 8 of 8 on them. Which extra set would show whether it learned the animals or the snow?

  • 10 more goats from the same snowy trip. Perfect, but it proves nothing. Every goat is still standing in snow, so “goat” and “snow” give exactly the same answer. More of the same data cannot tell them apart.
  • 10 sheep from a different farm, also on grass. Perfect again, and checking a new farm is a good habit. But the sheep are still on green grass, so this set cannot separate “sheep” from “grass.” It does not test the snow idea.
  • 5 goats on summer grass and 5 sheep in snow. This is the set that tests it. Animal and background now disagree, so the two explanations predict different answers, and every sheep in snow became a goat, and every goat on grass became a sheep. It learned the weather, not the animals. (This simulated model looks only at background color, which is exactly what a real model can end up doing when the data lets it.) The fix is in the data: collect both animals on both backgrounds, retrain, and test again on new photos.
  • The 8 training photos again, flipped left to right. Perfect, on photos it has already seen. Flipping a photo does not change its background, and a model should never be judged on its own training examples.

Going deeper · steps 1 and 2

Collection, features, labels and gaps

Every column and every row is the result of a choice someone made. Those choices decide what the model can learn.

The parts of a dataset
WordWhat it meansIn the goat and sheep photos
CollectionHow the examples were gathered: who, where, when, with what deviceOne farmer’s phone: goats on a winter trip, sheep at home in summer
Example (row)One item in the datasetOne photo
FeatureAn input the model can useThe pixel colors, or a summary such as “mostly white”
LabelThe answer someone attached to each example“goat” or “sheep,” typed by the farmer
Missing valueA blank where a value should beSix photos have no date (made up)
SampleThe part of the world you collected, standing in for the restTwo places, two seasons, daylight only

The sample is the part of the world you happened to collect, and the model only knows that part. Photos taken only in daylight teach nothing about dusk. A model trained on one farm’s animals may struggle on another farm. When the world the model meets later differs from its sample, people call it a distribution shift.

Labels are made by people or by a recording process, and both can be wrong or inconsistent. If one person labels every photo, that person’s mistakes become the “right answers.”

Missing values are rarely random. If a weather sensor tends to fail in hard frost, its blanks fall on the coldest days, and quietly deleting those rows deletes the coldest days from the data. You can drop rows, fill blanks with a typical value, or add a column that says “this was missing.” Each choice changes what the model sees.

Going deeper · step 3

Training, validation and test

Before any learning starts, set some examples aside. Like practice problems, a practice quiz and a final exam with questions you have not seen.

Three jobs, three separate sets
SetWhat it is forRule of thumb
Training setThe model learns its parameters from these examples.The largest part, for example 70%.
Validation setYou compare models and settings on these, such as which columns to use, and pick one.For example 15%. Each time you use it to choose, it becomes a little less fair.
Test setOne final check on examples that nothing was tuned on.For example 15%. Look once, at the end, and report that number.

If you keep comparing models on the validation set, you slowly fit your choices to it. That is why the test set stays locked until the end. The percentages above are only a starting point. With very little data, people often rotate which part is held out (called cross-validation).

Preprocessing counts as training. Filling blanks with an average, or rescaling a column, uses numbers computed from data. Compute them from the training set only, then apply those same numbers to the validation and test sets. In the library loans below, the 12 training loans average about 288 pages; if you also included the 8 test loans, the average would be about 294. The difference is small, but the test loans would no longer be unseen. The scikit-learn guide lists this as a common pitfall and recommends pipelines that fit each step on training data only.

Going further · grades 9+

When a random split is not fair

A random split assumes every row is a separate, independent example. Two common situations break that.

The same subject appears many times. Suppose a photo set has 20 photos each of 15 goats. A random split puts photos of the same goat in both training and test, so the model can score well by recognizing individual goats it has already seen, by a scar or a collar, not goats in general. A grouped split keeps every photo of one animal on the same side. The same goes for several recordings of one speaker or many readings from one sensor.

Time matters. If the model will predict next month, test it like next month. A random split of a year of library loans trains on June and tests on March, so the model learns from the future: for example, that a book became popular after a film came out. A time-based split trains on earlier loans and tests on later ones. In scikit-learn these are GroupKFold and TimeSeriesSplit.

3 · Try it

Perfect in testing, useless in use

Data leakage means training with information that will not be available when the model is actually used. The classic sign is a score that looks too good.

Lab

The library’s late-book predictor

Train with all three columns and test. Then use the model at checkout, when the books have not come back yet. Then switch off the column that leaked and try again.

Simulated in your browser: a one-rule learner on 28 made-up loans. No AI model runs here.

The library wants to send a friendly reminder for loans likely to come back late. Choose which columns the model may use (highlighted means switched on), then train it.

All three columns are switched on. Press “Train and test.”
See all 28 made-up loans
12 training loans
LoanPagesPast late returnsFine chargedLate?
A11200€0.00no
A23401€0.00no
A34103€1.50yes
A4950€0.00no
A52802€0.50yes
A65100€0.00no
A72202€0.00no
A81504€2.00yes
A93001€0.00no
A104601€1.00yes
A111800€0.00no
A123903€3.00yes
8 test loans (older loans kept aside)
LoanPagesPast late returnsFine chargedLate?
B12000€0.00no
B24503€2.00yes
B31302€0.00no
B43801€1.00yes
B52600€0.00no
B61702€0.50yes
B75200€0.00no
B83103€1.50yes
8 new loans at checkout (late or not, found out later)
LoanPagesPast late returnsFine chargedLate?
C12400€0.00 (not known yet)no
C24303€0.00 (not known yet)yes
C31602€0.00 (not known yet)yes
C43500€0.00 (not known yet)no
C51102€0.00 (not known yet)no
C64802€0.00 (not known yet)no
C72904€0.00 (not known yet)yes
C82000€0.00 (not known yet)no

A one-rule learner tries every rule of the form “column ≥ value → late” on 12 made-up training loans and keeps the best. The tables show all the data.

12 training loans
LoanPagesPast late returnsFine chargedLate?
A11200€0.00no
A23401€0.00no
A34103€1.50yes
A4950€0.00no
A52802€0.50yes
A65100€0.00no
A72202€0.00no
A81504€2.00yes
A93001€0.00no
A104601€1.00yes
A111800€0.00no
A123903€3.00yes
8 test loans (older loans kept aside)
LoanPagesPast late returnsFine chargedLate?
B12000€0.00no
B24503€2.00yes
B31302€0.00no
B43801€1.00yes
B52600€0.00no
B61702€0.50yes
B75200€0.00no
B83103€1.50yes
8 new loans at checkout (late or not, found out later)
LoanPagesPast late returnsFine chargedLate?
C12400€0.00 (not known yet)no
C24303€0.00 (not known yet)yes
C31602€0.00 (not known yet)yes
C43500€0.00 (not known yet)no
C51102€0.00 (not known yet)no
C64802€0.00 (not known yet)no
C72904€0.00 (not known yet)yes
C82000€0.00 (not known yet)no
  • All columns allowed: it picks “fine charged ≥ €0.50 → late.” Training 12 of 12, test 8 of 8 (100%). At checkout every fine is €0, so in use it says “on time” for everything: 5 of 8 right (63%), 0 of 3 late books caught.
  • Fine charged switched off: it picks “past late returns ≥ 2 → late.” Training 10 of 12, test 6 of 8 (75%), in use 6 of 8 (75%), 3 of 3 late books caught with 2 false alarms.
  • Pages only: “pages ≥ 390 → late.” Training 9 of 12, test 4 of 8 (50%), in use 5 of 8, 1 of 3 late books caught.

Worked example

Where the 100% went

  1. Rule learnedFine charged ≥ €0.50 → late. It is right on every training loan: 12 ÷ 12 = 100%.
  2. TestOn 8 older loans kept aside, which also have their fines filled in: 8 ÷ 8 = 100%.
  3. In useAt checkout every fine is €0, so it says “on time” for all 8 new loans. 5 really were on time and 3 came back late: 5 ÷ 8 = 62.5%.
  4. BaselineA rule that always says “on time,” with no model at all: also 5 ÷ 8 = 62.5%.
  5. Late books found0 of 3. The one job it had, it cannot do.
  6. Without the leakPast late returns ≥ 2 → late. Test 6 ÷ 8 = 75%, in use 6 ÷ 8 = 75%, and all 3 late books found (with 2 false alarms).

All loans are made up. A fine is only charged after a late return, so the column records the answer itself.

To catch a leak, ask of every input column: will I know this at the moment I need the prediction? A fine, a refund issued after a complaint, a “closed” date, or tomorrow’s weather reading all fail that question. A model without leaks may score lower in testing, but that score is a fair preview of how it will do in use.

Write it down

A dataset card, and why privacy starts here

A dataset card is a short document that travels with a dataset, so the next person knows what it is for and what it cannot do.

The idea comes from proposals such as Datasheets for Datasets (Gebru and colleagues), which ask dataset makers to answer questions about motivation, collection, labeling and intended uses. A classroom version can be one table with seven fields.

Privacy and fairness are decided at collection, long before any model exists:

What you record. The library did not need names, only a code to keep one borrower’s loans together. Who is in the sample. If one group borrows most, the model learns mostly about them. What a column stands for. “Past late returns” predicts well, but it is a fact about a person, and a reminder that follows them around can feel like a penalty. Allowed use. Data collected to run a library should not quietly be reused to rank students.

Classroom datasets should be made up, or public and non-sensitive.

A short dataset card
FieldSchool library loans 2025–26 (a made-up example)
PurposeTrain a model that suggests which loans should get a friendly reminder before the due date. Not for deciding who may borrow.
OriginOne school library’s loan system, exported by the librarian. September 2025 to June 2026.
Collection methodEvery loan recorded automatically at checkout and return: 1,240 loans. Names and student numbers removed; each borrower replaced by a random code, so loans by the same person can be kept together when splitting.
Label process“Late” means returned after the due date, read from the return date. 37 books never came back; they have no label and were left out.
Known gapsNo data from July and August, when the library is closed. Grades 1 and 2 borrow on one class card, so they look like one very busy borrower. The “fine charged” column was removed because it is only known after a book is returned.
Allowed useTraining and testing the reminder model inside the school. Only summary counts may be shared outside it.
Intended limitsNot for judging or ranking students. Not tested on other schools or on summer reading. Check again each school year.

In the real world

An application, a limit, a connection

An application: the weed-spotting camera on the farming page is only as good as its photos. Ask what changes with lighting, soil or season. A limitation: no dataset covers everything, so a dataset card’s “known gaps” field is a list of places the model has never been tested. A connection: the same habits decide whether a score means anything in evaluating AI, a later stop, and in fair classroom tests of chatbots.

4 · Common mix-ups

Three ideas that are easy to get backwards

Each one comes with what to think instead.

  • Mix-up: More data always fixes it.

    Better: More of the same data repeats the same shortcut. In the lab, 10 more snowy goats scored 10 of 10 and proved nothing. What helps is the right variety of examples. More of the same adds little.

  • Mix-up: A 100% test score means the model is great.

    Better: Often it means a leak. Check every input: will it be known at the moment of prediction?

  • Mix-up: Removing names makes a dataset private.

    Better: Other columns can still point to a person, especially rare combinations. And privacy is also about how the data is used, not only what it contains.

5 · Check yourself

Three quick questions

Answer in your head or on paper before you open each one.

  1. A class takes 20 photos each of 15 potted plants, labeled “needs water” or “fine,” to build a model for other people’s plants. How should they split the photos into training and test sets, and why?

    Check your answer

    By plant, not by photo: put all 20 photos of a plant on the same side, for example 11 plants for training and 4 for testing. A random split would put photos of the same pot in both sets, and the model could score well by recognizing those plants. The goal is plants it has never seen, so the test must contain only unseen plants.

  2. A rain predictor scores 98% on its test set. One input column is “rain gauge reading at 9 a.m. tomorrow.” What went wrong?

    Check your answer

    That is data leakage: the column holds information from the future, which will not exist when the forecast is made. Remove it, retrain, and expect a lower but honest score. A time-based split (train on earlier days, test on later ones) is also the fair way to test a forecast.

  3. A bird club collects 2,000 bird photos, all taken in summer, in daylight, at one lake, and all labeled by one member. Name two ways this dataset could mislead a model.

    Check your answer

    Any two of these: the sample misses winter plumage, dawn and dusk light, and birds of other places, so the model may fail there; the background (water, reeds) may become a shortcut for water birds; and the labels carry one person’s mistakes with no second check. A dataset card would list these as known gaps.

Honest limits

What this page leaves out

What this page leaves out

How to measure results in detail (accuracy, precision, recall), how much data is “enough,” data augmentation, labeling tools, and the law: privacy rules differ by country and by school, so check yours before collecting anything about people. Evaluating AI picks up the measuring part.

For younger learners

A grades 3–5 version

No splits or leaks by name: one picture-card sort and one good question.

20 minutes · teacher-led

What did the robot notice?

  1. Make eight picture cards: four goats on white snow and four sheep on green grass.
  2. One student is the “robot.” They study the cards, then guess new cards with their eyes half-closed, so they see mostly colors.
  3. Show a sheep on snow. What does the robot say? Why?
  4. Ask the class which new cards would catch the robot’s mistake.

Exit question

What did we teach it by accident?

Good answers name the shortcut (“white means goat”) and say how to fix it (“show it goats on grass and sheep in snow too”).

Next activity: Train the Machine, where better data fixes a bad model.