Go deeper · How machines learn

How sure should an AI be?

As sure as the evidence allows, and no surer. A probability is a number from 0 to 1 that says how often something happens in situations like this one. A trustworthy system gives probabilities that match what really happens, and a careful user asks what the number is “out of”: how common the thing was to begin with.

Grades 9+20 minutesFractions and percentagesGrades 6–8: count, don’t divide

What you will be able to do

Turn percentages into counts out of 1,000 to find the chance that a flagged item is really positive, explain why base rates matter, check whether forecasts are calibrated, and tell correlation from causation.

Before you start

Read How to evaluate AI first (false positives, precision and recall). You need to be comfortable with fractions and percentages.

1 · You have already seen this

The forecast said 70% chance of rain

Weather apps put a number on tomorrow, and many of us check it before deciding on an umbrella.

A forecast of “70% chance of rain” sounds simple, but people read it in different ways. Some think it means rain for 70% of the day, or over 70% of the town. A U.S. National Weather Service forecast means that, on days like this one, the place gets at least a little rain about 7 times out of 10.

AI systems give numbers like this all the time. A spam filter gives each email a score, and a language model gives every possible next word a chance. Each of these is a probability, and reading one well means asking how sure the system should be and how common the thing was to begin with. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From a score to a careful reading

Reading a probability from an AI system in five steps. The three sections after this one go deeper into step 1, steps 2 and 3, and step 4. After the lab, one more goes deeper into step 5.

How to read a probability

Five steps

  1. 1 · Ask which way roundThese two questions sound alike and can have very different answers: “Of all the spam, how much gets flagged?” and “Of everything flagged, how much is spam?”
  2. 2 · Start from the base rateAsk how common the thing is before any test. Imagine 1,000 cases and split them into the two groups.
  3. 3 · Count the flagsApply the catch rate to one group and the false-alarm rate to the other, then add the flags. Counting like this is much easier than multiplying percentages.
  4. 4 · Check the number is honestA probability is only useful if it is calibrated: across many cases where a system said 70%, it should be right about 70% of the time. One case cannot show that.
  5. 5 · Don’t read in a causeA model learns what goes together in its data. It cannot tell, on its own, which one causes which.

Going deeper · step 1

A number for how often

Probability is the language AI systems use for uncertainty, and the place where people most often misread them.

A probability runs from 0 (never) to 1 (always), often written as a percentage. When a U.S. National Weather Service forecast says “70% chance of rain,” it means that, on days like this, the place will get at least a little measurable rain about 7 times out of 10. It does not mean rain for 70% of the day or over 70% of the town.

AI systems output probabilities all the time. A spam filter gives each email a score such as 0.92. A language model gives every possible next word a probability, and all of them add up to 1: that list is a probability distribution, and the model picks from it (see How transformers work).

Most useful probabilities are conditional: the chance of one thing given that another is true. People write P(spam | flagged), read “the probability of spam, given that the email was flagged.” Swap the two sides and you get a different question, P(flagged | spam): how often spam gets flagged. Mixing these up is one of the most common mistakes in reading any test, human or AI.

Two questions that sound alike

Which way round?

P(flagged | spam): of all the spam, how much gets flagged? This is the filter’s catch rate (recall).

P(spam | flagged): of everything flagged, how much is really spam? This is what you want to know when you look at your spam folder (precision).

They can be very different, as the worked example shows.

Going deeper · steps 2 and 3

How common was it to begin with?

The base rate is how common something is before any test. It changes what a positive result means.

Predict first

A filter catches 90% of spam and wrongly flags only 5% of real emails. In a week when just 2% of emails are spam, an email gets flagged. Is it more likely to be spam or real?

Show the answer

More likely real. Only about 27% of flagged emails are spam that week. With so little spam around, the small false-alarm rate applied to the huge number of real emails produces more flags than the spam does. The worked example counts it out.

Worked example

Out of 1,000 emails: is a flagged email really spam?

  1. The filterA made-up spam filter catches 90% of spam and wrongly flags 5% of real emails. Both sound good.
  2. Busy week: 20% spamOf 1,000 emails, 200 are spam and 800 are real. Caught: 90% of 200 = 180. False alarms: 5% of 800 = 40. Flagged in total: 180 + 40 = 220.
  3. Chance a flagged one is spam180 ÷ 220 ≈ 82%. Not 90%: some flags land on real emails.
  4. Quiet week: 2% spamSame filter. 20 spam, 980 real. Caught: 90% of 20 = 18. False alarms: 5% of 980 = 49. Flagged: 18 + 49 = 67.
  5. Chance a flagged one is spam18 ÷ 67 ≈ 27%. Most flagged emails are now real. The filter did not get worse; spam got rarer, so the 5% of false alarms outnumber the real catches.
  6. The lessonP(flagged | spam) = 90% in both weeks. P(spam | flagged) was 82%, then 27%. The two “given that” numbers are different questions, and the base rate links them.

The filter and the emails are made up. Counting out of 1,000 (“natural frequencies”) makes this much easier than multiplying percentages.

Ignoring the base rate is called base-rate neglect, and it is easy to fall into because the filter’s “90%” is the number people remember. The fix is to count: imagine 1,000 cases, split them by the base rate, then apply the catch rate and the false-alarm rate to each group. Researchers have found that people reason far better with counts like these than with percentages.

The same logic applies to any detector of rare things: fraud alerts, factory defect checks, AI-text detectors. A detector that is “95% accurate” can still be wrong about most of the cases it flags if what it looks for is rare. The ARQIV health threshold lab shows the same trade-off with a synthetic screening example.

Going deeper · step 4

When it says 70%, is it right 70% of the time?

A probability is only useful if it is honest. Checking that is called calibration.

A forecaster is calibrated if, across all the days it said 70%, it rained on about 70% of them, and the same for 10%, 40% or any other number. You cannot judge calibration from one day: a 70% forecast followed by a dry day is not wrong, because dry days are expected 3 times in 10. You need many forecasts, grouped by what they said.

Modern neural networks are often overconfident: researchers have found that their top-choice probabilities tend to be higher than their real accuracy, unless they are adjusted after training. And a chatbot’s confident tone is not a probability at all; fluent text sounds just as sure when it is wrong. How to evaluate AI shows calibration as part of testing a model.

Now check your own calibration. People are often overconfident too. How Sure Are You? asks ten true-or-false questions, records how sure you are, and draws your own calibration chart.

3 · Try it

Count the flags, then check the forecasts

Part 1 is about base rates; part 2 is about calibration. Both use only counting.

Lab

How many flagged emails are really spam?

Each square is one of 1,000 made-up emails. Move the sliders and watch how many flagged emails are really spam. Then compare two made-up weather apps.

Simulated in your browser with made-up emails and forecasts. Nothing is stored.

Part 1 · The frequency grid

  • spam, flagged
  • real, flagged (false alarm)
  • spam, missed
  • real, not flagged

Part 2 · Calibration check

Two made-up weather apps, each tested on its own made-up set of 200 days. Each dot shows, for one forecast number, how often it really rained. On the dashed line, the forecasts are honest.

Three challenges

  1. Keep the filter at 90% and 5%. Find the spam rate where a flagged email is about a 50–50 coin flip.
  2. At 2% spam, which helps more: raising the catch rate to 100%, or lowering the false-alarm rate to 1%?
  3. On which forecast does App B differ most from what happened? Would you trust its “90%”?

Part 1. A made-up spam filter catches 90% of spam and wrongly flags 5% of real emails. Here is what happens to 1,000 emails as spam becomes more or less common.

Same filter, five base rates (1,000 made-up emails)
Spam rateSpam emailsSpam caughtReal emails flaggedAll flaggedFlagged that are spam
2%2018496718 ÷ 67 = 27%
5%5045489345 ÷ 93 = 48%
10%100904513590 ÷ 135 = 67%
20%20018040220180 ÷ 220 = 82%
50%50045025475450 ÷ 475 = 95%

Part 2. Two made-up weather apps, each tested on its own made-up set of 200 days. A calibrated app’s “30%” days should be rainy about 30% of the time.

Forecast vs what happened
ForecastDaysApp A: rainy daysApp B: rainy days
10%303 (10%)9 (30%)
20%256 (24%)9 (36%)
30%206 (30%)8 (40%)
40%208 (40%)9 (45%)
50%157 (47%)8 (53%)
60%2012 (60%)10 (50%)
70%2517 (68%)14 (56%)
80%2520 (80%)15 (60%)
90%2018 (90%)13 (65%)

Correlation vs causation

Moving together is not causing

Machine learning finds patterns that go together. It cannot tell, on its own, which one causes which.

In the made-up beach town, weeks with more ice cream sold also had more sunburns. The two are strongly correlated. Does ice cream cause sunburn? No: both go up with a third thing, sunny hours. A hidden cause like this is called a confounder.

A model trained on this data could predict sunburns from ice cream sales quite well, and that is fine as long as nothing changes. But if the shop runs a half-price sale on a cloudy week, sales jump and the model predicts a wave of sunburns that never comes. Models learn what goes together in their data, not why. To find a cause you need more: an experiment where you change one thing on purpose, or careful reasoning about what else could explain the pattern.

The same trap appears in data and datasets: a model that learned a background or a watermark instead of the object itself. Train the Machine lets students find one.

Six made-up summer weeks in a beach town
WeekSunny hoursIce creams soldSunburns reported
Week 1353809
Week 2121503
Week 35052014
Week 4282908
Week 5202304
Week 64141012

Optional: the correlation coefficient r between ice creams and sunburns in this table is about 0.97 (1 would be a perfect straight-line match). A high r still says nothing about cause.

4 · Common mix-ups

Numbers that are easy to misread

Four ideas that trip people up, and what to think instead.

  • Mix-up: The filter catches 90% of spam, so a flagged email is 90% likely to be spam.

    Better: That swaps P(flagged | spam) for P(spam | flagged). The second depends on the base rate; it was 82% in a busy week and 27% in a quiet one.

  • Mix-up: The forecast said 70% rain and it stayed dry, so the forecast was wrong.

    Better: One day cannot show that. A calibrated 70% forecast should be dry on about 3 days in 10.

  • Mix-up: If two things rise together, one causes the other.

    Better: They may share a hidden cause, like sunshine for ice cream and sunburn, or the cause may run the other way.

  • Mix-up: An AI that says 99% is almost certainly right.

    Better: Only if it is calibrated. Many models are overconfident, and a chatbot’s confident wording is not a measured probability.

5 · Check yourself

Three quick questions

Answer in your head or on paper before you open each one.

  1. A made-up smoke alarm sounds for 95% of real fires and also for 10% of burnt-toast moments. In a school kitchen, 20 of 1,000 smoky moments are real fires. When the alarm sounds, what is the chance it is a real fire?

    Check your answer

    Fires: 95% of 20 = 19 alarms. Toast: 10% of 980 = 98 alarms. Total 19 + 98 = 117 alarms, so 19 ÷ 117 ≈ 16%. Most alarms are toast, because fires are rare. That does not make the alarm useless: missing a fire is far worse than a false alarm, so a high catch rate is the right choice here.

  2. A made-up weather app said “30% chance of rain” on 40 days, and it rained on 22 of them. Is it calibrated at 30%?

    Check your answer

    No. 22 ÷ 40 = 55%, far above 30%. On those days the app was too sure it would stay dry (underconfident about rain). With 40 days there is some luck involved, but a gap that large is a real warning sign.

  3. A made-up survey finds that students who use a homework app more often have higher grades. Does the app cause the higher grades?

    Check your answer

    Not necessarily. Students who already work hard may use the app more (a confounder: motivation), or good grades might make students enjoy using it (cause the other way round). A fairer test assigns the app to some students at random and compares results.

For grades 6–8

Count, don’t divide

The whole idea works with 100 squares on paper and no percentages.

20 minutes · pairs

The 100-email grid

  1. Draw a 10 × 10 grid: each square is an email. Color 10 squares red: they are spam.
  2. The filter catches 9 of the 10 spam emails: circle 9 red squares.
  3. It also flags 1 in every 10 real emails by mistake: circle 9 of the 90 white squares.
  4. Count the circles: 18. How many are really spam? 9. So a flagged email is spam half the time.

Talk about it

Why only half?

Good answers notice that there were many more real emails than spam, so even a few mistakes on them add up to as many circles as the real catches. Then try it with 30 red squares and see the answer change.

Honest limits

What this page leaves out

What this page leaves out

This page counts instead of using formulas. Bayes’ rule, which does the same calculation with probabilities, gets no space; nor do probability distributions for numbers (like the bell curve), uncertainty ranges on predictions, or scores for forecasts such as the Brier score (the How Sure Are You? lab explains and computes one for your own answers). Real calibration checks need many more cases than 200 days. Proving a cause usually needs experiments or careful causal methods, a field of its own.