Go deeper · Use, question, build

Who checks that an AI system is fair and safe?

People do, and no single test or label settles it. Before, during and after an AI system is used, someone should be able to answer five questions: what is it for, who does it affect, what evidence shows it works, where does it fail, and who can check, correct or stop it. Responsible AI is the habit of asking those questions with real numbers, and writing down who decided.

Grades 6+25 minutesPercentages (optional math marked)Discussion version for grades 3–5

What you will be able to do

Review an AI deployment with five questions (purpose, affected people, evidence, limitations, oversight), tell strong evidence from weak, and show with numbers how one accuracy figure can hide unequal mistakes between groups.

Before you start

Helpful: What did we teach it by accident? (dataset cards) and Good at what, for whom? (misses and false alarms).

1 · You have already seen this

The email that went to the wrong folder

Most people have had a message they needed end up in the spam folder.

A spam filter decides which emails you see first. Sometimes it hides a message you needed, and sometimes it lets junk through. Usually nobody notices. When the hidden message mattered, someone paid for the mistake, and it is fair to ask who tested the filter and who could fix it.

Asking those questions before and after an AI system is switched on is what responsible use of AI means on this page. It comes down to five questions about purpose, affected people, evidence, limits and oversight. This page shows how a review works, lets you try one on a made-up library app, and shows where it goes wrong.

2 · What happens, step by step

From a claim to a decision someone owns

A review in five steps. The next section turns steps 2 to 5 into five questions, and the worked example on fairness goes deeper into step 3. After the lab, a section on trade-offs goes deeper into steps 4 and 5.

How a review works

Five steps

  1. 1 · Start from the claimA seller or a headline says a tool is “safe” or “fair.” Treat that as a claim that still needs checking.
  2. 2 · Name the job and the peopleWrite down the one job the tool is for and what it must not be used for. Then list who feels its results, including people who never open the app.
  3. 3 · Ask for evidenceAsk how anyone knows it works for those people. The sections below show what makes a test strong.
  4. 4 · Write down limits and trade-offsNote where it may fail and for whom, and which kind of mistake you are choosing to accept more of.
  5. 5 · Keep someone in chargeName a person who checks it, a way for people to question a result, a date to test it again and a way to switch it off.

The five questions

Five questions, one habit

Ask them of a chatbot, a spam filter, a photo filter or a school app. They work before you buy something and after it is running.

A company that calls its product “safe” or “fair” has made a claim, not given evidence. The purpose tells you what to test, the affected people whom to test it for, the evidence whether it works for them, the limitations where it will not, and oversight who can notice a problem and act.

This page follows the structure of the AI Risk Management Framework from NIST, the U.S. National Institute of Standards and Technology (version 1.0, January 2023). It is voluntary guidance, not a law, and it groups the work into four functions: Map the context and the people affected, Measure how the system performs, Manage the risks you found, and Govern, which means deciding who is responsible. Our five questions are a classroom-sized version of the same idea.

Five questions for any AI system
QuestionA strong answer saysPractice it on ARQIV
Purpose: what is it for?One job, how success will be measured, and what it must not be used for.Map of AI: separate the task from the product
Affected people: who feels the results?The people being scored or shown content, not just the users, and groups the data may treat differently.Learn with AI (your own thinking and privacy count too)
Evidence: how do we know it works?Tests on new data from the real setting, against a baseline, with both kinds of mistakes, by group.Evaluating AI, Test the models, Verify lab
Limitations: where does it fail?Gaps in the data, groups it was not tested on, how it can be misused, and when the tests go out of date.Dataset cards, Model guide
Oversight: who can check, correct or stop it?A named person, a way for people to question a result, regular re-testing, and an off switch.Health lab: a person picks the threshold, and owns the trade-off

Predict first

An ad says, “Our homework chatbot is safe for schools.” Which of the five questions does that sentence answer?

Show the answer

None of them. It does not say what the chatbot is for, who was considered, what was tested, where it fails or who can stop it. It is a claim. A useful reply is a question: “Safe for which ages, tested how, and what does it get wrong?” The Test the models page shows how a class can gather its own evidence.

Going deeper · step 3

Same accuracy, different mistakes

Fairness questions become concrete when you count each kind of mistake for each group.

“Is it fair?” sounds like a yes-or-no question, but you have to choose what to measure. A tool makes two kinds of mistakes, and the people affected fall into groups: ages, schools, first languages, skin tones. One accuracy number blends all of that together.

Worked example

Same accuracy, different mistakes

  1. The toolA made-up screener flags students who might need extra reading help, so a teacher can check on them. A miss (a false negative) is a student who needs help and is not flagged. A false alarm (a false positive) is a student flagged who did not need it: one extra check.
  2. The groupsTwo groups of 100 students, A and B. They could be two schools or two age groups. In A, 20 students need help; in B, 40 do. These rates are invented for the example.
  3. AccuracyGroup A: 18 caught + 72 correctly not flagged = 90 right out of 100: 90%. Group B: 32 caught + 58 correctly not flagged = 90 of 100: 90%. Equal.
  4. Miss rateA: 2 missed of 20 who need help = 10%. B: 8 missed of 40 = 20%. A student who needs help is twice as likely to be missed in group B. The accuracy figure hid it, because B also had fewer false alarms (2 of 60 versus 8 of 80).
  5. A fixThe school adjusts the tool for group B (a lower threshold, so it flags more). Now B catches 36 and misses 4: a 10% miss rate, and 6 of 60 false alarms: 10%. Accuracy is still 90% in both groups.
  6. A new gapAsk what a flag means. In A, 18 of the 26 flagged students really need help: about 69%. In B, 36 of 42: about 86%. And B now has 42 students flagged against A’s 26. Equal miss rates, unequal meaning of a flag.

Optional math: miss rate = missed ÷ (caught + missed). “Flag is right” is precision = caught ÷ (caught + false alarms). Accuracy = (caught + correctly not flagged) ÷ everyone.

A made-up reading screener, 200 students
VersionGroupNeed helpCaughtMissedFalse alarmsAccuracyMiss rateFlag is right
1 · as soldA20 of 1001828 of 8090%2 ÷ 20 = 10%18 ÷ 26 ≈ 69%
1 · as soldB40 of 1003282 of 6090%8 ÷ 40 = 20%32 ÷ 34 ≈ 94%
2 · adjustedA20 of 1001828 of 8090%10%18 ÷ 26 ≈ 69%
2 · adjustedB40 of 1003646 of 6090%4 ÷ 40 = 10%36 ÷ 42 ≈ 86%

Predict first

In version 2 the miss rates and false-alarm rates are equal for both groups. Could the school also make a flag equally trustworthy in both groups, without changing the rates again?

Show the answer

No. Group B has more students who need help (40 versus 20), so with the same error rates, a flag in group B is more often right. To make a flag mean the same thing in both groups, the school would have to make the error rates unequal again. Researchers have shown that when groups have different base rates and a tool makes any mistakes, these goals generally cannot all be met at once (Kleinberg and colleagues, 2016). Fairness definitions can conflict, so someone has to choose.

Three ways to define fair, one tool
Fairness goalIn plain wordsVersion 2 numbers
Equal miss and false-alarm ratesPeople who need help are caught equally often in each group, and people who do not are bothered equally oftenMet: 10% and 10% in both groups
Equal meaning of a flagA flag is right equally often in each groupNot met: about 69% in A, 86% in B
Equal flag ratesEach group is flagged at the same rateNot met: 26% in A, 42% in B

Which goal is right depends on the purpose and on who pays for each mistake. Here a miss leaves a student without help, while a false alarm costs one extra check, so equal miss rates is a sensible first goal. That is a judgment about values, not a math result, which is why the five questions start with purpose and affected people.

3 · Try it

AI decision review

A made-up deployment, five questions, three answers each. Pick the strongest one and read why.

Lab

Review the Late-Return Reminder

Read the deployment. For each question, choose the answer you would want in the school’s decision record. The feedback explains what makes an answer, and especially evidence, strong or weak. Then show your decision record.

No AI runs here. The deployment and feedback are written by ARQIV, and nothing you choose is stored.

The deployment · made up

A school library plans to buy a made-up app, the Late-Return Reminder. It gives every new loan a “late risk” score from 0 to 100. Loans scoring above 70 get a friendly email two days before the due date. The seller trained it on the library’s 1,240 loans from last school year, 100 of which came back late, and says it is “92% accurate.” The head teacher also asks whether students with high scores could be limited to two books at a time.

1 of 5 · Purpose

What should the school write down as the purpose?

2 of 5 · Affected people

Whose experience should the review ask about?

3 of 5 · Evidence

Which evidence would convince you that it works well enough?

4 of 5 · Limitations

Which limitation is most important to write down?

5 of 5 · Oversight

Who can check, correct or stop it?

The interactive version needs JavaScript. Read the made-up deployment, then each question with its three answers and what each one is worth. Cover the italic part and judge each answer yourself first.

A school library plans to buy a made-up app, the Late-Return Reminder. It gives every new loan a “late risk” score from 0 to 100. Loans scoring above 70 get a friendly email two days before the due date. The seller trained it on the library’s 1,240 loans from last school year, 100 of which came back late, and says it is “92% accurate.” The head teacher also asks whether students with high scores could be limited to two books at a time.

  1. 1. Purpose. What should the school write down as the purpose?
    • Weak: Use AI to modernize the library. Weak. This names a technology, not a job. Nobody can test whether something “modernizes” a library, so nobody can tell later whether it worked.
    • Strong: Send a friendly reminder before the due date on loans likely to be late. Not for limiting who may borrow. Strong. One job, a result you can measure (fewer late books), and a clear “not for.” Limiting borrowing is a different purpose with much higher stakes for students, so it would need its own review.
    • Partly there: Reduce late returns now, and later decide which students may borrow fewer books. Partly. The first half is a clear job. The second half quietly adds a high-stakes use, a step called purpose creep. A reminder that is wrong costs one email; a limit that is wrong costs a student their books.
  2. 2. Affected people. Whose experience should the review ask about?
    • Partly there: The librarian, who will use the app every day. Partly. Users matter, but the people most affected are the ones being scored, and they never open the app.
    • Weak: Nobody really. It only sends emails. Weak. Even a reminder affects people: who gets nagged, who gets missed, and what the email reveals to a family that reads it.
    • Strong: Every borrower, especially anyone wrongly flagged or missed; younger students who share one class card; families who read the emails; and the librarian. Strong. It names the people being scored, the groups the data may treat differently, and people outside the app who still feel its effects.
  3. 3. Evidence. Which evidence would convince you that it works well enough?
    • Weak: The seller’s figure: “92% accurate.” Weak evidence. 100 of the 1,240 loans were late, so an app that reminds nobody is right about the other 1,140: 1,140 ÷ 1,240 ≈ 92%. A number with no baseline, no test on new data and no breakdown by group tells you almost nothing.
    • Partly there: A survey: 9 out of 10 students liked getting the reminder emails. Partly. It answers a different question (are reminders welcome?), which is worth knowing. It says nothing about how many late loans the app catches or whom it misses.
    • Strong: A test on this term’s loans, which the app never saw: late loans caught and on-time borrowers bothered, compared with reminding nobody and reminding everyone, broken down by grade. Strong evidence. New data from the place it will be used, a baseline to beat, both kinds of mistakes counted, and results for each group, so an average cannot hide a group it fails.
  4. 4. Limitations. Which limitation is most important to write down?
    • Strong: It learned from one school year with no summer loans; grades 1 and 2 share one class card, so their scores mean little; and past late returns follow a student into every new score. Strong. Each point says where the scores may be wrong or unfair and for whom, which tells the school what to watch.
    • Weak: It was trained on real data, so it has no real limitations. Weak. Real data carries real gaps: missing months, shared cards, old habits. Every model has limits; a good review writes them down.
    • Partly there: Some emails may land in spam folders. Partly. A real practical limit, but it is about delivery, not about whether the scores are right or fair.
  5. 5. Oversight. Who can check, correct or stop it?
    • Partly there: Students can unsubscribe from all library emails. Partly. An exit is good, but it is not oversight: nobody is checking whether the scores are right, and the people it misses never get an email to unsubscribe from.
    • Strong: The librarian checks a sample of flags each month, students can ask why they got a reminder, the school re-tests it every year, and one named person can switch it off. Strong. Someone checks, people can question a result, the test is repeated as things change, and one person is accountable for stopping it.
    • Weak: The app monitors itself and improves automatically. Weak. A system checking itself is not an independent check, and “improves automatically” means it changes without anyone reviewing the change.

Trade-offs

Accountable decisions, written down

Every AI deployment trades something for something else. Responsible use means making the trade on purpose, in the open.

Catching more cases means more false alarms; collecting less personal data protects privacy but can lower accuracy; a person checking every result is safer but slower. None of these has one correct answer. A decision is accountable when a named person made it with evidence, wrote down why, and set a date to check again.

Documentation makes that possible. A dataset card (from the proposal Datasheets for Datasets) records where data came from and what it may be used for; ARQIV’s dataset card shows one for a library. A model card (Mitchell and colleagues) does the same for a trained model: intended use, results broken down by group, and known limits.

For generated images, video and audio, the matching question is provenance: where did this file come from? The C2PA standard attaches Content Credentials, a signed record of how a file was made and edited. They help when present, but they can be stripped, so a missing label proves nothing. See Is it real? for deepfakes and watermarks.

A one-page decision record

  1. Decision: what will be used, for what, starting when.
  2. Not for: uses that are ruled out.
  3. Evidence: the test, its baseline and results by group.
  4. Trade-off chosen: which mistake we accept more of, and why.
  5. Limits: gaps and groups not tested.
  6. Oversight: who checks, how people question a result, the off switch.
  7. Owner and review date: one name, one date.

4 · Common mix-ups

What people get wrong about fairness and checks

Four ideas that trip people up, and what to think instead.

  • Mix-up: Fair means the same accuracy for every group.

    Better: Accuracy can be equal while one group is missed twice as often, as in the worked example. Count each kind of mistake for each group.

  • Mix-up: Removing the column about a group makes a model fair.

    Better: Other columns can stand in for it, and without it you cannot even measure the gap. The only way to know whether a model is fair is to measure it.

  • Mix-up: Responsible AI is a checklist you complete once.

    Better: Data, people and uses change after launch. Oversight means re-testing and being able to stop the system.

  • Mix-up: A Content Credentials label proves an image is real, and no label proves it is fake.

    Better: Credentials record a file’s history when they are present. They can be stripped, so a missing label proves nothing either way.

5 · Check yourself

Three quick questions

Answer in your head or on paper before you open each one.

  1. A sports club’s app picks players for extra coaching. It is 85% accurate for both the under-12 and the under-14 team. What should you ask before calling it fair?

    Check your answer

    How each kind of mistake is spread across the teams: how many players who would benefit are missed in each team (miss rate), and how many are picked who did not need it. Equal accuracy can hide a much higher miss rate for one team.

  2. A city’s bus-route app is said to “save 10 minutes per trip.” Name one piece of weak evidence and one piece of strong evidence for that claim.

    Check your answer

    Weak: the seller’s figure, or a test on a different city. Strong: trip times measured in this city before and after, compared with the old timetable, and broken down by neighborhood, so you can see whether some riders now wait longer.

  3. What do NIST’s four functions Map, Measure, Manage and Govern have to do with the five questions?

    Check your answer

    Map covers purpose and affected people, Measure covers evidence, Manage covers limitations and what to do about them, and Govern covers oversight: who is responsible and who can stop the system.

For younger learners

A grades 3–5 version

No percentages: one story and one question, done together.

15 minutes · teacher-led

Who does it affect?

  1. Tell the story: “A robot helper in the library sends a reminder to anyone it thinks will bring a book back late.”
  2. Ask: who could be happy about this, and who could be upset? (Someone reminded who is never late; someone who needed a reminder and did not get one.)
  3. Ask: who should be allowed to check the robot, and switch it off?

Exit question

Say it in one sentence

“Who checks an AI?” Good answers name a person and an action: “The librarian looks at who got reminders and can turn it off.”

Honest limits

What this page leaves out

What this page leaves out

Laws and regulations (they differ by country and change often), security attacks such as data poisoning, the environmental cost of computing, workers who label data, and how large organizations run reviews. Fairness research also has more definitions and methods than the three shown here, and choosing among them is still debated.