Go deeper · Advanced track

How can AI systems be attacked, and how do we trace what they made?

Through the examples it learns from, the inputs it is shown and the tools it may use. Defenders answer with careful data handling, testing, narrow permissions and people who try to break the system first. To trace what AI made, some files carry a signed history called Content Credentials. They help when present, but a missing record proves nothing.

Grades 9+30 minutesAverages (optional math marked)Discussion version for grades 6–8

An optional advanced page: the main route does not need it. It explains attacks in principle and how defenders respond, with no attack instructions.

What you will be able to do

Explain how data poisoning, adversarial examples and prompt injection work, match each to a defense, and read a Content Credentials check, including what a broken or missing record does and does not prove.

Before you start

Read first: What changes when an AI can take actions? (prompt injection and permissions), Who checks that an AI system is fair and safe? and What did we teach it by accident?

1 · You have already seen this

A photo in the group chat, and nobody knows who took it

A dramatic photo gets shared, and someone asks “Is this even real?”

Looking harder at the picture rarely settles it. What you want is its history: who took it, with what, and whether anyone changed it since. Similar questions apply to the AI systems behind many apps. Someone could have slipped bad examples into what a system learned from, or planted a line of text for it to read.

This page is about how AI systems can be attacked and how people trace what AI made. Attacks can land on the examples a system learns from, the inputs it is shown and the tools it may use, and each needs its own defense. The page shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

Where attacks land, step by step

An AI system from learning to the files it sends out, in five steps. A short story comes first and touches all of them. Then four sections go deeper into steps 1 to 4, and the lab lets you check step 5 yourself.

How a system is attacked and defended

Five steps

  1. 1 · Learn from examplesThe system learns a rule from labeled examples. Whoever can change the examples can change the rule, which is called data poisoning.
  2. 2 · Answer new inputsOnce it is in use, someone can change what it is shown. A small change made on purpose, such as stickers on a stop sign, can flip the answer.
  3. 3 · Act with toolsA helper that can order or send things reads its instructions and web pages as one stream of text, so a planted line can look like an order. Limits written in ordinary code decide how much a fooled helper can do.
  4. 4 · Attack it firstA team with permission tries to make the system fail at each step. What they find becomes fixes and new tests.
  5. 5 · Trace what it madeFiles go out into the world. Some carry Content Credentials, a signed record of the tool and the edits, and the check fails if the record or the picture is changed afterward.

One story, every step

A juice stand, four problems

Before any vocabulary: one made-up school fair and one made-up AI system.

A school juice stand uses a camera sorter: lemons go to the lemonade line, apples to the apple-juice line. Volunteers taught it with labeled photos, and an AI helper orders supplies online. In one week, four things go wrong.

1. A prankster labeled a few lemon photos “apple,” so some lemons end up in the apple juice. 2. A small sticker on a lemon makes the camera call it an apple. 3. A supplier’s page hides a line telling the helper to order 50 extra crates. 4. A photo of a moldy lemon “from the stand” spreads online. Who really took it?

Each problem hits a different stage of the system, so each needs a different defense. MITRE ATLAS, a public knowledge base, collects real attacks on AI systems. The OWASP Top 10 for LLM Applications (2025 list) names the biggest risks for chatbot and agent apps, including prompt injection, data and model poisoning, and “excessive agency.”

Four stages, four kinds of attack
Where the attack landsThe attack, in principleMain defenses
Training dataData poisoning: bad or mislabeled examples slipped in, so the model learns the wrong ruleKnow where data came from, check samples of labels, keep a trusted test set, watch for mistakes that cluster
Input at use timeAdversarial example: a small, deliberate change to an input that flips the answerTest robustness, train on altered examples, never let one model’s answer alone trigger something important
Tools and actionsPrompt injection and too much permission: planted text steers an agent that can actLeast privilege, approvals, checks in ordinary code, logs
Content out in the worldFakes, or real files nobody can traceContent Credentials, watermarks, and checking the source

Going deeper · step 1

Data poisoning: bad examples in, bad rule out

A model learns whatever its examples say. Whoever can change the examples can change the rule.

The made-up sorter reads one yellowness score per fruit. It averages the apples, averages the lemons and puts its dividing line halfway between: a tiny version of how every model summarizes its examples.

Data poisoning means slipping bad examples into training data on purpose; flipping labels is the simplest kind. The risk grows when data comes from the open web or from anyone who uploads, because nobody checks every example. Targeted poisoning barely moves the overall score and breaks only the cases the attacker cares about.

Worked example

One flipped label out of ten

  1. The dataMade-up yellowness scores from 0 (deep red) to 10 (bright yellow). Apples: 1, 2, 3, 3, 4. Lemons: 6, 7, 8, 9, 9.
  2. Clean ruleApple average = (1 + 2 + 3 + 3 + 4) ÷ 5 = 13 ÷ 5 = 2.6. Lemon average = (6 + 7 + 8 + 9 + 9) ÷ 5 = 39 ÷ 5 = 7.8. Dividing line = (2.6 + 7.8) ÷ 2 = 5.2. Below 5.2 means apple.
  3. TestSix new fruits a careful person labeled: apples at 2, 3.5 and 4.5; lemons at 5.5, 6.5 and 8.5. All six sort correctly: 6 of 6.
  4. PoisonSomeone relabels the lemon at 6 as an apple. Apples: (13 + 6) ÷ 6 = 19 ÷ 6 ≈ 3.17. Lemons: (39 − 6) ÷ 4 = 33 ÷ 4 = 8.25. New line = (3.17 + 8.25) ÷ 2 ≈ 5.71.
  5. DamageThe lemon at 5.5 now falls below the line: apple juice. The test score drops only to 5 of 6 (about 83%); every other fruit sorts as before.

Optional math: dividing line = (apple average + lemon average) ÷ 2. All numbers are made up.

Predict first

A careful person tests the poisoned sorter on 100 random fruits. Will they notice that something is wrong?

Show the answer

Probably not. Only fruits scoring between 5.2 and about 5.71 changed sides. If few real fruits fall in that narrow band, the score barely moves. So defenders also check where data came from, review labels near the line and watch for complaints about one kind of case.

Lab

Flip the labels, move the line

Flip one or two training labels and watch the dividing line and the six new fruits. Find out which flips do damage right away and which ones only move the line.

Simulated in your browser with a tiny made-up example

Each button is one training photo. Press it to flip its label (pressed = label changed). The sorter re-learns its dividing line at once.

Circles: training photos (A = labeled apple, L = labeled lemon, dashed ring = label changed). Squares: six new fruits a careful person labeled, ✓ sorted right, ✗ sorted wrong. Vertical line: the dividing line.

The interactive version needs JavaScript. Here are the made-up data and what happens when labels are flipped.

Training photos (yellowness score): apples A1 = 1, A2 = 2, A3 = 3, A4 = 3, A5 = 4; lemons L1 = 6, L2 = 7, L3 = 8, L4 = 9, L5 = 9. New fruits: apples at 2, 3.5 and 4.5; lemons at 5.5, 6.5 and 8.5.

Flipping a few labels moves the line
Labels flippedApple averageLemon averageDividing lineNew fruits right
None (clean)13 ÷ 5 = 2.639 ÷ 5 = 7.85.26 of 6
L1 → apple19 ÷ 6 ≈ 3.1733 ÷ 4 = 8.25≈ 5.715 of 6 (lemon at 5.5 goes to apples)
L1 and L2 → apple26 ÷ 7 ≈ 3.7126 ÷ 3 ≈ 8.67≈ 6.195 of 6 (lemon at 5.5 goes to apples)
A5 → lemon9 ÷ 4 = 2.2543 ÷ 6 ≈ 7.17≈ 4.716 of 6 (line moved, no visible damage yet)
A4 and A5 → lemon6 ÷ 3 = 246 ÷ 7 ≈ 6.57≈ 4.295 of 6 (apple at 4.5 goes to lemons)

Defenses start before training: record where every dataset came from (a dataset card), limit who can add examples, have a second person check a sample of labels, and keep a trusted test set that outsiders cannot touch.

Going deeper · step 2

Adversarial examples: tiny change, different answer

Nobody touches the training. Someone changes what the model is shown.

In 2014, Ian Goodfellow and colleagues showed an image classifier a photo of a panda, then added a pattern too faint for people to notice. The model now answered “gibbon,” and was more confident than before. An input changed on purpose so that a model gives a wrong answer is called an adversarial example.

A photo is thousands of numbers, and the model adds up a little evidence from each. Changes too small to see in any one pixel can add up across all of them (see the optional math below). Physical changes can work too: in 2017, researchers made a sign-reading model misread stop signs that carried stickers.

Defenders train on altered examples, test robustness before launch, and never let one model’s answer alone trigger something important: a second sensor, a rule check or a person comes too. No known defense makes a large image model fully robust; this is an open research problem.

Going deeper · step 3

Safe tool boundaries: limit what a fooled model can do

You met prompt injection on the agents page. The most reliable defense does not depend on the model resisting it.

A language model reads its instructions and the pages it retrieves as one stream of text, so the planted line can look like an order. OWASP and other guides say no known method prevents prompt injection completely. The main protection sits outside the model, in ordinary code it cannot talk its way past. The opposite, more tools or permissions than the job needs, is what OWASP calls excessive agency.

Safe tool boundaries for a made-up helper
BoundaryFor the juice-stand helperWhat it protects
Least privilege: only the tools and access the job needsMay order lemons, apples and cups from two named suppliers, up to $40 a dayA fooled step cannot order 50 crates or pay a stranger
Approval before anything that cannot be undoneA person taps “yes” before any paymentCatches an odd request, if the person actually reads it
Instructions apart from dataSupplier pages are passed in as quoted information, never as instructionsLowers the chance a planted line is obeyed; does not remove it
Treat model output as untrustedThe order form checks items and quantities in ordinary code before sendingA bad request never reaches the payment system unchecked
Logs and budgetsEvery tool call is recorded; a step limit stops runaway loopsPeople can find, explain and undo a problem

Predict first

The helper is fooled by the planted line and asks to order 50 crates. Which boundary in the table stops it with no person involved?

Show the answer

Least privilege, enforced in ordinary code: 50 crates break the $40-a-day limit, so the order tool refuses. Approval helps only if the person reads the request instead of tapping “yes” out of habit.

Going deeper · step 4

Red-teaming: break it before others do

Finding weaknesses is a planned job, done with permission.

Red-teaming means a team with permission deliberately tries to make a system fail: mislabeled data, tricky inputs, planted instructions. Findings become fixes and new tests, repeated after every big change. The NIST AI Risk Management Framework (2023) lists “secure and resilient” among the qualities of trustworthy AI.

Testing systems you do not own, without being asked, can break school rules and laws. In class, attack your own toy.

3 · Try it

Provenance: where did this file come from?

Back to the moldy-lemon photo. Staring harder at the pixels is not enough; you want the file’s history.

Provenance is the documented history of a piece of content: who or what made it, and what was done to it since. The C2PA standard lets a camera, editing app or image generator attach Content Credentials: a record of the tool, the editing steps and a fingerprint of the content, sealed with a digital signature. Generators can use it to say “made with AI.”

The signature works like a tamper-evident seal: change the record or the picture afterward and the check fails. But credentials are easy to remove (a screenshot will do), and most real photos never had any, so a missing record proves nothing. A valid one shows who signed and that nothing changed since, not that the scene was honest. Watermarks, hidden signals a generator mixes into pixels, sound or text, survive some edits but weaken under heavy editing or rewording. AI detectors that guess from content alone are unreliable (see Think first).

Lab

Check a toy Content Credentials record

Press “Check credentials” first. Then change the date in the record, brighten a pixel in an ordinary app or a credential-aware one, or remove the credentials, and check again each time. Reset between experiments.

A toy in your browser. Real Content Credentials use cryptographic signatures; this short checksum only imitates the idea. Nothing is uploaded.

The picture · 3 × 3 pixels, made up

The attached record · toy Content Credentials

History (each step signed):

    The interactive version needs JavaScript. The toy record says the picture was made with a made-up camera on 14 May 2026, and it stores a fingerprint of the pixels, sealed with a signature. Here is what a checker reports in five cases.

    Five cases for one toy record
    What happenedRecord signaturePicture fingerprintWhat it tells you
    Nothing (as captured)MatchesMatchesWho signed, and that nothing changed since. Not that the scene is honest.
    Someone typed a new date into the recordDoes not matchMatchesThe record was changed after signing. Trust nothing in it.
    One pixel brightened in an ordinary appMatchesDoes not matchThe record is genuine but describes an earlier version of the picture.
    One pixel brightened in a credential-aware appMatchesMatchesThe app checked the old record, then added its own signed step.
    Credentials removed by a screenshotNoneNoneNothing either way. Most real photos have no record at all.

    Predict first

    The moldy-lemon photo carries valid Content Credentials signed by a phone maker. Does that prove the stand sold moldy lemons?

    Show the answer

    No. It shows a phone from that maker captured the picture and nothing changed since. The lemon could be from anywhere, with a false caption. Whether the claim is true takes the SIFT habits from the Verify lab: check who posted it and look for other sources.

    Going further · optional math

    Why tiny changes add up

    For older students: the arithmetic behind adversarial examples, with made-up numbers.

    Worked example

    Ten thousand tiny nudges

    1. One numberThe juice-stand sorter reads a single score. To push the lemon at 5.5 across the 5.2 line, the score must change by more than 0.3 on a 0 to 10 scale: 0.3 ÷ 10 = 3% of the range, a change in color you could see.
    2. Many numbersNow a made-up image model reads a 100 × 100 gray picture: 10,000 pixel values from 0 to 255. Its score adds up (weight × pixel) over every pixel, and a score above 0 means “lemon.” Say each weight is small: about 0.001 in size, ignoring its sign.
    3. A tiny changeShift every pixel by just 2 brightness levels (2 ÷ 255 ≈ 0.8% of the range), each one in whichever direction lowers the score. Each pixel moves the score by about 2 × 0.001 = 0.002.
    4. It adds up10,000 × 0.002 = 20. A picture that scored +5 (a confident lemon) now scores 5 − 20 = −15: a confident apple. No single pixel changed enough to see.

    This is the “linear explanation” Goodfellow and colleagues gave in 2014: in pictures with many numbers, tiny changes add up. Real models are not purely linear, and real defenses are more involved. All numbers are made up.

    4 · Common mix-ups

    Four beliefs about AI attacks that do not hold up

    Each one comes with a better way to think about it.

    • Mix-up: AI is attacked by hacking into the computer.

      Better: Many attacks need no break-in at all. They change training data, inputs or retrieved text, and the system does exactly what it was built to do with them.

    • Mix-up: A good overall score means the model was not poisoned.

      Better: Targeted poisoning can leave the score almost unchanged, as in the worked example. Check data sources and look at where the mistakes cluster.

    • Mix-up: No credentials means fake; valid credentials mean true.

      Better: Missing credentials prove nothing, because they are easy to strip. Valid ones show who signed and that nothing changed since, not that the scene is honest.

    • Mix-up: A smarter model will simply learn to ignore prompt injection.

      Better: Models may get better at it, but no known method stops it completely. Put limits in permissions and in checks outside the model.

    5 · Check yourself

    Three quick questions

    Answer in your head or on paper before you open each one.

    1. A homework-help app keeps learning from answers that any user can submit and upvote. Which attack does that invite, and what are two defenses?

      Check your answer

      Data poisoning: anyone can add bad examples. Defenses: limit or review who can contribute, check a sample of new examples before training, keep a trusted test set nobody outside can edit, and watch for mistakes that cluster on one topic.

    2. An AI helper can read the class calendar and email anyone. It only needs to remind your class about tests. What would you change first, and why?

      Check your answer

      Least privilege: allow emails only to the class list, enforced in ordinary code, and require approval for anything else. Prompt injection cannot be fully prevented, so limit what a fooled helper is able to do.

    3. A video of a principal announcing a snow day has no Content Credentials. A classmate says, “No credentials, so it’s fake.” What do you reply?

      Check your answer

      A missing record proves nothing: most videos have none, and re-uploading often strips them. Check the source instead, such as the school’s own website or messages, before you believe or share it.

    For younger learners

    A grades 6–8 version

    No averages needed: one card game and one envelope activity.

    20 minutes · groups of three

    The swapped card

    1. Each group gets ten fruit cards with a yellowness number and a label, and lays them on a number line drawn on paper.
    2. Groups draw their dividing line by eye. Secretly, the teacher has swapped one label in one group’s deck.
    3. All groups sort the same six new fruit cards and compare. Which group got one wrong? Can they find the bad card?

    10 minutes · whole class

    Three envelopes

    Show an envelope with an unbroken seal sticker, one with a torn seal, and one that never had a seal. Ask what each one proves. The torn seal shows tampering; the unsealed one proves nothing either way. That is how Content Credentials work.

    Exit question: “What does a missing seal tell you?” Good answers: “Nothing on its own. Check where it came from.”

    Honest limits

    What this page leaves out

    What this page leaves out

    Privacy attacks (coaxing a model to reveal its training data, or guessing whether someone’s data was used), copying a model by querying it, attacks on the software and model supply chain, the technical details of signatures and certificates, and laws about deepfakes, which differ by country and change often.