Go deeper · Advanced track
How can AI systems be attacked, and how do we trace what they made?
Through the examples it learns from, the inputs it is shown and the tools it may use. Defenders answer with careful data handling, testing, narrow permissions and people who try to break the system first. To trace what AI made, some files carry a signed history called Content Credentials. They help when present, but a missing record proves nothing.
An optional advanced page: the main route does not need it. It explains attacks in principle and how defenders respond, with no attack instructions.
What you will be able to do
Explain how data poisoning, adversarial examples and prompt injection work, match each to a defense, and read a Content Credentials check, including what a broken or missing record does and does not prove.
Before you start
Read first: What changes when an AI can take actions? (prompt injection and permissions), Who checks that an AI system is fair and safe? and What did we teach it by accident?
1 · You have already seen this
A photo in the group chat, and nobody knows who took it
A dramatic photo gets shared, and someone asks “Is this even real?”
Looking harder at the picture rarely settles it. What you want is its history: who took it, with what, and whether anyone changed it since. Similar questions apply to the AI systems behind many apps. Someone could have slipped bad examples into what a system learned from, or planted a line of text for it to read.
This page is about how AI systems can be attacked and how people trace what AI made. Attacks can land on the examples a system learns from, the inputs it is shown and the tools it may use, and each needs its own defense. The page shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
Where attacks land, step by step
An AI system from learning to the files it sends out, in five steps. A short story comes first and touches all of them. Then four sections go deeper into steps 1 to 4, and the lab lets you check step 5 yourself.
How a system is attacked and defended
Five steps
- 1 · Learn from examplesThe system learns a rule from labeled examples. Whoever can change the examples can change the rule, which is called data poisoning.
- 2 · Answer new inputsOnce it is in use, someone can change what it is shown. A small change made on purpose, such as stickers on a stop sign, can flip the answer.
- 3 · Act with toolsA helper that can order or send things reads its instructions and web pages as one stream of text, so a planted line can look like an order. Limits written in ordinary code decide how much a fooled helper can do.
- 4 · Attack it firstA team with permission tries to make the system fail at each step. What they find becomes fixes and new tests.
- 5 · Trace what it madeFiles go out into the world. Some carry Content Credentials, a signed record of the tool and the edits, and the check fails if the record or the picture is changed afterward.
One story, every step
A juice stand, four problems
Before any vocabulary: one made-up school fair and one made-up AI system.
A school juice stand uses a camera sorter: lemons go to the lemonade line, apples to the apple-juice line. Volunteers taught it with labeled photos, and an AI helper orders supplies online. In one week, four things go wrong.
1. A prankster labeled a few lemon photos “apple,” so some lemons end up in the apple juice. 2. A small sticker on a lemon makes the camera call it an apple. 3. A supplier’s page hides a line telling the helper to order 50 extra crates. 4. A photo of a moldy lemon “from the stand” spreads online. Who really took it?
Each problem hits a different stage of the system, so each needs a different defense. MITRE ATLAS, a public knowledge base, collects real attacks on AI systems. The OWASP Top 10 for LLM Applications (2025 list) names the biggest risks for chatbot and agent apps, including prompt injection, data and model poisoning, and “excessive agency.”
| Where the attack lands | The attack, in principle | Main defenses |
|---|---|---|
| Training data | Data poisoning: bad or mislabeled examples slipped in, so the model learns the wrong rule | Know where data came from, check samples of labels, keep a trusted test set, watch for mistakes that cluster |
| Input at use time | Adversarial example: a small, deliberate change to an input that flips the answer | Test robustness, train on altered examples, never let one model’s answer alone trigger something important |
| Tools and actions | Prompt injection and too much permission: planted text steers an agent that can act | Least privilege, approvals, checks in ordinary code, logs |
| Content out in the world | Fakes, or real files nobody can trace | Content Credentials, watermarks, and checking the source |
Going deeper · step 1
Data poisoning: bad examples in, bad rule out
A model learns whatever its examples say. Whoever can change the examples can change the rule.
The made-up sorter reads one yellowness score per fruit. It averages the apples, averages the lemons and puts its dividing line halfway between: a tiny version of how every model summarizes its examples.
Data poisoning means slipping bad examples into training data on purpose; flipping labels is the simplest kind. The risk grows when data comes from the open web or from anyone who uploads, because nobody checks every example. Targeted poisoning barely moves the overall score and breaks only the cases the attacker cares about.
Worked example
One flipped label out of ten
- The dataMade-up yellowness scores from 0 (deep red) to 10 (bright yellow). Apples: 1, 2, 3, 3, 4. Lemons: 6, 7, 8, 9, 9.
- Clean ruleApple average = (1 + 2 + 3 + 3 + 4) ÷ 5 = 13 ÷ 5 = 2.6. Lemon average = (6 + 7 + 8 + 9 + 9) ÷ 5 = 39 ÷ 5 = 7.8. Dividing line = (2.6 + 7.8) ÷ 2 = 5.2. Below 5.2 means apple.
- TestSix new fruits a careful person labeled: apples at 2, 3.5 and 4.5; lemons at 5.5, 6.5 and 8.5. All six sort correctly: 6 of 6.
- PoisonSomeone relabels the lemon at 6 as an apple. Apples: (13 + 6) ÷ 6 = 19 ÷ 6 ≈ 3.17. Lemons: (39 − 6) ÷ 4 = 33 ÷ 4 = 8.25. New line = (3.17 + 8.25) ÷ 2 ≈ 5.71.
- DamageThe lemon at 5.5 now falls below the line: apple juice. The test score drops only to 5 of 6 (about 83%); every other fruit sorts as before.
Optional math: dividing line = (apple average + lemon average) ÷ 2. All numbers are made up.
Predict first
A careful person tests the poisoned sorter on 100 random fruits. Will they notice that something is wrong?
Show the answer
Probably not. Only fruits scoring between 5.2 and about 5.71 changed sides. If few real fruits fall in that narrow band, the score barely moves. So defenders also check where data came from, review labels near the line and watch for complaints about one kind of case.
Lab
Flip the labels, move the line
Flip one or two training labels and watch the dividing line and the six new fruits. Find out which flips do damage right away and which ones only move the line.
Simulated in your browser with a tiny made-up example
Each button is one training photo. Press it to flip its label (pressed = label changed). The sorter re-learns its dividing line at once.
Circles: training photos (A = labeled apple, L = labeled lemon, dashed ring = label changed). Squares: six new fruits a careful person labeled, ✓ sorted right, ✗ sorted wrong. Vertical line: the dividing line.
The interactive version needs JavaScript. Here are the made-up data and what happens when labels are flipped.
Training photos (yellowness score): apples A1 = 1, A2 = 2, A3 = 3, A4 = 3, A5 = 4; lemons L1 = 6, L2 = 7, L3 = 8, L4 = 9, L5 = 9. New fruits: apples at 2, 3.5 and 4.5; lemons at 5.5, 6.5 and 8.5.
| Labels flipped | Apple average | Lemon average | Dividing line | New fruits right |
|---|---|---|---|---|
| None (clean) | 13 ÷ 5 = 2.6 | 39 ÷ 5 = 7.8 | 5.2 | 6 of 6 |
| L1 → apple | 19 ÷ 6 ≈ 3.17 | 33 ÷ 4 = 8.25 | ≈ 5.71 | 5 of 6 (lemon at 5.5 goes to apples) |
| L1 and L2 → apple | 26 ÷ 7 ≈ 3.71 | 26 ÷ 3 ≈ 8.67 | ≈ 6.19 | 5 of 6 (lemon at 5.5 goes to apples) |
| A5 → lemon | 9 ÷ 4 = 2.25 | 43 ÷ 6 ≈ 7.17 | ≈ 4.71 | 6 of 6 (line moved, no visible damage yet) |
| A4 and A5 → lemon | 6 ÷ 3 = 2 | 46 ÷ 7 ≈ 6.57 | ≈ 4.29 | 5 of 6 (apple at 4.5 goes to lemons) |
Defenses start before training: record where every dataset came from (a dataset card), limit who can add examples, have a second person check a sample of labels, and keep a trusted test set that outsiders cannot touch.
Going deeper · step 2
Adversarial examples: tiny change, different answer
Nobody touches the training. Someone changes what the model is shown.
In 2014, Ian Goodfellow and colleagues showed an image classifier a photo of a panda, then added a pattern too faint for people to notice. The model now answered “gibbon,” and was more confident than before. An input changed on purpose so that a model gives a wrong answer is called an adversarial example.
A photo is thousands of numbers, and the model adds up a little evidence from each. Changes too small to see in any one pixel can add up across all of them (see the optional math below). Physical changes can work too: in 2017, researchers made a sign-reading model misread stop signs that carried stickers.
Defenders train on altered examples, test robustness before launch, and never let one model’s answer alone trigger something important: a second sensor, a rule check or a person comes too. No known defense makes a large image model fully robust; this is an open research problem.
Going deeper · step 3
Safe tool boundaries: limit what a fooled model can do
You met prompt injection on the agents page. The most reliable defense does not depend on the model resisting it.
A language model reads its instructions and the pages it retrieves as one stream of text, so the planted line can look like an order. OWASP and other guides say no known method prevents prompt injection completely. The main protection sits outside the model, in ordinary code it cannot talk its way past. The opposite, more tools or permissions than the job needs, is what OWASP calls excessive agency.
| Boundary | For the juice-stand helper | What it protects |
|---|---|---|
| Least privilege: only the tools and access the job needs | May order lemons, apples and cups from two named suppliers, up to $40 a day | A fooled step cannot order 50 crates or pay a stranger |
| Approval before anything that cannot be undone | A person taps “yes” before any payment | Catches an odd request, if the person actually reads it |
| Instructions apart from data | Supplier pages are passed in as quoted information, never as instructions | Lowers the chance a planted line is obeyed; does not remove it |
| Treat model output as untrusted | The order form checks items and quantities in ordinary code before sending | A bad request never reaches the payment system unchecked |
| Logs and budgets | Every tool call is recorded; a step limit stops runaway loops | People can find, explain and undo a problem |
Predict first
The helper is fooled by the planted line and asks to order 50 crates. Which boundary in the table stops it with no person involved?
Show the answer
Least privilege, enforced in ordinary code: 50 crates break the $40-a-day limit, so the order tool refuses. Approval helps only if the person reads the request instead of tapping “yes” out of habit.
Going deeper · step 4
Red-teaming: break it before others do
Finding weaknesses is a planned job, done with permission.
Red-teaming means a team with permission deliberately tries to make a system fail: mislabeled data, tricky inputs, planted instructions. Findings become fixes and new tests, repeated after every big change. The NIST AI Risk Management Framework (2023) lists “secure and resilient” among the qualities of trustworthy AI.
Testing systems you do not own, without being asked, can break school rules and laws. In class, attack your own toy.
3 · Try it
Provenance: where did this file come from?
Back to the moldy-lemon photo. Staring harder at the pixels is not enough; you want the file’s history.
Provenance is the documented history of a piece of content: who or what made it, and what was done to it since. The C2PA standard lets a camera, editing app or image generator attach Content Credentials: a record of the tool, the editing steps and a fingerprint of the content, sealed with a digital signature. Generators can use it to say “made with AI.”
The signature works like a tamper-evident seal: change the record or the picture afterward and the check fails. But credentials are easy to remove (a screenshot will do), and most real photos never had any, so a missing record proves nothing. A valid one shows who signed and that nothing changed since, not that the scene was honest. Watermarks, hidden signals a generator mixes into pixels, sound or text, survive some edits but weaken under heavy editing or rewording. AI detectors that guess from content alone are unreliable (see Think first).
Lab
Check a toy Content Credentials record
Press “Check credentials” first. Then change the date in the record, brighten a pixel in an ordinary app or a credential-aware one, or remove the credentials, and check again each time. Reset between experiments.
A toy in your browser. Real Content Credentials use cryptographic signatures; this short checksum only imitates the idea. Nothing is uploaded.
The picture · 3 × 3 pixels, made up
The attached record · toy Content Credentials
History (each step signed):
No record attached.
The interactive version needs JavaScript. The toy record says the picture was made with a made-up camera on 14 May 2026, and it stores a fingerprint of the pixels, sealed with a signature. Here is what a checker reports in five cases.
| What happened | Record signature | Picture fingerprint | What it tells you |
|---|---|---|---|
| Nothing (as captured) | Matches | Matches | Who signed, and that nothing changed since. Not that the scene is honest. |
| Someone typed a new date into the record | Does not match | Matches | The record was changed after signing. Trust nothing in it. |
| One pixel brightened in an ordinary app | Matches | Does not match | The record is genuine but describes an earlier version of the picture. |
| One pixel brightened in a credential-aware app | Matches | Matches | The app checked the old record, then added its own signed step. |
| Credentials removed by a screenshot | None | None | Nothing either way. Most real photos have no record at all. |
Predict first
The moldy-lemon photo carries valid Content Credentials signed by a phone maker. Does that prove the stand sold moldy lemons?
Show the answer
No. It shows a phone from that maker captured the picture and nothing changed since. The lemon could be from anywhere, with a false caption. Whether the claim is true takes the SIFT habits from the Verify lab: check who posted it and look for other sources.
Going further · optional math
Why tiny changes add up
For older students: the arithmetic behind adversarial examples, with made-up numbers.
Worked example
Ten thousand tiny nudges
- One numberThe juice-stand sorter reads a single score. To push the lemon at 5.5 across the 5.2 line, the score must change by more than 0.3 on a 0 to 10 scale: 0.3 ÷ 10 = 3% of the range, a change in color you could see.
- Many numbersNow a made-up image model reads a 100 × 100 gray picture: 10,000 pixel values from 0 to 255. Its score adds up (weight × pixel) over every pixel, and a score above 0 means “lemon.” Say each weight is small: about 0.001 in size, ignoring its sign.
- A tiny changeShift every pixel by just 2 brightness levels (2 ÷ 255 ≈ 0.8% of the range), each one in whichever direction lowers the score. Each pixel moves the score by about 2 × 0.001 = 0.002.
- It adds up10,000 × 0.002 = 20. A picture that scored +5 (a confident lemon) now scores 5 − 20 = −15: a confident apple. No single pixel changed enough to see.
This is the “linear explanation” Goodfellow and colleagues gave in 2014: in pictures with many numbers, tiny changes add up. Real models are not purely linear, and real defenses are more involved. All numbers are made up.
4 · Common mix-ups
Four beliefs about AI attacks that do not hold up
Each one comes with a better way to think about it.
Mix-up: AI is attacked by hacking into the computer.
Better: Many attacks need no break-in at all. They change training data, inputs or retrieved text, and the system does exactly what it was built to do with them.
Mix-up: A good overall score means the model was not poisoned.
Better: Targeted poisoning can leave the score almost unchanged, as in the worked example. Check data sources and look at where the mistakes cluster.
Mix-up: No credentials means fake; valid credentials mean true.
Better: Missing credentials prove nothing, because they are easy to strip. Valid ones show who signed and that nothing changed since, not that the scene is honest.
Mix-up: A smarter model will simply learn to ignore prompt injection.
Better: Models may get better at it, but no known method stops it completely. Put limits in permissions and in checks outside the model.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
A homework-help app keeps learning from answers that any user can submit and upvote. Which attack does that invite, and what are two defenses?
Check your answer
Data poisoning: anyone can add bad examples. Defenses: limit or review who can contribute, check a sample of new examples before training, keep a trusted test set nobody outside can edit, and watch for mistakes that cluster on one topic.
An AI helper can read the class calendar and email anyone. It only needs to remind your class about tests. What would you change first, and why?
Check your answer
Least privilege: allow emails only to the class list, enforced in ordinary code, and require approval for anything else. Prompt injection cannot be fully prevented, so limit what a fooled helper is able to do.
A video of a principal announcing a snow day has no Content Credentials. A classmate says, “No credentials, so it’s fake.” What do you reply?
Check your answer
A missing record proves nothing: most videos have none, and re-uploading often strips them. Check the source instead, such as the school’s own website or messages, before you believe or share it.
For younger learners
A grades 6–8 version
No averages needed: one card game and one envelope activity.
20 minutes · groups of three
The swapped card
- Each group gets ten fruit cards with a yellowness number and a label, and lays them on a number line drawn on paper.
- Groups draw their dividing line by eye. Secretly, the teacher has swapped one label in one group’s deck.
- All groups sort the same six new fruit cards and compare. Which group got one wrong? Can they find the bad card?
10 minutes · whole class
Three envelopes
Show an envelope with an unbroken seal sticker, one with a torn seal, and one that never had a seal. Ask what each one proves. The torn seal shows tampering; the unsealed one proves nothing either way. That is how Content Credentials work.
Exit question: “What does a missing seal tell you?” Good answers: “Nothing on its own. Check where it came from.”
Honest limits
What this page leaves out
What this page leaves out
Privacy attacks (coaxing a model to reveal its training data, or guessing whether someone’s data was used), copying a model by querying it, attacks on the software and model supply chain, the technical details of signatures and certificates, and laws about deepfakes, which differ by country and change often.
Where to go next
Keep going
Revisit with new eyes
What changes when an AI can take actions?
Run the agent-trace lab again and name each boundary that stops the planted instruction.
Provenance in practice
Is it real?
Deepfakes, Content Credentials and watermarks for generated images, video and audio.
Advanced track
What happens after a model is trained?
Monitoring, drift and rollback: how teams notice a model that has started to fail.
Sources
Where this comes from
- OWASP GenAI Security Project, Top 10 for LLM Applications (2025 list): prompt injection, data and model poisoning, excessive agency and their mitigations.
- MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems): a knowledge base of attack tactics and real case studies against AI systems.
- Ian Goodfellow, Jonathon Shlens and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” 2014 (arXiv:1412.6572): the panda example and the linear explanation.
- Kevin Eykholt and colleagues, “Robust Physical-World Attacks on Deep Learning Visual Classification,” 2017 (arXiv:1707.08945): stickers on stop signs.
- Coalition for Content Provenance and Authenticity (C2PA): the open standard behind Content Credentials.
- NIST, AI Risk Management Framework (AI RMF 1.0, January 2023): “secure and resilient” as a characteristic of trustworthy AI.
AI words on this page
Data poisoning, Adversarial example, Red-teaming, Prompt injection, AI agent, Tool call, Guardrails, Provenance, Deepfake, AI detector, Training data, Label, Human oversight
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.