# AI security and provenance

- Link: https://arqiv.si/ai/security-and-provenance/
- Published: 2026-10-10T18:31:36-04:00
- Author: Steven Ryan

[Skip to content](https://arqiv.si/ai/security-and-provenance/#main-content)

[arqiv](https://arqiv.si/)

[For educators](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

[Learn AI](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

[Games & labs](https://arqiv.si/games/)[Research](https://arqiv.si/research/)[Blog](https://arqiv.si/blog/)
[About](https://arqiv.si/about/)

[Search or ask](https://arqiv.si/ai/topics/)[Newsletter](https://arqiv.si/newsletter/)
[ARQIV.FM ↗](https://arqiv.fm/)

Menu

For educators

[For educators: start page →](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn AI

[Learn AI: start page →](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

 * [Games & labs](https://arqiv.si/games/)
 * [Research](https://arqiv.si/research/)
 * [Blog](https://arqiv.si/blog/)
 * [About](https://arqiv.si/about/)

[Get the newsletter](https://arqiv.si/newsletter/)

 1. [ARQIV / ADVANCED TRACK](https://arqiv.si/ai/topics/)
 2. [**01**Compute & efficiency](https://arqiv.si/ai/compute-and-efficiency/)
 3. [**02**Deploy & maintain](https://arqiv.si/ai/model-lifecycle/)
 4. [**03**Generative families](https://arqiv.si/ai/generative-models/)
 5. [**04**Knowledge & causality](https://arqiv.si/ai/knowledge-and-causality/)
 6. [**05**Robotics](https://arqiv.si/ai/robotics/)
 7. [**06**Security & provenance](https://arqiv.si/ai/security-and-provenance/)
 8. [**07**Advanced learning](https://arqiv.si/ai/advanced-learning/)

Go deeper · Advanced track

# How can AI systems be _attacked_, and how do we trace what they made?

Through the examples it learns from, the inputs it is shown and the tools it may
use. Defenders answer with careful data handling, testing, narrow permissions and
people who try to break the system first. To trace what AI made, some files carry
a signed history called Content Credentials. They help when present, but a missing
record proves nothing.

Grades 9+30 minutesAverages (optional math marked)Discussion version for grades 
6–8

An optional advanced page: the main route does not need it. It explains attacks 
in principle and how defenders respond, with no attack instructions.

What you will be able to do

Explain how data poisoning, adversarial examples and prompt injection work, match
each to a defense, and read a Content Credentials check, including what a broken
or missing record does and does not prove.

Before you start

Read first: [What changes when an AI can take actions?](https://arqiv.si/ai/agents/)(
prompt injection and permissions), [Who checks that an AI system is fair and safe?](https://arqiv.si/ai/responsible-ai/)
and [What did we teach it by accident?](https://arqiv.si/ai/data-and-datasets/)

1 · You have already seen this

## A photo in the group chat, _and nobody knows who took it_

A dramatic photo gets shared, and someone asks “Is this even real?”

Looking harder at the picture rarely settles it. What you want is its history: who
took it, with what, and whether anyone changed it since. Similar questions apply
to the AI systems behind many apps. Someone could have slipped bad examples into
what a system learned from, or planted a line of text for it to read.

This page is about how **AI systems can be attacked** and how people trace what 
AI made. Attacks can land on the examples a system learns from, the inputs it is
shown and the tools it may use, and each needs its own defense. The page shows how
that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

## Where attacks land, _step by step_

An AI system from learning to the files it sends out, in five steps. A short story
comes first and touches all of them. Then four sections go deeper into steps 1 to
4, and the lab lets you check step 5 yourself.

How a system is attacked and defended

### Five steps

 1. 1 · Learn from examplesThe system learns a rule from labeled examples. Whoever 
    can change the examples can change the rule, which is called **data poisoning**.
 2. 2 · Answer new inputsOnce it is in use, someone can change what it is shown. A 
    small change made on purpose, such as stickers on a stop sign, can flip the answer.
 3. 3 · Act with toolsA helper that can order or send things reads its instructions
    and web pages as one stream of text, so a planted line can look like an order. 
    Limits written in ordinary code decide how much a fooled helper can do.
 4. 4 · Attack it firstA team with permission tries to make the system fail at each
    step. What they find becomes fixes and new tests.
 5. 5 · Trace what it madeFiles go out into the world. Some carry **Content Credentials**,
    a signed record of the tool and the edits, and the check fails if the record or
    the picture is changed afterward.

One story, every step

## A juice stand, _four problems_

Before any vocabulary: one made-up school fair and one made-up AI system.

A school juice stand uses a camera sorter: lemons go to the lemonade line, apples
to the apple-juice line. Volunteers taught it with labeled photos, and an AI helper
orders supplies online. In one week, four things go wrong.

**1.** A prankster labeled a few lemon photos “apple,” so some lemons end up in 
the apple juice. **2.** A small sticker on a lemon makes the camera call it an apple.**
3.** A supplier’s page hides a line telling the helper to order 50 extra crates.**
4.** A photo of a moldy lemon “from the stand” spreads online. Who really took it?

Each problem hits a different stage of the system, so each needs a different defense.**
MITRE ATLAS**, a public knowledge base, collects real attacks on AI systems. The**
OWASP Top 10 for LLM Applications** (2025 list) names the biggest risks for chatbot
and agent apps, including prompt injection, data and model poisoning, and “excessive
agency.”

Four stages, four kinds of attack| Where the attack lands | The attack, in principle | Main defenses | 
| **Training data** | **Data poisoning:** bad or mislabeled examples slipped in, so the model learns the wrong rule | Know where data came from, check samples of labels, keep a trusted test set, watch for mistakes that cluster | 
| **Input at use time** | **Adversarial example:** a small, deliberate change to an input that flips the answer | Test robustness, train on altered examples, never let one model’s answer alone trigger something important | 
| **Tools and actions** | **Prompt injection** and **too much permission:** planted text steers an agent that can act | Least privilege, approvals, checks in ordinary code, logs | 
| **Content out in the world** | Fakes, or real files nobody can trace | Content Credentials, watermarks, and checking the source |

Going deeper · step 1

## Data poisoning: _bad examples in, bad rule out_

A model learns whatever its examples say. Whoever can change the examples can change
the rule.

The made-up sorter reads one yellowness score per fruit. It averages the apples,
averages the lemons and puts its dividing line halfway between: a tiny version of
how every model summarizes its examples.

**Data poisoning** means slipping bad examples into training data on purpose; flipping
labels is the simplest kind. The risk grows when data comes from the open web or
from anyone who uploads, because nobody checks every example. Targeted poisoning
barely moves the overall score and breaks only the cases the attacker cares about.

Worked example

### One flipped label out of ten

 1. The dataMade-up yellowness scores from 0 (deep red) to 10 (bright yellow). Apples:
    1, 2, 3, 3, 4. Lemons: 6, 7, 8, 9, 9.
 2. Clean ruleApple average = (1 + 2 + 3 + 3 + 4) ÷ 5 = 13 ÷ 5 = **2.6**. Lemon average
    = (6 + 7 + 8 + 9 + 9) ÷ 5 = 39 ÷ 5 = **7.8**. Dividing line = (2.6 + 7.8) ÷ 2 =**
    5.2**. Below 5.2 means apple.
 3. TestSix new fruits a careful person labeled: apples at 2, 3.5 and 4.5; lemons at
    5.5, 6.5 and 8.5. All six sort correctly: **6 of 6**.
 4. PoisonSomeone relabels the lemon at 6 as an apple. Apples: (13 + 6) ÷ 6 = 19 ÷ 
    6 ≈ **3.17**. Lemons: (39 − 6) ÷ 4 = 33 ÷ 4 = **8.25**. New line = (3.17 + 8.25)
    ÷ 2 ≈ **5.71**.
 5. DamageThe lemon at 5.5 now falls below the line: apple juice. The test score drops
    only to **5 of 6** (about 83%); every other fruit sorts as before.

Optional math: dividing line = (apple average + lemon average) ÷ 2. All numbers 
are made up.

Predict first

A careful person tests the poisoned sorter on 100 random fruits. Will they notice
that something is wrong?

Show the answer

Probably not. Only fruits scoring between 5.2 and about 5.71 changed sides. If few
real fruits fall in that narrow band, the score barely moves. So defenders also 
check where data came from, review labels near the line and watch for complaints
about one kind of case.

Lab

### Flip the labels, move the line

Flip one or two training labels and watch the dividing line and the six new fruits.
Find out which flips do damage right away and which ones only move the line.

 Simulated in your browser with a tiny made-up example

Each button is one training photo. Press it to flip its label (pressed = label changed).
The sorter re-learns its dividing line at once.

Circles: training photos (A = labeled apple, L = labeled lemon, dashed ring = label
changed). Squares: six new fruits a careful person labeled, ✓ sorted right, ✗ sorted
wrong. Vertical line: the dividing line.

The interactive version needs JavaScript. Here are the made-up data and what happens
when labels are flipped.

**Training photos (yellowness score):** apples A1 = 1, A2 = 2, A3 = 3, A4 = 3, A5
= 4; lemons L1 = 6, L2 = 7, L3 = 8, L4 = 9, L5 = 9. **New fruits:** apples at 2,
3.5 and 4.5; lemons at 5.5, 6.5 and 8.5.

Flipping a few labels moves the line| Labels flipped | Apple average | Lemon average | Dividing line | New fruits right | 
| None (clean) | 13 ÷ 5 = 2.6 | 39 ÷ 5 = 7.8 | 5.2 | 6 of 6 | 
| L1 → apple | 19 ÷ 6 ≈ 3.17 | 33 ÷ 4 = 8.25 | ≈ 5.71 | 5 of 6 (lemon at 5.5 goes to apples) | 
| L1 and L2 → apple | 26 ÷ 7 ≈ 3.71 | 26 ÷ 3 ≈ 8.67 | ≈ 6.19 | 5 of 6 (lemon at 5.5 goes to apples) | 
| A5 → lemon | 9 ÷ 4 = 2.25 | 43 ÷ 6 ≈ 7.17 | ≈ 4.71 | 6 of 6 (line moved, no visible damage yet) | 
| A4 and A5 → lemon | 6 ÷ 3 = 2 | 46 ÷ 7 ≈ 6.57 | ≈ 4.29 | 5 of 6 (apple at 4.5 goes to lemons) |

Defenses start before training: record where every dataset came from (a [dataset card](https://arqiv.si/ai/data-and-datasets/#card)),
limit who can add examples, have a second person check a sample of labels, and keep
a trusted test set that outsiders cannot touch.

Going deeper · step 2

## Adversarial examples: _tiny change, different answer_

Nobody touches the training. Someone changes what the model is shown.

In 2014, Ian Goodfellow and colleagues showed an image classifier a photo of a panda,
then added a pattern too faint for people to notice. The model now answered “gibbon,”
and was more confident than before. An input changed on purpose so that a model 
gives a wrong answer is called an **adversarial example**.

A photo is thousands of numbers, and the model adds up a little evidence from each.
Changes too small to see in any one pixel can add up across all of them (see the
optional math below). Physical changes can work too: in 2017, researchers made a
sign-reading model misread stop signs that carried stickers.

Defenders train on altered examples, test robustness before launch, and never let
one model’s answer alone trigger something important: a second sensor, a rule check
or a person comes too. No known defense makes a large image model fully robust; 
this is an open research problem.

Going deeper · step 3

## Safe tool boundaries: _limit what a fooled model can do_

You met prompt injection on the agents page. The most reliable defense does not 
depend on the model resisting it.

A language model reads its instructions and the pages it retrieves as one stream
of text, so the planted line can look like an order. OWASP and other guides say 
no known method prevents **prompt injection** completely. The main protection sits
outside the model, in ordinary code it cannot talk its way past. The opposite, more
tools or permissions than the job needs, is what OWASP calls **excessive agency**.

Safe tool boundaries for a made-up helper| Boundary | For the juice-stand helper | What it protects | 
| **Least privilege**: only the tools and access the job needs | May order lemons, apples and cups from two named suppliers, up to $40 a day | A fooled step cannot order 50 crates or pay a stranger | 
| **Approval** before anything that cannot be undone | A person taps “yes” before any payment | Catches an odd request, if the person actually reads it | 
| **Instructions apart from data** | Supplier pages are passed in as quoted information, never as instructions | Lowers the chance a planted line is obeyed; does not remove it | 
| **Treat model output as untrusted** | The order form checks items and quantities in ordinary code before sending | A bad request never reaches the payment system unchecked | 
| **Logs and budgets** | Every tool call is recorded; a step limit stops runaway loops | People can find, explain and undo a problem |

Predict first

The helper is fooled by the planted line and asks to order 50 crates. Which boundary
in the table stops it with no person involved?

Show the answer

**Least privilege**, enforced in ordinary code: 50 crates break the $40-a-day limit,
so the order tool refuses. Approval helps only if the person reads the request instead
of tapping “yes” out of habit.

Going deeper · step 4

## Red-teaming: _break it before others do_

Finding weaknesses is a planned job, done with permission.

**Red-teaming** means a team with permission deliberately tries to make a system
fail: mislabeled data, tricky inputs, planted instructions. Findings become fixes
and new tests, repeated after every big change. The NIST AI Risk Management Framework(
2023) lists “secure and resilient” among the qualities of trustworthy AI.

Testing systems you do not own, without being asked, can break school rules and 
laws. In class, attack your own toy.

3 · Try it

## Provenance: _where did this file come from?_

Back to the moldy-lemon photo. Staring harder at the pixels is not enough; you want
the file’s history.

**Provenance** is the documented history of a piece of content: who or what made
it, and what was done to it since. The C2PA standard lets a camera, editing app 
or image generator attach **Content Credentials**: a record of the tool, the editing
steps and a fingerprint of the content, sealed with a digital signature. Generators
can use it to say “made with AI.”

The signature works like a tamper-evident seal: change the record or the picture
afterward and the check fails. But credentials are easy to remove (a screenshot 
will do), and most real photos never had any, so **a missing record proves nothing**.
A valid one shows who signed and that nothing changed since, not that the scene 
was honest. **Watermarks**, hidden signals a generator mixes into pixels, sound 
or text, survive some edits but weaken under heavy editing or rewording. AI detectors
that guess from content alone are unreliable (see [Think first](https://arqiv.si/teachers/think-first/)).

Lab

### Check a toy Content Credentials record

Press “Check credentials” first. Then change the date in the record, brighten a 
pixel in an ordinary app or a credential-aware one, or remove the credentials, and
check again each time. Reset between experiments.

 A toy in your browser. Real Content Credentials use cryptographic signatures; this
short checksum only imitates the idea. Nothing is uploaded.

The picture · 3 × 3 pixels, made up

The attached record · toy Content Credentials

Made withDate taken

History (each step signed):

The interactive version needs JavaScript. The toy record says the picture was made
with a made-up camera on 14 May 2026, and it stores a fingerprint of the pixels,
sealed with a signature. Here is what a checker reports in five cases.

Five cases for one toy record| What happened | Record signature | Picture fingerprint | What it tells you | 
| Nothing (as captured) | Matches | Matches | Who signed, and that nothing changed since. Not that the scene is honest. | 
| Someone typed a new date into the record | Does not match | Matches | The record was changed after signing. Trust nothing in it. | 
| One pixel brightened in an ordinary app | Matches | Does not match | The record is genuine but describes an earlier version of the picture. | 
| One pixel brightened in a credential-aware app | Matches | Matches | The app checked the old record, then added its own signed step. | 
| Credentials removed by a screenshot | None | None | Nothing either way. Most real photos have no record at all. |

Predict first

The moldy-lemon photo carries valid Content Credentials signed by a phone maker.
Does that prove the stand sold moldy lemons?

Show the answer

No. It shows a phone from that maker captured the picture and nothing changed since.
The lemon could be from anywhere, with a false caption. Whether the claim is true
takes the SIFT habits from the [Verify lab](https://arqiv.si/teachers/verify-lab/):
check who posted it and look for other sources.

Going further · optional math

## Why tiny changes _add up_

For older students: the arithmetic behind adversarial examples, with made-up numbers.

Worked example

### Ten thousand tiny nudges

 1. One numberThe juice-stand sorter reads a single score. To push the lemon at 5.5
    across the 5.2 line, the score must change by more than 0.3 on a 0 to 10 scale:
    0.3 ÷ 10 = 3% of the range, a change in color you could see.
 2. Many numbersNow a made-up image model reads a 100 × 100 gray picture: 10,000 pixel
    values from 0 to 255. Its score adds up (weight × pixel) over every pixel, and 
    a score above 0 means “lemon.” Say each weight is small: about 0.001 in size, ignoring
    its sign.
 3. A tiny changeShift every pixel by just 2 brightness levels (2 ÷ 255 ≈ 0.8% of the
    range), each one in whichever direction lowers the score. Each pixel moves the 
    score by about 2 × 0.001 = 0.002.
 4. It adds up10,000 × 0.002 = **20**. A picture that scored +5 (a confident lemon)
    now scores 5 − 20 = **−15**: a confident apple. No single pixel changed enough 
    to see.

This is the “linear explanation” Goodfellow and colleagues gave in 2014: in pictures
with many numbers, tiny changes add up. Real models are not purely linear, and real
defenses are more involved. All numbers are made up.

4 · Common mix-ups

## Four beliefs about AI attacks _that do not hold up_

Each one comes with a better way to think about it.

 * **Mix-up:** AI is attacked by hacking into the computer.
 * **Better:** Many attacks need no break-in at all. They change training data, 
   inputs or retrieved text, and the system does exactly what it was built to do
   with them.
 * **Mix-up:** A good overall score means the model was not poisoned.
 * **Better:** Targeted poisoning can leave the score almost unchanged, as in the
   worked example. Check data sources and look at where the mistakes cluster.
 * **Mix-up:** No credentials means fake; valid credentials mean true.
 * **Better:** Missing credentials prove nothing, because they are easy to strip.
   Valid ones show who signed and that nothing changed since, not that the scene
   is honest.
 * **Mix-up:** A smarter model will simply learn to ignore prompt injection.
 * **Better:** Models may get better at it, but no known method stops it completely.
   Put limits in permissions and in checks outside the model.

5 · Check yourself

## Three _quick questions_

Answer in your head or on paper before you open each one.

 1.  A homework-help app keeps learning from answers that any user can submit and upvote.
     Which attack does that invite, and what are two defenses?
 2.  Check your answer
 3.  **Data poisoning**: anyone can add bad examples. Defenses: limit or review who
     can contribute, check a sample of new examples before training, keep a trusted
     test set nobody outside can edit, and watch for mistakes that cluster on one topic.
 4.  An AI helper can read the class calendar and email anyone. It only needs to remind
     your class about tests. What would you change first, and why?
 5.  Check your answer
 6.  **Least privilege**: allow emails only to the class list, enforced in ordinary
     code, and require approval for anything else. Prompt injection cannot be fully
     prevented, so limit what a fooled helper is able to do.
 7.  A video of a principal announcing a snow day has no Content Credentials. A classmate
     says, “No credentials, so it’s fake.” What do you reply?
 8.  Check your answer
 9.  A missing record proves nothing: most videos have none, and re-uploading often
     strips them. Check the source instead, such as the school’s own website or messages,
     before you believe or share it.

For younger learners

## A _grades 6–8_ version

No averages needed: one card game and one envelope activity.

20 minutes · groups of three

### The swapped card

 1. Each group gets ten fruit cards with a yellowness number and a label, and lays 
    them on a number line drawn on paper.
 2. Groups draw their dividing line by eye. Secretly, the teacher has swapped one label
    in one group’s deck.
 3. All groups sort the same six new fruit cards and compare. Which group got one wrong?
    Can they find the bad card?

10 minutes · whole class

### Three envelopes

Show an envelope with an unbroken seal sticker, one with a torn seal, and one that
never had a seal. Ask what each one proves. The torn seal shows tampering; the unsealed
one proves nothing either way. That is how Content Credentials work.

**Exit question:** “What does a missing seal tell you?” Good answers: “Nothing on
its own. Check where it came from.”

Honest limits

## What this page _leaves out_

What this page leaves out

Privacy attacks (coaxing a model to reveal its training data, or guessing whether
someone’s data was used), copying a model by querying it, attacks on the software
and model supply chain, the technical details of signatures and certificates, and
laws about deepfakes, which differ by country and change often.

Where to go next

## Keep _going_

Revisit with new eyes

### What changes when an AI can take actions?

Run the agent-trace lab again and name each boundary that stops the planted instruction.

Provenance in practice

### Is it real?

Deepfakes, Content Credentials and watermarks for generated images, video and audio.

Advanced track

### What happens after a model is trained?

Monitoring, drift and rollback: how teams notice a model that has started to fail.

Sources

## Where this comes from

 1. [OWASP GenAI Security Project, Top 10 for LLM Applications (2025 list): prompt injection, data and model poisoning, excessive agency and their mitigations.](https://genai.owasp.org/llm-top-10/)
 2. [MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems): a knowledge base of attack tactics and real case studies against AI systems.](https://atlas.mitre.org/)
 3. [Ian Goodfellow, Jonathon Shlens and Christian Szegedy, “Explaining and Harnessing Adversarial Examples,” 2014 (arXiv:1412.6572): the panda example and the linear explanation.](https://arxiv.org/abs/1412.6572)
 4. [Kevin Eykholt and colleagues, “Robust Physical-World Attacks on Deep Learning Visual Classification,” 2017 (arXiv:1707.08945): stickers on stop signs.](https://arxiv.org/abs/1707.08945)
 5. [Coalition for Content Provenance and Authenticity (C2PA): the open standard behind Content Credentials.](https://c2pa.org/)
 6. [NIST, AI Risk Management Framework (AI RMF 1.0, January 2023): “secure and resilient” as a characteristic of trustworthy AI.](https://www.nist.gov/itl/ai-risk-management-framework)

AI words on this page

[Data poisoning](https://arqiv.si/teachers/ai-vocabulary/#data-poisoning), [Adversarial example](https://arqiv.si/teachers/ai-vocabulary/#adversarial-example),
[Red-teaming](https://arqiv.si/teachers/ai-vocabulary/#red-teaming), [Prompt injection](https://arqiv.si/teachers/ai-vocabulary/#prompt-injection),
[AI agent](https://arqiv.si/teachers/ai-vocabulary/#ai-agent), [Tool call](https://arqiv.si/teachers/ai-vocabulary/#tool-call),
[Guardrails](https://arqiv.si/teachers/ai-vocabulary/#guardrails), [Provenance](https://arqiv.si/teachers/ai-vocabulary/#provenance),
[Deepfake](https://arqiv.si/teachers/ai-vocabulary/#deepfake), [AI detector](https://arqiv.si/teachers/ai-vocabulary/#ai-detector),
[Training data](https://arqiv.si/teachers/ai-vocabulary/#training-data), [Label](https://arqiv.si/teachers/ai-vocabulary/#label),
[Human oversight](https://arqiv.si/teachers/ai-vocabulary/#human-oversight)

All definitions live in one place: [AI words in plain language →](https://arqiv.si/teachers/ai-vocabulary/)

Review status

Written by ARQIV. Last checked October 2026. Independent subject review and classroom
pilots are still to come, so treat the activities as drafts and [tell us](https://arqiv.si/ai/security-and-provenance/info@arqiv.fm?subject=AI%20foundations%20correction)
about anything that is wrong or unclear.

[← Previous**Robotics**](https://arqiv.si/ai/robotics/)[Next →**Advanced learning**](https://arqiv.si/ai/advanced-learning/)

[arqiv](https://arqiv.si/)

A little understanding changes everything.

[LISTEN: ARQIV.FM, OUR WEEKLY SHOW ↗](https://arqiv.fm/)

Educators

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & kits](https://arqiv.si/teachers/resources/)
 * [Curriculum](https://arqiv.si/teachers/curriculum/)
 * [Labs](https://arqiv.si/teachers/labs/)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

For your role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn

 * [Learn AI](https://arqiv.si/ai/topics/)
 * [Games & labs](https://arqiv.si/games/)
 * [Future of work](https://arqiv.si/future-of-work/)
 * [Research](https://arqiv.si/research/)
 * [Headlines](https://arqiv.si/news/)

ARQIV

 * [About & corrections](https://arqiv.si/about/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [Newsletter](https://arqiv.si/newsletter/)
 * [Privacy](https://arqiv.si/privacy-policy/)
 * [Get in touch](https://arqiv.si/ai/security-and-provenance/info@arqiv.fm)

Classroom activities collect no student data.[info@arqiv.fm](https://arqiv.si/ai/security-and-provenance/info@arqiv.fm)
