Go deeper · Ai at work
What does a camera-based AI actually measure?
Light. A camera records how much light reached each tiny spot on its sensor, and stores it as numbers. Computer vision is the part of AI that turns those numbers into useful outputs: a label, boxes around objects, a map of every pixel, or the words in a picture. The camera does not understand anything, and when the light changes, the numbers change too.
What you will be able to do
Tell apart classification, object detection, segmentation and OCR by their outputs, explain why a change in lighting can break a vision system that worked before, and name two privacy or fairness questions to ask about cameras that recognize people.
Before you start
Helpful: How does a picture become numbers? (pixels as numbers) and the weed-or-crop sprayer lab on the farming page. No math beyond counting.
1 · You have already seen this
Your camera already does more than take photos
Phones and apps pick things out of pictures every day.
Point a phone at a sign and it can copy the words so you can paste them. A plant app looks at a leaf and says “dandelion.” On a video call, everything behind you can blur while you stay sharp, and some phones unlock when they see your face.
All of these use computer vision, the part of AI that turns a camera’s picture into a label, boxes around objects, a map of every pixel or the words in the picture. The camera only records light as numbers, and when the light changes, the numbers change too. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From light to a useful answer
What happens between the camera and the answer, in four steps. The next section goes deeper into steps 1 and 2, and the one after it into step 4.
How computer vision works
Four steps
- 1 · Light hits the sensorA camera sensor is a grid of millions of tiny light detectors. Each one counts how much light of a color reached it.
- 2 · Light becomes numbersEach count becomes a pixel value, often from 0 (dark) to 255 (bright), with three values per pixel for red, green and blue.
- 3 · A model looks for patternsThe model has learned from many labeled examples which patterns of numbers usually go with which answers. That is why it can be fooled when the numbers change in a way it never saw during training.
- 4 · Out comes an answerDepending on the job, it gives one label for the whole picture, a box around each object, a label for every pixel, or the text in the picture.
Going deeper · steps 1 and 2
Pixels are measurements
Before any AI runs, the picture is already a grid of numbers. Those numbers record how much light reached the camera. They say nothing yet about what the objects are.
A camera sensor is a grid of millions of tiny light detectors. Each one counts how much light of a color hit it during the photo. That count becomes a pixel value, often from 0 (dark) to 255 (bright), with three values per pixel for red, green and blue. In Lightprint, a spectrum is a picture only one row tall; a photo is the same idea in a grid.
Nothing in those numbers says “weed” or “stop sign.” The same leaf gives different numbers at noon, at dusk, in shadow, through a dusty lens or from a different angle. A vision model has to learn which patterns of numbers usually go with which answers, from many labeled examples. That is why it can be fooled when the numbers change in a way it never saw during training.
Going deeper · step 4
Label, boxes, mask or text
Computer-vision systems differ most in what they output. The output decides what the system can be used for.
| Job | What comes out | Everyday example | What the training labels look like |
|---|---|---|---|
| Classification | One label for the whole picture | A plant app says “dandelion” | One word per photo |
| Object detection | A box and a label for each object it finds | A self-checkout camera draws boxes around items | A box drawn around every object |
| Segmentation | A label for every pixel (a mask) | A video call blurs everything except you | Every pixel colored by hand or with tools: slow |
| OCR (optical character recognition) | The text in the picture, as letters | A phone copies words from a photo of a sign | Pictures of text with the correct text typed out |
Predict first
A sprayer must squirt one weed growing between two crop plants, without hitting the crops. Which output does it need: a label, boxes or a mask?
Show the answer
A whole-picture label (“weed present”) cannot aim a nozzle. Boxes say where each plant is, which may be enough when plants are spaced apart. When leaves overlap, a mask is better, because it says exactly which pixels are weed. Finer outputs are more useful, and their training labels take much longer to make.
3 · Try it
Find the plants, then change the light
A tiny made-up field, a simple brightness rule, and three ways to report what it found.
Predict first
Before you open the lab: the rule says “a pixel is a plant if its brightness is 5 or more.” The sun comes out and every pixel gets 4 brighter. What happens?
Show the answer
Soil was 1 to 3; now it is 5 to 7. The rule marks every pixel as a plant: all 76 soil pixels are wrongly marked, and the boxes merge into one box around the whole field. Try it, then look at what the whole-picture label says.
Lab
Plant finder under changing light
Pick an output type, then change the lighting. The rule never changes, but the numbers it sees do. In the box view, dashed outlines show where the plants really are. Then try the relative rule, which compares each pixel with the picture’s average.
A made-up 12 × 8 field and a hand-written rule, computed in your browser. Real systems use trained models.
The lab needs JavaScript, so here are its numbers. A made-up camera gives each of the 12 × 8 = 96 squares one brightness number from 0 (dark) to 9 (bright). There are 20 plant pixels: a crop on the left and a weed on the right. The rule calls a pixel “plant” if it is bright enough.
| Row | 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Row 0 | 2 | 1 | 2 | 3 | 2 | 1 | 2 | 3 | 2 | 1 | 2 | 3 |
| Row 1 | 3 | 2 | 3 | 2 | 1 | 2 | 1 | 2 | 1 | 2 | 3 | 2 |
| Row 2 | 2 | 1 | 6 | 7 | 6 | 3 | 2 | 1 | 2 | 1 | 2 | 3 |
| Row 3 | 3 | 2 | 7 | 8 | 8 | 7 | 3 | 2 | 1 | 2 | 3 | 2 |
| Row 4 | 2 | 1 | 6 | 8 | 7 | 2 | 2 | 3 | 1 | 7 | 6 | 3 |
| Row 5 | 3 | 2 | 1 | 7 | 6 | 2 | 1 | 2 | 6 | 8 | 7 | 2 |
| Row 6 | 2 | 3 | 2 | 1 | 2 | 3 | 2 | 1 | 7 | 8 | 6 | 3 |
| Row 7 | 1 | 2 | 3 | 2 | 1 | 2 | 3 | 2 | 1 | 2 | 3 | 2 |
| Lighting · rule | Plant pixels found | Soil pixels wrongly marked | Boxes | Whole-picture label |
|---|---|---|---|---|
| Normal light · Fixed | 20 of 20 | 0 | 2 | plants: yes |
| Normal light · Relative | 20 of 20 | 0 | 2 | plants: yes |
| Bright sun (+4) · Fixed | 20 of 20 | 76 | 1 | plants: yes |
| Bright sun (+4) · Relative | 20 of 20 | 0 | 2 | plants: yes |
| Dusk (−3) · Fixed | 5 of 20 | 0 | 2 | plants: yes |
| Dusk (−3) · Relative | 20 of 20 | 0 | 2 | plants: yes |
| Shadow on the left half (−3) · Fixed | 11 of 20 | 0 | 2 | plants: yes |
| Shadow on the left half (−3) · Relative | 16 of 20 | 0 | 2 | plants: yes |
Notice the last column: the whole-picture label says “plants: yes” every time, even in bright sun when every single pixel is marked as a plant.
Worked example
Counting what the rule gets right
- The field12 × 8 = 96 pixels. Soil is 1 to 3; plant pixels are 6, 7 or 8. There are 20 plant pixels: 12 in the crop and 8 in the weed. Five of them are 8s.
- Normal lightRule: 5 or brighter is a plant. All 20 plant pixels pass and no soil does: 20 of 20 found, 0 false alarms, 2 boxes.
- Dusk (−3)Plants drop to 3, 4 or 5. Only the five 8s reach 5: 5 of 20 found (25%), 15 missed. There are still 2 boxes, but they are tiny.
- Bright sun (+4)Soil rises to 5, 6 or 7 (nothing goes above 9). All 96 pixels are marked, so 76 are false alarms, and the two plants merge into 1 box around everything.
- Relative ruleThreshold = the picture’s average + 2. Average brightness is 291 ÷ 96 ≈ 3.03 in normal light, ≈ 6.64 in bright sun and ≈ 0.81 at dusk, so the threshold moves with the light and all 20 plants are found with no false alarms in all three.
- ShadowDarken only the left half. Now the fixed rule finds 11 of 20 and the relative rule 16 of 20: one average cannot fit two kinds of light at once.
All pixel values are made up and every count is checked. A real camera gives three color numbers per pixel and real systems use trained models, but they face the same problem: the numbers depend on the light.
What changes with lighting?
When the world stops matching the training photos
A model learns from the photos it was given. New conditions mean new numbers.
The lab rule failed because the light changed, not because the plants did. Trained models fail the same way, only less obviously. If every training photo was taken at noon on dry soil, the model has never seen dusk, rain, mud or a different camera. When the data at work differs from the training data, that is called distribution shift. In one well-known test, image classifiers lost much of their accuracy on photos with added fog, snow, blur or brightness changes.
The farming page lists the same worries for real sprayers: look-alike weeds, dust, shadows and speed. Common defenses: collect training photos in many conditions, adjust the numbers before the model sees them (like the relative rule), test in the exact conditions of use, and keep people checking. A good habit for any vision system: ask “what changes with lighting, weather, camera or place?”
Cameras that look at people
Privacy, surveillance and unequal errors
The same technology that finds weeds can find faces. That raises questions a plant never asks.
Face recognition compares the measurements of a face in a photo with stored faces. People in a crowd usually did not agree to be measured, a face cannot be changed like a password, and stored images can be shared or leaked. That is why many people ask who is recorded, who keeps the images and for how long, and who checks the matches.
Errors are not shared equally. In 2019 the U.S. National Institute of Standards and Technology (NIST) tested 189 face-recognition algorithms from 99 developers. For most of them, false matches were more common for some demographic groups than for others, often by a factor of 10 to more than 100, and the differences varied a lot between algorithms. A 2018 study, Gender Shades, found that commercial systems guessing gender from faces were wrong up to 34.7% of the time for darker-skinned women but at most 0.8% for lighter-skinned men.
Questions to ask about any camera system
- What exactly does it output: a label, boxes, masks, text or identities?
- Was it tested in conditions like ours, and for people like us?
- What are the error rates for each group, not only overall?
- Who sees the images, how long are they kept, and can people say no?
- What happens to a person when it makes a mistake, and who checks?
4 · Common mix-ups
The camera does not see the way we do
Three ideas that trip people up, and what to think instead.
Mix-up: The camera sees the way we see.
Better: It records numbers for light at each pixel. Recognizing a weed or a face is a separate step done by a model, and that step can be wrong.
Mix-up: If it worked in testing, it works everywhere.
Better: New lighting, weather, cameras or places change the numbers. Test in the conditions where the system will be used.
Mix-up: A single accuracy number tells you whether face recognition is fair.
Better: An overall rate can hide much higher error rates for some groups. Ask for results by group, as NIST reported.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
Name the job: (a) an app says “this photo contains a dog,” (b) a store camera draws a box around every item in a basket, (c) a video call blurs everything behind you.
Check your answer
(a) Classification: one label for the whole picture. (b) Object detection: a box and label per object. (c) Segmentation: every pixel is marked as “person” or “background,” so the background can be blurred.
In the lab’s bright sun, the whole-picture label still said “plants: yes.” Why is that not proof that the rule works?
Check your answer
The label was right by luck. The mask showed that all 96 pixels were marked as plant, including 76 soil pixels. A coarse output can hide a broken system, so check finer outputs or test cases where you know the answer.
A company says its face-recognition system is “99% accurate.” What would you ask before trusting it?
Check your answer
Accurate on what photos, and for whom? Ask for false-match and missed-match rates for each demographic group and in conditions like yours (lighting, cameras). NIST found that differences between groups were often large and varied by algorithm. Also ask who checks matches and what happens to a person after a mistake.
In the real world
An application, a limit and a link
Application
Reading old books
Libraries scan old books and newspapers and use OCR to turn the page images into searchable text. Faded ink, unusual fonts and handwriting still cause mistakes, so important texts get checked by people.
Limitation
Measurements, not understanding
A system that detects “weed” has learned patterns of light, not what a weed is. A photo of a weed printed on a T-shirt can produce similar numbers. Outputs need checking against the real world.
Connection
The networks behind it
Most vision models are neural networks built for grids, such as convolutional networks. Slide a filter yourself on Why are there different kinds of neural networks?
For younger learners
A grades 3–5 version
No numbers needed: the same object, two kinds of light.
20 minutes · small groups
Same toy, different light
- Photograph one toy by a sunny window and again in a dim corner.
- Zoom in until you see the squares (pixels). Compare one square from each photo: are they the same color?
- Ask: if a robot only knew the sunny photo, would it recognize the toy in the dim one?
Exit question
Say it in one sentence
“What does a camera really collect?” A good answer: “Colors of light in tiny squares, and they change when the light changes.” Then try the sprayer lab together.
Honest limits
What this page leaves out
What this page leaves out
How vision models are trained and what their layers learn, depth cameras and lidar, video tracking, image generation (see creative AI), medical imaging, and the laws about cameras and faces, which differ from place to place and change over time.
Where to go next
Keep looking closer
Next in this branch
What else can AI do with language?
Translation, speech recognition and more, beyond chatbots.
Vision at work
Be the sprayer’s brain
Set how sure the AI must be before it sprays, and see misses versus wasted spray.
Testing fairly
Good at what, for whom?
Misses, false alarms and why one score is never the whole story.
Sources
Where this comes from
- Patrick Grother, Mei Ngan and Kayee Hanaoka, “Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects,” NISTIR 8280, National Institute of Standards and Technology, December 2019.
- Joy Buolamwini and Timnit Gebru, “Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification,” Proceedings of Machine Learning Research 81, 2018.
- Dan Hendrycks and Thomas Dietterich, “Benchmarking Neural Network Robustness to Common Corruptions and Perturbations,” 2019: accuracy drops under fog, snow, blur and brightness changes.
- Tsung-Yi Lin and colleagues, “Microsoft COCO: Common Objects in Context,” 2014: a dataset with labels, boxes and per-pixel masks for detection and segmentation.
- Stanford CS231n course notes, “Image Classification”: images as arrays of numbers, and challenges such as illumination and viewpoint.
- AI4K12, the Five Big Ideas in AI: Big Idea 1, perception (computers perceive the world using sensors).
AI words on this page
Computer vision, Classification, Object detection, OCR, Convolutional neural network, Distribution shift, Training data, Label, False positive, Bias, Fairness, Tensor
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.