AI foundations · 04

How does a picture become numbers?

A digital picture is already a grid of numbers: one brightness value for each tiny square (pixel), or three values when the picture has color. Machine learning software keeps numbers like these in arrays called tensors. A tensor’s shape tells you how many directions it has and how long each one is, and that is most of what you need to read one.

Grades 6+15 minutesCounting and multiplyingOptional math for grades 9+

What you will be able to do

Read the shape of a small array of numbers, find one value from its coordinates, and explain how adding color and grouping pictures into a batch each add an axis.

Before you start

Knowing that screen pictures are made of pixels helps. That is all. You do not need to master this page to continue the route: skim it now and come back when the word “tensor” turns up. The Map of AI shows where tensors fit.

1 · You have already seen this

Zoom in far enough and a photo turns to squares

Zoom far into a photo on a phone or computer and it breaks into tiny squares of color.

Each square is a pixel, and real photos have millions of them. For every pixel the computer keeps numbers: how much red, green and blue light there is, each from 0 to 255. Those numbers record light and nothing else.

Before an AI model can use a photo, the numbers have to be kept in order, so the model knows which one belongs to which row, column and color. That ordered block of numbers is a tensor. This page shows how a picture becomes one, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From a picture to a block of numbers

How a picture becomes a tensor, in five steps. The next section goes deeper into steps 1 and 2, the worked example after it into steps 3 to 5, and the explorer lets you do steps 1 to 4 yourself.

How a picture becomes numbers

Five steps

  1. 1 · Measure each pixelA gray pixel becomes one number for its brightness, from 0 (black) to 255 (white).
  2. 2 · Keep the gridThe numbers stay in rows and columns. A gray picture 2 pixels tall and 3 wide has shape (2, 3) and holds 2 × 3 = 6 values, each with its own coordinates counted from 0.
  3. 3 · Add colorColor needs a red, a green and a blue number for every pixel, so a new axis of length 3 appears.
  4. 4 · Stack a batchModels usually take several pictures at once, so a batch axis is added in front. Multiply the lengths in the shape to count the numbers.
  5. 5 · Match the modelA model expects one exact shape, such as (224, 224, 3), so apps resize and crop photos before the model sees them.

Going deeper · steps 1 and 2

From one number to many axes

Each step adds one direction you can move along. Count the directions and you know how many axes the array has.

0 axes · shape ( )

One number

A scalar: today’s temperature, 21 °C. There is nothing to move along, so the shape is empty.

1 axis · shape (3,)

A list

A vector: one apple’s weight in grams, height and width in cm. One axis of length 3. Python writes a one-axis shape with a comma: (3,).

2 axes · shape (2, 3)

A grid

A matrix: a gray picture 2 pixels tall and 3 wide. Two axes, rows and columns: 2 × 3 = 6 values.

3 or more axes · shape (2, 3, 3)

A stack of grids

A color picture: the same grid, with red, green and blue numbers for every pixel. Three axes, 18 values. Stack four pictures and you get a fourth axis.

An axis is one direction you can count along: across a row, down a column, through the color channels, or from one picture to the next. The shape lists the length of every axis, in order. Multiply the lengths and you get the number of values: shape (2, 3) holds 2 × 3 = 6 numbers.

Software counts from 0, so the first row is row 0 and the top-left pixel sits at [0, 0]. A value’s position, written with one number per axis like [1, 2], is called its coordinates or index.

You have met a one-axis picture before if you played Lightprint: a light spectrum is a single row of brightness values, one for each color of light. That row is a vector.

Why the word “tensor”? In machine learning software such as PyTorch and TensorFlow, a tensor is simply an array of numbers with any number of axes (zero or more), all of one number type. In mathematics and physics the word is stricter: a tensor is defined by how its numbers must change when you change your point of view, for example when you rotate the coordinate axes. Engineers use tensors like that to describe the stresses inside a steel beam. This page uses the software meaning, which is the one you will meet in AI.

Going deeper · steps 3 to 5

One tiny picture, step by step

Real photos have millions of pixels. This one has six, so you can check every number.

Predict first

A gray picture has shape (2, 3). Give it color, then put four such pictures together in one batch. What is the new shape, and how many numbers does it hold?

Show the answer

Shape (4, 2, 3, 3) and 72 numbers: 4 pictures × 2 rows × 3 columns × 3 colors. That is 4 axes but 72 values. The number of axes and the number of values are different things.

Worked example

From six numbers to seventy-two

  1. Start: grayBrightness runs from 0 (black) to 255 (white). Row 0 is 0, 128, 255; row 1 is 64, 192, 32. Shape (2, 3), so 2 × 3 = 6 values.
  2. Add colorEach pixel now needs three numbers, one each for red, green and blue light, each from 0 to 255. A new axis of length 3 appears at the end. Shape (2, 3, 3), so 2 × 3 × 3 = 18 values.
  3. Make a batchModels usually process several pictures at once, called a batch. Four color pictures: shape (4, 2, 3, 3), so 4 × 2 × 3 × 3 = 72 values, in the order batch, height (rows), width (columns), channel.
  4. Read one valueIn the lab below, picture 0 is a made-up sunset. Its bottom-right pixel is (25, 25, 112), a dark blue. Its blue number sits at coordinates [0, 1, 2, 2]: picture 0, row 1, column 2, channel 2. The value is 112.
  5. Another orderLibraries do not all use the same axis order. PyTorch commonly puts channels first, (N, C, H, W), where N is the number of pictures in the batch. There the same batch has shape (4, 3, 2, 3), and the same 112 sits at [0, 2, 1, 2]. Same 72 numbers, arranged differently.
  6. Axes vs valuesFlatten the batch into one long list and the shape becomes (72,): 1 axis, still 72 values. The (4, 2, 3, 3) version has 4 axes and the same 72 values. Shape records axis lengths; the number of axes is not the number of values.

All pixel values on this page are made up. Many image tools store 0 to 255 as whole numbers and divide by 255 before a model uses them, so 128 becomes about 0.5.

Real pictures work the same way, just bigger. A picture 224 pixels on each side, a common input size for image classifiers, has shape (224, 224, 3) and holds 224 × 224 × 3 = 150,528 numbers. A model that expects that shape cannot take anything else, which is why apps resize and crop photos before a model sees them.

3 · Try it

Tensor explorer

Pick a picture, a pixel and a color channel. Watch the coordinates, the shape and the count change as you add axes.

Lab

Find any number in a batch of pictures

Start with one gray picture, then add color, then make a batch. Select a picture, row, column and channel with the buttons (or click a pixel). The explorer shows the number you picked, its coordinates and the shape of the whole array. Switch the axis order to see the same numbers stored channels-first.

Four made-up 2 × 3 pictures stored in your browser. No AI model runs here.

1 · How many axes?
2 · Picture (batch axis)
3 · Row (height axis)
4 · Column (width axis)
5 · Channel (color axis)
6 · Axis order

How the computer stores it: one long line of numbers, in the order set by the shape. Each bar is one number (red, green and blue numbers are tinted to match). The outlined bar is the one you picked.

Show every number as a text table
Every number in the explorer (made-up pictures)
Picture and rowColumn 0Column 1Column 2
Gray picture, row 00128255
Gray picture, row 16419232
Picture 0 (Sunset), row 0R 255 · G 94 · B 77R 255 · G 154 · B 0R 255 · G 215 · B 0
Picture 0 (Sunset), row 1R 128 · G 0 · B 128R 75 · G 0 · B 130R 25 · G 25 · B 112
Picture 1 (Leaf), row 0R 34 · G 139 · B 34R 144 · G 238 · B 144R 0 · G 100 · B 0
Picture 1 (Leaf), row 1R 107 · G 142 · B 35R 85 · G 107 · B 47R 154 · G 205 · B 50
Picture 2 (Sea), row 0R 135 · G 206 · B 235R 0 · G 191 · B 255R 30 · G 144 · B 255
Picture 2 (Sea), row 1R 0 · G 0 · B 128R 70 · G 130 · B 180R 240 · G 248 · B 255
Picture 3 (Gray stripes), row 0R 0 · G 0 · B 0R 128 · G 128 · B 128R 255 · G 255 · B 255
Picture 3 (Gray stripes), row 1R 255 · G 255 · B 255R 128 · G 128 · B 128R 0 · G 0 · B 0

The explorer needs JavaScript, so here is the same data as text. The gray picture has shape (2, 3). Each color picture has shape (2, 3, 3): 2 rows, 3 columns, 3 channels (red, green, blue). All four together have shape (4, 2, 3, 3) and hold 4 × 2 × 3 × 3 = 72 numbers.

Every number in the explorer (made-up pictures)
Picture and rowColumn 0Column 1Column 2
Gray picture, row 00128255
Gray picture, row 16419232
Picture 0 (Sunset), row 0R 255 · G 94 · B 77R 255 · G 154 · B 0R 255 · G 215 · B 0
Picture 0 (Sunset), row 1R 128 · G 0 · B 128R 75 · G 0 · B 130R 25 · G 25 · B 112
Picture 1 (Leaf), row 0R 34 · G 139 · B 34R 144 · G 238 · B 144R 0 · G 100 · B 0
Picture 1 (Leaf), row 1R 107 · G 142 · B 35R 85 · G 107 · B 47R 154 · G 205 · B 50
Picture 2 (Sea), row 0R 135 · G 206 · B 235R 0 · G 191 · B 255R 30 · G 144 · B 255
Picture 2 (Sea), row 1R 0 · G 0 · B 128R 70 · G 130 · B 180R 240 · G 248 · B 255
Picture 3 (Gray stripes), row 0R 0 · G 0 · B 0R 128 · G 128 · B 128R 255 · G 255 · B 255
Picture 3 (Gray stripes), row 1R 255 · G 255 · B 255R 128 · G 128 · B 128R 0 · G 0 · B 0

Try it on paper: coordinates [b, r, c, ch] mean picture b, row r, column c, channel ch, all counted from 0. So [2, 1, 0, 2] is picture 2 (Sea), row 1, column 0, blue: the value is 128. Now find [1, 0, 2, 0] yourself. (Answer: picture 1, row 0, column 2, red: 0.)

Other data, same idea (sizes chosen to keep the numbers small)
DataHow it becomes numbersOne exampleA batch of 8
SoundAir-pressure readings, often 16,000 per second for speechOne second: (16000,)(8, 16000)
Weather7 days × 3 readings (temperature, rain, wind)(7, 3)(8, 7, 3)
TextEach token becomes an ID number, then an embedding vector (4 numbers here; real models use hundreds or thousands)Five tokens: (5, 4)(8, 5, 4)
VideoA sequence of color pictures (frames)10 frames of 2 × 3: (10, 2, 3, 3)(8, 10, 2, 3, 3)

Going further · optional, grades 9+

What models do with tensors

Skip this on a first read. It shows the arithmetic that runs inside a neural network, on tensors like the ones above.

Dot product

Multiply pairs, then add

Inputs x = [1, 2, 3], weights w = [4, 0, −1]:

1×4 + 2×0 + 3×(−1) = 4 + 0 − 3 = 1

A single artificial neuron does exactly this weighted sum, then adds a bias and passes the result through an activation function. See the neuron on How AI works.

Matrix multiplication

Many dot products at once

Put two neurons’ weights in the rows of a (2, 3) matrix, W = [[4, 0, −1], [1, 1, 1]]. Then W times x = [1, 2, 3] is two dot products: [1, 6], because 1 + 2 + 3 = 6.

Shapes: (2, 3) times (3,) gives (2,). The inner lengths must match (3 and 3). One layer of a neural network is mostly this operation, run on a whole batch at once.

Data type and memory

How many bytes?

Every tensor stores one kind of number, its data type. The page’s batch as 32-bit decimals (float32, 4 bytes each): 72 × 4 = 288 bytes. As 8-bit whole numbers (uint8, 0 to 255, 1 byte each): 72 bytes.

One 224 × 224 color picture in float32: 150,528 × 4 = 602,112 bytes, about 0.6 MB. A batch of 32 is about 19.3 MB, which is why batch size is limited by the computer’s memory.

Embeddings

A vector with a meaning

An embedding is a vector a model has learned so that similar things (such as related words) get similar numbers. A language model keeps all its token embeddings in one matrix, one row per token. So an embedding can be a vector stored in a tensor, but most tensors are not embeddings: the pixel batch above is raw measurement, not something learned. See turning words into places on a map.

4 · Common mix-ups

Four ideas about tensors that trip people up

Each one comes with what to think instead.

  • Mix-up: A tensor is a kind of AI, or a special AI chip.

    Better: A tensor is a way to store numbers. Some chips are named after tensors because they are built to do tensor arithmetic fast, but the tensor itself is just the data.

  • Mix-up: More axes means more numbers.

    Better: Shape (72,) and shape (4, 2, 3, 3) both hold 72 numbers. Axes describe the arrangement; multiply the shape to count the values.

  • Mix-up: A tensor with shape (2, 3, 3) is a 3D object.

    Better: It is a flat picture with three axes; the third one is color. People also call a list of 768 numbers a “768-dimensional vector,” although it has one axis. “Dimension” can mean either, so ask which.

  • Mix-up: The AI tensor and the physics tensor are the same idea.

    Better: They share a name and the idea of several axes. A mathematical tensor also follows rules for how its values change when the coordinates change; an array in AI software does not have to.

5 · Check yourself

Three quick questions

Work them out on paper before you open each one.

  1. A set of handwritten digits is stored with shape (10, 28, 28, 1) in batch, height, width, channel order. How many axes does it have, how many values, and what is inside?

    Check your answer

    4 axes and 10 × 28 × 28 × 1 = 7,840 values: a batch of 10 gray pictures, each 28 pixels tall and 28 wide, with one brightness channel.

  2. Your friend says shapes (3, 4) and (4, 3) are the same, because both hold 12 numbers. Are they right?

    Check your answer

    Same count, different arrangement: 3 rows of 4 versus 4 rows of 3. Coordinates [2, 3] exist in the first (row 2, column 3) but not in the second, which has only columns 0, 1 and 2. Software treats them as different shapes.

  3. A model expects channels-first input, (N, C, H, W). What shape should this page’s batch of four 2 × 3 color pictures have?

    Check your answer

    (4, 3, 2, 3). The 72 numbers are the same, but they have to be rearranged into the new order, not just relabeled. Try “Channels first” in the explorer and watch the storage bars regroup by color.

In the real world

An application, a limit and a link

Application

A plant-photo app

An app that names plants from photos resizes every photo to the shape its model expects, for example (224, 224, 3), scales the values to between 0 and 1, and stacks photos into a batch. Feed it the wrong shape and the software usually stops with a “shape mismatch” error.

Limitation

Numbers record light, not meaning

Nothing in the tensor says “sunset.” A model has to learn which patterns of numbers go with which labels. Some mistakes even run without an error: the OpenCV library loads pictures in blue, green, red order, so feeding them to a model trained on red, green, blue keeps the shape but silently swaps the colors.

Connection

Same arrays, everywhere

Whatever the data, a neural network sees tensors: pixels for image generators, embeddings for chatbots, sensor readings for farm robots. Classical models usually take a simpler table: one row per example.

For younger learners

A grades 3–5 version

No tensor words needed: pictures as number codes, on graph paper.

20 minutes · pairs

Number pictures

  1. Each student draws a simple picture on a 4 × 4 grid using three shades: 0 = white, 1 = gray, 2 = black.
  2. They write it as four rows of four numbers, fold the drawing away and pass only the numbers to a partner.
  3. The partner redraws the picture from the numbers. Compare: did it survive the trip?
  4. Ask: what would you need to send for a picture in color? (More than one number per square.)

Exit question

Count the numbers

“How many numbers did your 4 × 4 picture need? How many if every square needed a red, a green and a blue number?” (16, then 48.)

Then play Lightprint: a spectrum is a picture that is only one row tall.

Honest limits

What this page leaves out

What this page leaves out

How tensors are laid out in memory (strides), how operations stretch a small tensor to fit a bigger one (broadcasting), how libraries track gradients through tensors during training, sparse tensors, how files such as JPEG are decoded into pixels, and where tensors live (main memory or a graphics chip). It also skips the full mathematics of tensors, which needs linear algebra.