Go deeper · Advanced track
Why does AI need so much computing, and how can it need less?
Because a neural network is mostly multiplication. To write one word, a large language model multiplies and adds with each of its billions of learned numbers, and must hold them all in fast memory. It can need less by storing each number with fewer bits (quantization), by training a small model to imitate a big one (distillation), and by choosing where it runs: your device or the cloud.
What you will be able to do
Estimate the memory a model needs from its parameter count and bits per number; explain why GPUs and accelerators suit neural networks; and weigh batching, quantization, distillation, and edge versus cloud for a given job.
Before you start
Read first: How does a picture become numbers? (data types and bytes) and Training vs inference. Helpful: Neural-network families. This is an optional advanced page: nothing on the main route depends on it.
1 · You have already seen this
Rounding the prices in your head
When you add up a basket of shopping in your head, you probably round each price first.
Rounded numbers are shorter, so they are easier to hold in your head and quicker to add. The total ends up a little away from the real one, and if you round too roughly it moves a lot.
A neural network does a far bigger version of that sum. To write one word, a large language model multiplies and adds with each of its billions of learned numbers, and it has to hold them all in fast memory. Compute and efficiency is about how much memory and arithmetic that takes, and how to make it need less. This page shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From learned numbers to a written word
Running a model in five steps. The next three sections go deeper into steps 1 to 4 in order, and the lab lets you do steps 3 and 4 yourself. After the lab, three sections go deeper into step 5.
What a model needs to run
Five steps
- 1 · Count the parametersA model is mostly its learned numbers, called parameters. A 7-billion-parameter model has 7 billion of them.
- 2 · Multiply and addFor each token, a word or part of one, the chip does about two operations per parameter: a multiply and an add. GPUs and accelerators do many of these at once.
- 3 · Store each numberEach parameter takes a fixed size: 4 bytes at 32-bit, 2 at 16-bit, 1 at 8-bit and half a byte at 4-bit. Memory for the weights is parameters × bits ÷ 8 bytes.
- 4 · Fit in working memoryThe weights must fit in fast working memory, not just storage. The chip reads every weight once per token, so fewer bytes usually means faster answers too.
- 5 · Shrink it or move itIf it does not fit or is too slow, store each number with fewer bits (quantization), train a smaller model to imitate it (distillation), or choose where it runs: on the device or in the cloud.
Going deeper · step 1
Same question, two very different trips
Optional advanced page: it builds on the tensors and training pages, and the main route does not need it.
Turn on airplane mode and type a message. Your phone’s keyboard still suggests the next word, instantly. Ask a large chatbot something with no signal, and nothing happens. The keyboard’s model is small enough to live on the phone; the chatbot’s model lives in a data center.
The difference comes down to two numbers: how many parameters (learned numbers) a model has, and how many bytes each one takes. They decide how much memory it needs and how much arithmetic, or compute, it uses for every word.
Each layer of a neural network multiplies its inputs by a matrix of weights and adds up the results, as on the tensors page. A rule of thumb from language-model research: for a dense model, one token (a word or part of one) takes about two operations per parameter, a multiply and an add. Sparse mixture-of-experts models use only the parameters that are active for each token. A 7-billion-parameter model does about 14 billion operations per token.
Going deeper · step 2
CPU, GPU, accelerator: who does the multiplying?
All three are chips that compute. They are built for different kinds of work.
A CPU (central processing unit) runs your operating system and apps. It has a few powerful cores, each fast at long, branching, step-by-step instructions.
A GPU (graphics processing unit) was built to color millions of pixels, running the same small calculation on many numbers at once with thousands of simpler cores. Neural-network work has exactly that shape: millions of multiply-adds that do not depend on each other.
An accelerator is a chip, or part of one, designed mainly for this job. Google’s first Tensor Processing Unit, described in 2017, was built around a 256 × 256 grid of 8-bit multiply-add units. Many phones and laptops now include a small “neural processing unit” for the same reason: less flexibility, more speed per watt.
CPU
a few powerful cores
step-by-step, branching work
GPU or accelerator
many simple cores (not to scale)
the same multiply-add on many numbers
Predict first
Job A follows a chain of clues, where each step needs the answer to the one before. Job B multiplies a million unrelated pairs of numbers. Which gains more from a GPU?
Show the answer
Job B. Independent multiplications can be spread over thousands of cores at once. Job A must wait for each answer, so extra cores sit idle; a fast CPU core may finish it sooner. Neural networks look like Job B.
Going deeper · steps 3 and 4
Parameters × bytes = memory
Before a model can answer anything, its parameters have to be loaded into fast working memory.
Each parameter is stored in a fixed-size data type: 32-bit (float32) takes 4 bytes, 16-bit 2 bytes, 8-bit 1 byte, 4-bit half a byte. Memory for the weights is parameters × bits ÷ 8 bytes.
It has to fit in working memory (RAM, or a GPU’s own memory), not just storage. A phone can store a 14 GB file and still be unable to run it.
Memory also sets the speed. To write each token, the chip reads every weight once, and that reading is often slower than the arithmetic. Fewer bytes per weight means less to read, so a smaller format is usually faster too, and moving less data saves energy. Training needs far more memory than running; “Going further” puts a number on it.
Worked example
A 7-billion-parameter model at 16-bit and at 4-bit
- The ruleMemory for the weights = parameters × bits per number ÷ 8 bytes. Here 1 GB = 1,000,000,000 bytes.
- 16-bit7,000,000,000 × 16 ÷ 8 = 7,000,000,000 × 2 = 14,000,000,000 bytes = 14 GB.
- 4-bit7,000,000,000 × 4 ÷ 8 = 7,000,000,000 × 0.5 = 3,500,000,000 bytes = 3.5 GB. 16 ÷ 4 = 4, so the 4-bit version is four times smaller.
- The fine printReal 4-bit formats also store a scale for each small group of weights. One 16-bit scale per 64 weights adds 16 ÷ 64 = 0.25 bits per weight: 7,000,000,000 × 4.25 ÷ 8 ≈ 3.72 GB.
- Arithmetic per tokenAbout 2 operations per parameter (one multiply, one add): 2 × 7 billion = 14 billion operations for every token, at either precision. Quantization shrinks the bytes, not the number of multiply-adds.
- Speed (illustrative)Say memory delivers a made-up 100 GB per second, and every weight is read once per token. 16-bit: 14 ÷ 100 = 0.14 s per token, about 7 tokens per second. 4-bit: 3.5 ÷ 100 = 0.035 s, about 28 per second.
Weights only: running a model also needs working memory for the conversation. Computers often report GiB (1,073,741,824 bytes), so 14 GB shows as about 13.0 GiB. The memory speed is made up.
3 · Try it
Fit the model on the device
Choose a model size and a number format, and see which made-up device has room for it.
Lab
Does it fit?
Pick a parameter count and bits per number. The lab multiplies them, divides by 8 and compares the result with three devices. Then read what lower precision may cost.
A calculator in your browser. The device limits are made up for teaching, not the specs of real products.
1 · Model size (parameters)
2 · Bits per number
Memory for the weights
- Phone4 GB for the model (made up)
- Laptop16 GB for the model (made up)
- Cloud server320 GB for the model (made up)
Challenges: find the biggest model that fits the phone. Which setting puts the 13-billion model on the laptop? Can the 400-billion model fit on one cloud server at all?
The interactive version needs JavaScript. Here is the same calculation for every setting: memory for the weights (parameters × bits ÷ 8 bytes), and which made-up device has enough memory for it. Made-up limits: phone 4 GB, laptop 16 GB, cloud server 320 GB.
| Parameters | 32-bit | 16-bit | 8-bit | 4-bit |
|---|---|---|---|---|
| 100 million | 0.4 GB Phone, Laptop, Cloud server | 0.2 GB Phone, Laptop, Cloud server | 0.1 GB Phone, Laptop, Cloud server | 0.05 GB Phone, Laptop, Cloud server |
| 1 billion | 4 GB Phone, Laptop, Cloud server | 2 GB Phone, Laptop, Cloud server | 1 GB Phone, Laptop, Cloud server | 0.5 GB Phone, Laptop, Cloud server |
| 3 billion | 12 GB Laptop, Cloud server | 6 GB Laptop, Cloud server | 3 GB Phone, Laptop, Cloud server | 1.5 GB Phone, Laptop, Cloud server |
| 7 billion | 28 GB Cloud server | 14 GB Laptop, Cloud server | 7 GB Laptop, Cloud server | 3.5 GB Phone, Laptop, Cloud server |
| 13 billion | 52 GB Cloud server | 26 GB Cloud server | 13 GB Laptop, Cloud server | 6.5 GB Laptop, Cloud server |
| 70 billion | 280 GB Cloud server | 140 GB Cloud server | 70 GB Cloud server | 35 GB Cloud server |
| 400 billion | 1,600 GB none | 800 GB none | 400 GB none | 200 GB Cloud server |
- 32-bit: a common full-precision format. Large models usually train with mixed precision: 16-bit or lower math, with some values such as optimizer state kept in 32-bit. Rarely used just to run a large model, because 16-bit usually works about as well at half the memory.
- 16-bit: the common format for running large models. Quality is usually indistinguishable from 32-bit.
- 8-bit: often only a small loss in quality, but it depends on the model and the task, so it has to be measured.
- 4-bit: each number can take only 16 values. A noticeable loss is more likely, especially for smaller models; careful rounding methods reduce it. Test before you trust it.
Batch size
Why the cloud serves people in groups
A batch is a group of inputs processed together.
In training, the batch is how many examples the model sees before each update; it is limited by memory, because every example’s in-between results must be kept. When a model serves many people, the batch is how many requests share one step.
Batching is how a cloud server gets its money’s worth: the weights are read once per step and used for every request in the batch, so 32 requests get 32 times the work from the same reading. The price is waiting for a batch to form, and each request still needs its own working memory. On your phone the batch is nearly always 1, one more reason on-device models are kept small.
Needing less
Quantization and distillation: two ways to shrink
One stores the same model with rougher numbers. The other trains a different, smaller model.
Same model, fewer bits
Quantization
Quantization stores each weight with fewer bits: 32 → 16 → 8 → 4. A 4-bit number has only 24 = 16 codes, against 256 for 8 bits. Each weight is rounded to the nearest code times a shared scale (table below). The design and parameter count stay the same; only the precision drops.
The cost depends on the model, task and method, so it must be measured. A 2017 Google study ran image models in 8-bit whole numbers on phone chips with a small loss in accuracy. At 4 bits a loss is more likely to show, especially in small models.
Small student, big teacher
Distillation
Distillation trains a small student to imitate a big teacher, learning from the teacher’s full probabilities (Hinton and colleagues, 2015). If the teacher says 80% wolf, 18% husky, 2% car, the student learns that wolves and huskies look alike, which the label “wolf” never says.
The DistilBERT authors (2019) report a model 40% smaller and 60% faster than its teacher, keeping 97% of its score on language-understanding tests. The student has its own design, not the teacher’s parameters, and tends to inherit its mistakes.
| Weight (made up) | 8-bit code | 8-bit value | 4-bit code | 4-bit value |
|---|---|---|---|---|
| 0.82 | 114 | 0.817 | 6 | 0.78 |
| −0.37 | −52 | −0.373 | −3 | −0.39 |
| 0.05 | 7 | 0.050 | 0 | 0.00 |
| −0.91 | −127 | −0.910 | −7 | −0.91 |
| 0.44 | 61 | 0.437 | 3 | 0.39 |
Code = weight ÷ scale, rounded; value = code × scale. Scale = 0.91 ÷ 127 ≈ 0.00717 for 8 bits, 0.91 ÷ 7 = 0.13 for 4 bits (codes −7 to 7 keep zero exact). Biggest error: about 0.003 at 8 bits, 0.05 at 4 bits.
Predict first
A model goes from 16-bit to 4-bit and needs a quarter of the memory. Does it become four times worse?
Show the answer
Usually not. Memory falls exactly 4 times; quality does not follow that rule. Many models lose little at 8 bits and some lose little at 4, but some tasks and small models suffer more, so test the shrunk model on its real job. See Evaluating AI.
Where it runs
Edge or cloud: the trade-offs
“Edge” means on the device in your hand or in the room. “Cloud” means on servers in a data center, reached over the internet.
| Trade-off | Edge (on your device) | Cloud (in a data center) |
|---|---|---|
| Latency (waiting time) | No trip over the network. Quick for small models; a big model on a weak chip can be slow. | The request travels there and back, and may queue at busy times; a powerful chip can then answer fast. |
| Privacy | Data can stay on the device, if the app is built that way. | Data is sent to and processed on someone else’s computers, under their policy. |
| Cost | Paid once, with the device. No bill per question. | Paid per use or by subscription; the provider pays for chips and electricity. |
| Energy | Uses your battery. Small chips are designed to sip power. | Data center electricity and cooling, shared across many users through batching. |
| Model size | Limited by the device’s memory: small, quantized or distilled models. | The largest models, split across many chips if needed. |
| Works offline? | Yes. | No. |
| Updates | A new model must reach every device. | Change it once on the server. |
Many products split the work: a small on-device model handles quick, private jobs such as next-word suggestions, and harder requests go to the cloud. On-device is not automatically private, though; the app may still send data elsewhere. And every saving costs something: a smaller or rougher model fits more places but may make more mistakes.
Predict first
A wildlife camera in a forest has no internet and a small battery. It should keep only photos with an animal. Edge or cloud, and what would you do to the model?
Show the answer
Edge: there is no connection. Pick a small model, perhaps distilled from a bigger one, quantize it to 8 bits, and test it on real camera photos, including night shots, to see what the shrinking cost.
Going further · optional
Why training needs even more memory
Optional numbers for anyone who wants to see the gap between running and training.
One common setup trains with 16-bit weights and an optimizer called Adam. The ZeRO paper (Samyam Rajbhandari and colleagues, 2019) counts the memory per parameter: 2 bytes for the 16-bit weight, 2 for its gradient, and 12 for the optimizer’s records (a 32-bit copy of the weight plus two running averages, 4 bytes each). That is 16 bytes per parameter, before any activations.
For 7 billion parameters: 7,000,000,000 × 16 = 112,000,000,000 bytes = 112 GB, eight times the 14 GB needed to run the same model at 16-bit. Training also does more arithmetic: the scaling-law paper by Jared Kaplan and colleagues (2020) estimates about 6 operations per parameter for every training token, three times the 2 needed to run it. That is why large models are trained across many accelerators at once.
4 · Common mix-ups
Easy ideas that turn out wrong
Four common beliefs about chips, memory and shrinking models, each with what to think instead.
Mix-up: A 4-bit model is four times worse than a 16-bit one.
Better: It is four times smaller. Quality usually drops much less, but it must be measured on the task.
Mix-up: A GPU is just a faster CPU.
Better: It is a different design: many simple cores doing the same operation at once. For step-by-step work, a CPU is often faster.
Mix-up: If the model file fits in my phone’s storage, the phone can run it.
Better: It must fit in working memory (RAM), which is much smaller and shared with everything else.
Mix-up: Distillation copies the big model’s parameters.
Better: The student has its own smaller design and is trained to match the teacher’s outputs.
5 · Check yourself
Three quick questions
Answer in your head or on paper before you open each one.
A model has 3 billion parameters. How much memory do its weights need at 16-bit and at 8-bit? Which version fits a device that can give a model 4 GB?
Check your answer
16-bit: 3,000,000,000 × 2 bytes = 6 GB. 8-bit: 3,000,000,000 × 1 byte = 3 GB. Only the 8-bit version fits, with 1 GB left for working memory.
A cloud service starts handling requests in batches of 64 instead of one at a time. Why is that cheaper per person, and what might a user notice?
Check your answer
The weights are read from memory once per step and used for all 64 requests, so the same chip, memory reads and electricity do far more useful work. A user may wait a moment for a batch to fill, and each request still needs its own working memory, so the batch cannot grow forever.
A school wants a writing-feedback tool that works offline and never sends student writing anywhere. Edge or cloud? What would it give up, and which techniques help?
Check your answer
Edge. The model must fit the school devices’ memory, so it will be smaller and possibly less capable than a cloud model. Quantization and distillation help it fit. It also uses battery, updates must reach every device, and someone should check that the app itself sends nothing away.
For younger learners
A grades 6–8 rounding activity
No chips and no code: the same trade-off, with pencils.
20 minutes · pairs
How rough can the numbers get?
- Give each pair a card with five made-up “weights” (such as 0.82, −0.37, 0.05, −0.91, 0.44) and five “inputs” (such as 3, 1, 4, 1, 5).
- Multiply each weight by its input and add the five results. That is one neuron.
- Round the weights to one decimal place and do it again; then round to the nearest 0.5; then to whole numbers.
- How far did the answer move each time? Which rounding is still “good enough,” and how many digits did it save?
Exit question
Say it in one sentence
“What do you lose and what do you gain when you store numbers with fewer digits?” Good answers name both sides: “It takes less space and is quicker to work with, but the answer moves a bit, and too much rounding changes it a lot.”
Honest limits
What this page leaves out
What this page leaves out
Chip design; splitting a model across many chips; pruning (deleting weights); mixture-of-experts models, which use only some parameters per token (see Neural-network families); the working memory long conversations need; and data center energy figures, which change fast. No chip names or speeds are given, because they date quickly.
Where to go next
Keep going
Next in the advanced track
What happens after a model is trained?
Versions, monitoring, drift and rollback once a model is in use.
Measure the cost of shrinking
Evaluating AI
Test a quantized or distilled model on the job it will actually do.
Different designs, different costs
Neural-network families
Including mixture of experts, which uses only part of the model for each token.
The memory long chats need
Inside a large language model
KV cache, speculative decoding and vLLM, plus LoRA and QLoRA for cheap fine-tuning, with two labs.
Sources
Where this comes from
- Norman Jouppi and colleagues, “In-Datacenter Performance Analysis of a Tensor Processing Unit,” 2017 (arXiv:1704.04760): an accelerator built around a grid of 8-bit multiply-add units.
- Jared Kaplan and colleagues, “Scaling Laws for Neural Language Models,” 2020 (arXiv:2001.08361): about 2 operations per parameter per token to run a model, about 6 to train it.
- Benoit Jacob and colleagues, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” 2017 (arXiv:1712.05877): 8-bit models on phone chips.
- Geoffrey Hinton, Oriol Vinyals and Jeff Dean, “Distilling the Knowledge in a Neural Network,” 2015 (arXiv:1503.02531): students learning from a teacher’s probabilities.
- Victor Sanh and colleagues, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 2019 (arXiv:1910.01108).
- Samyam Rajbhandari and colleagues, “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” 2019 (arXiv:1910.02054): 16 bytes per parameter for mixed-precision training with Adam.
AI words on this page
Compute, Parameters, Accelerator, Quantization, Distillation, Batch, Inference, Edge AI, Latency, Tensor, Token, Hyperparameter, Large language model
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.