# Compute and efficiency

- Link: https://arqiv.si/ai/compute-and-efficiency/
- Published: 2026-10-10T18:31:30-04:00
- Author: Steven Ryan

[Skip to content](https://arqiv.si/ai/compute-and-efficiency/#main-content)

[arqiv](https://arqiv.si/)

[For educators](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

[Learn AI](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

[Games & labs](https://arqiv.si/games/)[Research](https://arqiv.si/research/)[Blog](https://arqiv.si/blog/)
[About](https://arqiv.si/about/)

[Search or ask](https://arqiv.si/ai/topics/)[Newsletter](https://arqiv.si/newsletter/)
[ARQIV.FM ↗](https://arqiv.fm/)

Menu

For educators

[For educators: start page →](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn AI

[Learn AI: start page →](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

 * [Games & labs](https://arqiv.si/games/)
 * [Research](https://arqiv.si/research/)
 * [Blog](https://arqiv.si/blog/)
 * [About](https://arqiv.si/about/)

[Get the newsletter](https://arqiv.si/newsletter/)

 1. [ARQIV / ADVANCED TRACK](https://arqiv.si/ai/topics/)
 2. [**01**Compute & efficiency](https://arqiv.si/ai/compute-and-efficiency/)
 3. [**02**Deploy & maintain](https://arqiv.si/ai/model-lifecycle/)
 4. [**03**Generative families](https://arqiv.si/ai/generative-models/)
 5. [**04**Knowledge & causality](https://arqiv.si/ai/knowledge-and-causality/)
 6. [**05**Robotics](https://arqiv.si/ai/robotics/)
 7. [**06**Security & provenance](https://arqiv.si/ai/security-and-provenance/)
 8. [**07**Advanced learning](https://arqiv.si/ai/advanced-learning/)

Go deeper · Advanced track

# Why does AI need so much computing, _and how can it need less?_

Because a neural network is mostly multiplication. To write one word, a large language
model multiplies and adds with each of its billions of learned numbers, and must
hold them all in fast memory. It can need less by storing each number with fewer
bits (quantization), by training a small model to imitate a big one (distillation),
and by choosing where it runs: your device or the cloud.

Grades 9+25 minutesMultiplication onlyRounding activity for grades 6–8

What you will be able to do

Estimate the memory a model needs from its parameter count and bits per number; 
explain why GPUs and accelerators suit neural networks; and weigh batching, quantization,
distillation, and edge versus cloud for a given job.

Before you start

Read first: [How does a picture become numbers?](https://arqiv.si/ai/tensors/) (
data types and bytes) and [Training vs inference](https://arqiv.si/ai/training-and-inference/).
Helpful: [Neural-network families](https://arqiv.si/ai/neural-network-families/).
This is an optional advanced page: nothing on the main route depends on it.

1 · You have already seen this

## Rounding the prices _in your head_

When you add up a basket of shopping in your head, you probably round each price
first.

Rounded numbers are shorter, so they are easier to hold in your head and quicker
to add. The total ends up a little away from the real one, and if you round too 
roughly it moves a lot.

A neural network does a far bigger version of that sum. To write one word, a large
language model multiplies and adds with each of its billions of learned numbers,
and it has to hold them all in fast memory. **Compute and efficiency** is about 
how much memory and arithmetic that takes, and how to make it need less. This page
shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

## From learned numbers _to a written word_

Running a model in five steps. The next three sections go deeper into steps 1 to
4 in order, and the lab lets you do steps 3 and 4 yourself. After the lab, three
sections go deeper into step 5.

What a model needs to run

### Five steps

 1. 1 · Count the parametersA model is mostly its learned numbers, called **parameters**.
    A 7-billion-parameter model has 7 billion of them.
 2. 2 · Multiply and addFor each token, a word or part of one, the chip does about 
    two operations per parameter: a multiply and an add. GPUs and accelerators do many
    of these at once.
 3. 3 · Store each numberEach parameter takes a fixed size: 4 bytes at 32-bit, 2 at
    16-bit, 1 at 8-bit and half a byte at 4-bit. Memory for the weights is parameters
    × bits ÷ 8 bytes.
 4. 4 · Fit in working memoryThe weights must fit in fast working memory, not just 
    storage. The chip reads every weight once per token, so fewer bytes usually means
    faster answers too.
 5. 5 · Shrink it or move itIf it does not fit or is too slow, store each number with
    fewer bits (**quantization**), train a smaller model to imitate it (**distillation**),
    or choose where it runs: on the device or in the cloud.

Going deeper · step 1

## Same question, _two very different trips_

Optional advanced page: it builds on the tensors and training pages, and the main
route does not need it.

Turn on airplane mode and type a message. Your phone’s keyboard still suggests the
next word, instantly. Ask a large chatbot something with no signal, and nothing 
happens. The keyboard’s model is small enough to live on the phone; the chatbot’s
model lives in a data center.

The difference comes down to two numbers: how many **parameters** (learned numbers)
a model has, and how many bytes each one takes. They decide how much memory it needs
and how much arithmetic, or **compute**, it uses for every word.

Each layer of a neural network multiplies its inputs by a matrix of weights and 
adds up the results, as on the [tensors page](https://arqiv.si/ai/tensors/). A rule
of thumb from language-model research: for a dense model, one token (a word or part
of one) takes about two operations per parameter, a multiply and an add. Sparse 
mixture-of-experts models use only the parameters that are active for each token.
A 7-billion-parameter model does about 14 billion operations per token.

Going deeper · step 2

## CPU, GPU, accelerator: _who does the multiplying?_

All three are chips that compute. They are built for different kinds of work.

A **CPU** (central processing unit) runs your operating system and apps. It has 
a few powerful cores, each fast at long, branching, step-by-step instructions.

A **GPU** (graphics processing unit) was built to color millions of pixels, running
the same small calculation on many numbers at once with thousands of simpler cores.
Neural-network work has exactly that shape: millions of multiply-adds that do not
depend on each other.

An **accelerator** is a chip, or part of one, designed mainly for this job. Google’s
first Tensor Processing Unit, described in 2017, was built around a 256 × 256 grid
of 8-bit multiply-add units. Many phones and laptops now include a small “neural
processing unit” for the same reason: less flexibility, more speed per watt.

CPU

a few powerful cores

step-by-step, branching work

GPU or accelerator

many simple cores (not to scale)

the same multiply-add on many numbers

A sketch, not a chip diagram. Each small core is simpler and slower than a CPU core;
the speed comes from how many there are.

Predict first

Job A follows a chain of clues, where each step needs the answer to the one before.
Job B multiplies a million unrelated pairs of numbers. Which gains more from a GPU?

Show the answer

**Job B.** Independent multiplications can be spread over thousands of cores at 
once. Job A must wait for each answer, so extra cores sit idle; a fast CPU core 
may finish it sooner. Neural networks look like Job B.

Going deeper · steps 3 and 4

## Parameters × bytes = _memory_

Before a model can answer anything, its parameters have to be loaded into fast working
memory.

Each parameter is stored in a fixed-size data type: **32-bit** (float32) takes 4
bytes, 16-bit 2 bytes, 8-bit 1 byte, 4-bit half a byte. Memory for the weights is
parameters × bits ÷ 8 bytes.

It has to fit in working memory (RAM, or a GPU’s own memory), not just storage. 
A phone can store a 14 GB file and still be unable to run it.

Memory also sets the speed. To write each token, the chip reads every weight once,
and that reading is often slower than the arithmetic. Fewer bytes per weight means
less to read, so a smaller format is usually faster too, and moving less data saves
energy. Training needs far more memory than running; “Going further” puts a number
on it.

Worked example

### A 7-billion-parameter model at 16-bit and at 4-bit

 1. The ruleMemory for the weights = parameters × bits per number ÷ 8 bytes. Here 1
    GB = 1,000,000,000 bytes.
 2. 16-bit7,000,000,000 × 16 ÷ 8 = 7,000,000,000 × 2 = 14,000,000,000 bytes = 14 GB.
 3. 4-bit7,000,000,000 × 4 ÷ 8 = 7,000,000,000 × 0.5 = 3,500,000,000 bytes = 3.5 GB.
    16 ÷ 4 = 4, so the 4-bit version is four times smaller.
 4. The fine printReal 4-bit formats also store a scale for each small group of weights.
    One 16-bit scale per 64 weights adds 16 ÷ 64 = 0.25 bits per weight: 7,000,000,000
    × 4.25 ÷ 8 ≈ 3.72 GB.
 5. Arithmetic per tokenAbout 2 operations per parameter (one multiply, one add): 2
    × 7 billion = 14 billion operations for every token, at either precision. Quantization
    shrinks the bytes, not the number of multiply-adds.
 6. Speed (illustrative)Say memory delivers a made-up 100 GB per second, and every 
    weight is read once per token. 16-bit: 14 ÷ 100 = 0.14 s per token, about 7 tokens
    per second. 4-bit: 3.5 ÷ 100 = 0.035 s, about 28 per second.

Weights only: running a model also needs working memory for the conversation. Computers
often report GiB (1,073,741,824 bytes), so 14 GB shows as about 13.0 GiB. The memory
speed is made up.

3 · Try it

## Fit the model _on the device_

Choose a model size and a number format, and see which made-up device has room for
it.

Lab

### Does it fit?

Pick a parameter count and bits per number. The lab multiplies them, divides by 
8 and compares the result with three devices. Then read what lower precision may
cost.

 A calculator in your browser. The device limits are made up for teaching, not the
specs of real products.

1 · Model size (parameters)

2 · Bits per number

Memory for the weights

 * **Phone**4 GB for the model (made up)
 * **Laptop**16 GB for the model (made up)
 * **Cloud server**320 GB for the model (made up)

**Challenges:** find the biggest model that fits the phone. Which setting puts the
13-billion model on the laptop? Can the 400-billion model fit on one cloud server
at all?

The interactive version needs JavaScript. Here is the same calculation for every
setting: memory for the weights (parameters × bits ÷ 8 bytes), and which made-up
device has enough memory for it. Made-up limits: phone 4 GB, laptop 16 GB, cloud
server 320 GB.

Weights in GB, and the made-up devices they fit on| Parameters | 32-bit | 16-bit | 8-bit | 4-bit | 
| 100 million | 0.4 GB
Phone, Laptop, Cloud server | 0.2 GBPhone, Laptop, Cloud server | 0.1 GBPhone, Laptop, Cloud server | 0.05 GBPhone, Laptop, Cloud server | 
| 1 billion | 4 GBPhone, Laptop, Cloud server | 2 GBPhone, Laptop, Cloud server | 1 GBPhone, Laptop, Cloud server | 0.5 GBPhone, Laptop, Cloud server | 
| 3 billion | 12 GBLaptop, Cloud server | 6 GBLaptop, Cloud server | 3 GBPhone, Laptop, Cloud server | 1.5 GBPhone, Laptop, Cloud server | 
| 7 billion | 28 GBCloud server | 14 GBLaptop, Cloud server | 7 GBLaptop, Cloud server | 3.5 GBPhone, Laptop, Cloud server | 
| 13 billion | 52 GBCloud server | 26 GBCloud server | 13 GBLaptop, Cloud server | 6.5 GBLaptop, Cloud server | 
| 70 billion | 280 GBCloud server | 140 GBCloud server | 70 GBCloud server | 35 GBCloud server | 
| 400 billion | 1,600 GBnone | 800 GBnone | 400 GBnone | 200 GBCloud server |

 * 32-bit: a common full-precision format. Large models usually train with mixed
   precision: 16-bit or lower math, with some values such as optimizer state kept
   in 32-bit. Rarely used just to run a large model, because 16-bit usually works
   about as well at half the memory.
 * 16-bit: the common format for running large models. Quality is usually indistinguishable
   from 32-bit.
 * 8-bit: often only a small loss in quality, but it depends on the model and the
   task, so it has to be measured.
 * 4-bit: each number can take only 16 values. A noticeable loss is more likely,
   especially for smaller models; careful rounding methods reduce it. Test before
   you trust it.

Batch size

## Why the cloud serves people _in groups_

A batch is a group of inputs processed together.

In training, the **batch** is how many examples the model sees before each update;
it is limited by memory, because every example’s in-between results must be kept.
When a model serves many people, the batch is how many requests share one step.

Batching is how a cloud server gets its money’s worth: the weights are read once
per step and used for every request in the batch, so 32 requests get 32 times the
work from the same reading. The price is waiting for a batch to form, and each request
still needs its own working memory. On your phone the batch is nearly always 1, 
one more reason on-device models are kept small.

Needing less

## Quantization and distillation: _two ways to shrink_

One stores the same model with rougher numbers. The other trains a different, smaller
model.

Same model, fewer bits

### Quantization

**Quantization** stores each weight with fewer bits: 32 → 16 → 8 → 4. A 4-bit number
has only 2^(4) = 16 codes, against 256 for 8 bits. Each weight is rounded to the
nearest code times a shared scale (table below). The design and parameter count 
stay the same; only the precision drops.

The cost depends on the model, task and method, so it must be measured. A 2017 Google
study ran image models in 8-bit whole numbers on phone chips with a small loss in
accuracy. At 4 bits a loss is more likely to show, especially in small models.

Small student, big teacher

### Distillation

**Distillation** trains a small _student_ to imitate a big _teacher_, learning from
the teacher’s full probabilities (Hinton and colleagues, 2015). If the teacher says
80% wolf, 18% husky, 2% car, the student learns that wolves and huskies look alike,
which the label “wolf” never says.

The DistilBERT authors (2019) report a model 40% smaller and 60% faster than its
teacher, keeping 97% of its score on language-understanding tests. The student has
its own design, not the teacher’s parameters, and tends to inherit its mistakes.

Five made-up weights rounded two ways| Weight (made up) | 8-bit code | 8-bit value | 4-bit code | 4-bit value | 
| 0.82 | 114 | 0.817 | 6 | 0.78 | 
| −0.37 | −52 | −0.373 | −3 | −0.39 | 
| 0.05 | 7 | 0.050 | 0 | 0.00 | 
| −0.91 | −127 | −0.910 | −7 | −0.91 | 
| 0.44 | 61 | 0.437 | 3 | 0.39 |

Code = weight ÷ scale, rounded; value = code × scale. Scale = 0.91 ÷ 127 ≈ 0.00717
for 8 bits, 0.91 ÷ 7 = 0.13 for 4 bits (codes −7 to 7 keep zero exact). Biggest 
error: about 0.003 at 8 bits, 0.05 at 4 bits.

Predict first

A model goes from 16-bit to 4-bit and needs a quarter of the memory. Does it become
four times worse?

Show the answer

**Usually not.** Memory falls exactly 4 times; quality does not follow that rule.
Many models lose little at 8 bits and some lose little at 4, but some tasks and 
small models suffer more, so test the shrunk model on its real job. See [Evaluating AI](https://arqiv.si/ai/evaluating-ai/).

Where it runs

## Edge or cloud: _the trade-offs_

“Edge” means on the device in your hand or in the room. “Cloud” means on servers
in a data center, reached over the internet.

Edge or cloud: seven trade-offs| Trade-off | Edge (on your device) | Cloud (in a data center) | 
| Latency (waiting time) | No trip over the network. Quick for small models; a big model on a weak chip can be slow. | The request travels there and back, and may queue at busy times; a powerful chip can then answer fast. | 
| Privacy | Data can stay on the device, if the app is built that way. | Data is sent to and processed on someone else’s computers, under their policy. | 
| Cost | Paid once, with the device. No bill per question. | Paid per use or by subscription; the provider pays for chips and electricity. | 
| Energy | Uses your battery. Small chips are designed to sip power. | Data center electricity and cooling, shared across many users through batching. | 
| Model size | Limited by the device’s memory: small, quantized or distilled models. | The largest models, split across many chips if needed. | 
| Works offline? | Yes. | No. | 
| Updates | A new model must reach every device. | Change it once on the server. |

Many products split the work: a small on-device model handles quick, private jobs
such as next-word suggestions, and harder requests go to the cloud. On-device is
not automatically private, though; the app may still send data elsewhere. And every
saving costs something: a smaller or rougher model fits more places but may make
more mistakes.

Predict first

A wildlife camera in a forest has no internet and a small battery. It should keep
only photos with an animal. Edge or cloud, and what would you do to the model?

Show the answer

**Edge**: there is no connection. Pick a small model, perhaps distilled from a bigger
one, quantize it to 8 bits, and test it on real camera photos, including night shots,
to see what the shrinking cost.

Going further · optional

## Why training needs _even more memory_

Optional numbers for anyone who wants to see the gap between running and training.

One common setup trains with 16-bit weights and an optimizer called Adam. The ZeRO
paper (Samyam Rajbhandari and colleagues, 2019) counts the memory per parameter:
2 bytes for the 16-bit weight, 2 for its gradient, and 12 for the optimizer’s records(
a 32-bit copy of the weight plus two running averages, 4 bytes each). That is 16
bytes per parameter, before any activations.

For 7 billion parameters: 7,000,000,000 × 16 = 112,000,000,000 bytes = 112 GB, eight
times the 14 GB needed to run the same model at 16-bit. Training also does more 
arithmetic: the scaling-law paper by Jared Kaplan and colleagues (2020) estimates
about 6 operations per parameter for every training token, three times the 2 needed
to run it. That is why large models are trained across many accelerators at once.

4 · Common mix-ups

## Easy ideas that _turn out wrong_

Four common beliefs about chips, memory and shrinking models, each with what to 
think instead.

 * **Mix-up:** A 4-bit model is four times worse than a 16-bit one.
 * **Better:** It is four times smaller. Quality usually drops much less, but it
   must be measured on the task.
 * **Mix-up:** A GPU is just a faster CPU.
 * **Better:** It is a different design: many simple cores doing the same operation
   at once. For step-by-step work, a CPU is often faster.
 * **Mix-up:** If the model file fits in my phone’s storage, the phone can run it.
 * **Better:** It must fit in working memory (RAM), which is much smaller and shared
   with everything else.
 * **Mix-up:** Distillation copies the big model’s parameters.
 * **Better:** The student has its own smaller design and is trained to match the
   teacher’s outputs.

5 · Check yourself

## Three _quick questions_

Answer in your head or on paper before you open each one.

 1.  A model has 3 billion parameters. How much memory do its weights need at 16-bit
     and at 8-bit? Which version fits a device that can give a model 4 GB?
 2.  Check your answer
 3.  16-bit: 3,000,000,000 × 2 bytes = **6 GB**. 8-bit: 3,000,000,000 × 1 byte = **
     3 GB**. Only the 8-bit version fits, with 1 GB left for working memory.
 4.  A cloud service starts handling requests in batches of 64 instead of one at a 
     time. Why is that cheaper per person, and what might a user notice?
 5.  Check your answer
 6.  The weights are read from memory once per step and used for all 64 requests, so
     the same chip, memory reads and electricity do far more useful work. A user may
     wait a moment for a batch to fill, and each request still needs its own working
     memory, so the batch cannot grow forever.
 7.  A school wants a writing-feedback tool that works offline and never sends student
     writing anywhere. Edge or cloud? What would it give up, and which techniques help?
 8.  Check your answer
 9.  **Edge.** The model must fit the school devices’ memory, so it will be smaller
     and possibly less capable than a cloud model. Quantization and distillation help
     it fit. It also uses battery, updates must reach every device, and someone should
     check that the app itself sends nothing away.

For younger learners

## A _grades 6–8_ rounding activity

No chips and no code: the same trade-off, with pencils.

20 minutes · pairs

### How rough can the numbers get?

 1. Give each pair a card with five made-up “weights” (such as 0.82, −0.37, 0.05, −
    0.91, 0.44) and five “inputs” (such as 3, 1, 4, 1, 5).
 2. Multiply each weight by its input and add the five results. That is one neuron.
 3. Round the weights to one decimal place and do it again; then round to the nearest
    0.5; then to whole numbers.
 4. How far did the answer move each time? Which rounding is still “good enough,” and
    how many digits did it save?

Exit question

### Say it in one sentence

“What do you lose and what do you gain when you store numbers with fewer digits?”
Good answers name both sides: “It takes less space and is quicker to work with, 
but the answer moves a bit, and too much rounding changes it a lot.”

Honest limits

## What this page _leaves out_

What this page leaves out

Chip design; splitting a model across many chips; pruning (deleting weights); mixture-
of-experts models, which use only some parameters per token (see [Neural-network families](https://arqiv.si/ai/neural-network-families/));
the working memory long conversations need; and data center energy figures, which
change fast. No chip names or speeds are given, because they date quickly.

Where to go next

## Keep _going_

Next in the advanced track

### What happens after a model is trained?

Versions, monitoring, drift and rollback once a model is in use.

Measure the cost of shrinking

### Evaluating AI

Test a quantized or distilled model on the job it will actually do.

Different designs, different costs

### Neural-network families

Including mixture of experts, which uses only part of the model for each token.

The memory long chats need

### Inside a large language model

KV cache, speculative decoding and vLLM, plus LoRA and QLoRA for cheap fine-tuning,
with two labs.

Sources

## Where this comes from

 1. [Norman Jouppi and colleagues, “In-Datacenter Performance Analysis of a Tensor Processing Unit,” 2017 (arXiv:1704.04760): an accelerator built around a grid of 8-bit multiply-add units.](https://arxiv.org/abs/1704.04760)
 2. [Jared Kaplan and colleagues, “Scaling Laws for Neural Language Models,” 2020 (arXiv:2001.08361): about 2 operations per parameter per token to run a model, about 6 to train it.](https://arxiv.org/abs/2001.08361)
 3. [Benoit Jacob and colleagues, “Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference,” 2017 (arXiv:1712.05877): 8-bit models on phone chips.](https://arxiv.org/abs/1712.05877)
 4. [Geoffrey Hinton, Oriol Vinyals and Jeff Dean, “Distilling the Knowledge in a Neural Network,” 2015 (arXiv:1503.02531): students learning from a teacher’s probabilities.](https://arxiv.org/abs/1503.02531)
 5. [Victor Sanh and colleagues, “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter,” 2019 (arXiv:1910.01108).](https://arxiv.org/abs/1910.01108)
 6. [Samyam Rajbhandari and colleagues, “ZeRO: Memory Optimizations Toward Training Trillion Parameter Models,” 2019 (arXiv:1910.02054): 16 bytes per parameter for mixed-precision training with Adam.](https://arxiv.org/abs/1910.02054)

AI words on this page

[Compute](https://arqiv.si/teachers/ai-vocabulary/#compute), [Parameters](https://arqiv.si/teachers/ai-vocabulary/#parameters),
[Accelerator](https://arqiv.si/teachers/ai-vocabulary/#accelerator), [Quantization](https://arqiv.si/teachers/ai-vocabulary/#quantization),
[Distillation](https://arqiv.si/teachers/ai-vocabulary/#distillation), [Batch](https://arqiv.si/teachers/ai-vocabulary/#batch),
[Inference](https://arqiv.si/teachers/ai-vocabulary/#inference), [Edge AI](https://arqiv.si/teachers/ai-vocabulary/#edge-ai),
[Latency](https://arqiv.si/teachers/ai-vocabulary/#latency), [Tensor](https://arqiv.si/teachers/ai-vocabulary/#tensor),
[Token](https://arqiv.si/teachers/ai-vocabulary/#token), [Hyperparameter](https://arqiv.si/teachers/ai-vocabulary/#hyperparameter),
[Large language model](https://arqiv.si/teachers/ai-vocabulary/#large-language-model)

All definitions live in one place: [AI words in plain language →](https://arqiv.si/teachers/ai-vocabulary/)

Review status

Written by ARQIV. Last checked October 2026. Independent subject review and classroom
pilots are still to come, so treat the activities as drafts and [tell us](https://arqiv.si/ai/compute-and-efficiency/info@arqiv.fm?subject=AI%20foundations%20correction)
about anything that is wrong or unclear.

[← Previous**All AI topics**](https://arqiv.si/ai/topics/)[Next →**Deploy & maintain**](https://arqiv.si/ai/model-lifecycle/)

[arqiv](https://arqiv.si/)

A little understanding changes everything.

[LISTEN: ARQIV.FM, OUR WEEKLY SHOW ↗](https://arqiv.fm/)

Educators

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & kits](https://arqiv.si/teachers/resources/)
 * [Curriculum](https://arqiv.si/teachers/curriculum/)
 * [Labs](https://arqiv.si/teachers/labs/)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

For your role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn

 * [Learn AI](https://arqiv.si/ai/topics/)
 * [Games & labs](https://arqiv.si/games/)
 * [Future of work](https://arqiv.si/future-of-work/)
 * [Research](https://arqiv.si/research/)
 * [Headlines](https://arqiv.si/news/)

ARQIV

 * [About & corrections](https://arqiv.si/about/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [Newsletter](https://arqiv.si/newsletter/)
 * [Privacy](https://arqiv.si/privacy-policy/)
 * [Get in touch](https://arqiv.si/ai/compute-and-efficiency/info@arqiv.fm)

Classroom activities collect no student data.[info@arqiv.fm](https://arqiv.si/ai/compute-and-efficiency/info@arqiv.fm)
