# Inside a large language model: how LLMs are trained, tuned and served

- Link: https://arqiv.si/ai/inside-llms/
- Published: 2026-11-07T21:24:51-05:00
- Author: Steven Ryan

[Skip to content](https://arqiv.si/ai/inside-llms/#main-content)

[arqiv](https://arqiv.si/)

[For educators](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

[Learn AI](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

[Games & labs](https://arqiv.si/games/)[Research](https://arqiv.si/research/)[Blog](https://arqiv.si/blog/)
[About](https://arqiv.si/about/)

[Search or ask](https://arqiv.si/ai/topics/)[Newsletter](https://arqiv.si/newsletter/)
[ARQIV.FM ↗](https://arqiv.fm/)

Menu

For educators

[For educators: start page →](https://arqiv.si/teachers/)

Teach

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & printable kits](https://arqiv.si/teachers/resources/)
 * [Classroom labs](https://arqiv.si/teachers/labs/)
 * [Free 20-minute lesson](https://arqiv.si/teachers/free-ai-lesson/)

Plan

 * [Curriculum guide](https://arqiv.si/teachers/curriculum/)
 * [AI Learning Center](https://arqiv.si/teachers/ai-literacy/)
 * [Future of work unit](https://arqiv.si/future-of-work/)
 * [What the research says](https://arqiv.si/teachers/what-works/)
 * [AI education worldwide](https://arqiv.si/teachers/around-the-world/)

Understand

 * [Think First method](https://arqiv.si/teachers/think-first/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

By role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn AI

[Learn AI: start page →](https://arqiv.si/ai/topics/)

Start

 * [All AI topics](https://arqiv.si/ai/topics/)
 * [The AI journey, from zero](https://arqiv.si/ai/)
 * [Using AI well](https://arqiv.si/ai/talking-to-ai/)
 * [AI builds](https://arqiv.si/ai-builds/)

AI in the world

 * [Space](https://arqiv.si/space/)
 * [Weather](https://arqiv.si/weather/)
 * [Health](https://arqiv.si/health/)
 * [Engineering](https://arqiv.si/engineering/)
 * [Farming](https://arqiv.si/farming/)
 * [Quantum](https://arqiv.si/quantum/)

What it means

 * [Future of work](https://arqiv.si/future-of-work/)
 * [Your AI path](https://arqiv.si/ai/your-path/)
 * [Research, explained](https://arqiv.si/research/)

 * [Games & labs](https://arqiv.si/games/)
 * [Research](https://arqiv.si/research/)
 * [Blog](https://arqiv.si/blog/)
 * [About](https://arqiv.si/about/)

[Get the newsletter](https://arqiv.si/newsletter/)

[AI BASICS](https://arqiv.si/ai/)[USING AI](https://arqiv.si/ai/talking-to-ai/)[AI BUILDS](https://arqiv.si/ai-builds/)
[GAMES](https://arqiv.si/games/)AI IN →[SPACE](https://arqiv.si/space/)[WEATHER](https://arqiv.si/weather/)
[HEALTH](https://arqiv.si/health/)[ENGINEERING](https://arqiv.si/engineering/)[FARMING](https://arqiv.si/farming/)
[QUANTUM](https://arqiv.si/quantum/)[RESEARCH](https://arqiv.si/research/)

 1. [ARQIV / BUILD WITH AI](https://arqiv.si/ai/topics/)
 2. [**01**How agents are built](https://arqiv.si/ai/how-agents-are-built/)
 3. [**02**Inside an LLM](https://arqiv.si/ai/inside-llms/)
 4. [**03**AI in production](https://arqiv.si/ai/ai-in-production/)
 5. [**04**Deploy and maintain](https://arqiv.si/ai/model-lifecycle/)

Go deeper · Build with AI

# Inside a large language model: _trained, tuned and served_

A chatbot answer is the last step of a long chain. First a model is given a shape(
its architecture). Then it is trained in stages, from reading the internet to being
rewarded for good answers. Builders adapt it to their own jobs with cheap add-ons,
and finally it is served to millions of people by software designed to squeeze every
drop out of the chips. This page walks the chain, naming the terms you will meet
in AI news and job ads.

Grades 9+ and adults35 minutesTwo labsMultiplication only

What you will be able to do

Tell apart dense, mixture-of-experts, state-space, diffusion and small language 
models; put SFT, RLHF, DPO, GRPO and RLVR in the right order and say what each adds;
explain test-time scaling; estimate what LoRA saves; and explain why the KV cache,
speculative decoding and vLLM make serving cheaper.

Before you start

Read first: [What is a transformer?](https://arqiv.si/ai/transformers/) and [Training vs inference](https://arqiv.si/ai/training-and-inference/).
Helpful: [Compute and efficiency](https://arqiv.si/ai/compute-and-efficiency/) and
[Reinforcement learning](https://arqiv.si/ai/reinforcement-learning/). This is an
optional advanced page.

You have already seen this

## The reply that appears _a few words at a time_

When you ask a chatbot something, the answer often fills the screen piece by piece
instead of all at once.

The model really does write its reply one small piece at a time. Each piece is called
a **token**: a whole word, part of a word or a bit of punctuation. Every new token
depends on the ones before it.

That reply is the last step of a long chain inside a **large language model**. The
chapters below follow the chain: the basic words, the different shapes a model can
have, the stages of training that turn it from autocomplete into an assistant, how
it can think longer before it answers, how builders adapt it cheaply for their own
jobs, and how it is served to millions of people at once. Two labs let you work 
out the numbers yourself.

01 · The basics in one minute

## Four words _everything else builds on_

If these are new, follow the links first. The rest of the page assumes them.

Tokens

### Tokenization

Text is cut into pieces called tokens: whole words, parts of words or punctuation.
A model reads and writes tokens, and companies charge per token. In English, 1,000
tokens is roughly 750 words.

Meaning

### Embeddings

Each token becomes a long list of numbers, a point in a “meaning space” where related
ideas sit close together. See [Embeddings](https://arqiv.si/ai/embeddings/).

Context

### Attention

Each token looks back at the earlier tokens and decides how much each one matters
for what comes next, which is how “it” gets linked to the right noun.

Engine

### Transformer

The 2017 design that stacks attention and small neural networks dozens of times.
Almost every large language model since is built on it. See [What is a transformer?](https://arqiv.si/ai/transformers/)

02 · Model architectures

## Not every model has _the same shape_

The plain transformer is expensive in two ways: every parameter works on every token,
and attention gets slower as the text gets longer. Four families attack those costs
differently.

Five shapes of language model| Shape | The idea | What it buys | What it costs | 
 | Dense transformer | Every parameter is used for every token. | Simple and well understood. | Cost grows with model size; attention slows down on long text. | 
 | Mixture of experts (MoE) | Each layer holds many small “expert” networks. A **router** sends each token to only a few of them. DeepSeek-V3 (2024) has 671 billion parameters but uses about 37 billion per token. | The knowledge of a huge model at the running cost of a much smaller one. | All experts must still sit in memory, and balancing the router is tricky. | 
 | State-space models (Mamba) | Instead of looking back at every earlier token, the model carries a fixed-size running summary that it updates token by token (Gu and Dao, 2023). | Cost grows in a straight line with length, and memory stays fixed, so very long inputs are cheap. | A fixed summary can lose exact details. Many 2025–2026 models are **hybrids** that mix Mamba layers with some attention layers. | 
 | Diffusion language models | Borrowed from image generators: start with a rough or masked draft of the whole answer and refine all of it over several passes, instead of writing one token at a time. | Many tokens are produced in parallel, so they can be very fast. | Newer and less proven on long, careful reasoning. Examples: Inception’s Mercury and Google’s Gemini Diffusion research model (2025). | 
 | Small language models (SLMs) | Models with a few billion parameters or fewer, often trained on carefully chosen or [synthetic](https://arqiv.si/ai/inside-llms/#adapt) data, or distilled from a big model. | Run on a phone or laptop, privately and cheaply. Good for narrow jobs. | Less general knowledge; more likely to fail on unusual questions. |

Predict first

A mixture-of-experts model has 600 billion parameters but uses 30 billion per token.
Does it need the memory of a 30-billion model or a 600-billion model?

Show the answer

The memory of the **600-billion** model. Different tokens go to different experts,
so all of them must be ready. What gets smaller is the **computing per token**, 
which is what makes each answer faster and cheaper. Use the numbers on [Compute and efficiency](https://arqiv.si/ai/compute-and-efficiency/)
to see how much memory that means.

03 · Training and alignment

## From autocomplete to assistant _in four stages_

A freshly pre-trained model only continues text. Each later stage teaches it something
the previous one could not.

 1. **Pre-training**Predict the next token across trillions of tokens of text. No labels
    are needed (self-supervised), and this is where almost all of the knowledge and
    cost goes.
 2. **SFT**Supervised fine-tuning: thousands to millions of example prompts with good
    answers written or checked by people. The model learns the _format_ of being helpful.
 3. **Preference tuning**Show two answers, learn which one people prefer. Done with
    RLHF and a reward model, or more directly with DPO.
 4. **RL with checkable rewards**RLVR: practise on problems whose answers a program
    can check, such as maths and code with tests, often with GRPO. This builds reasoning.

2022

### RLHF and reward models

**Reinforcement learning from human feedback.** People rank several answers to the
same prompt. A separate **reward model** is trained to predict those rankings, then
the chatbot is trained with reinforcement learning to score highly on it. OpenAI’s
InstructGPT paper (2022) showed this made a small model preferred over a model 100
times bigger.

Weak spot: **reward hacking**. The model learns what pleases the reward model, such
as long, confident answers, rather than what is actually good.

2023

### DPO

**Direct preference optimization** (Rafailov and colleagues, 2023) skips the reward
model and the reinforcement-learning loop. It uses the same “answer A beats answer
B” pairs and adjusts the model directly with an ordinary training step that makes
preferred answers more likely.

Why it spread: simpler, cheaper and more stable to run, so open-model builders adopted
it widely.

2024

### RLVR

**Reinforcement learning with verifiable rewards.** Instead of a human or a reward
model judging, a program does: is the final number right, do the unit tests pass,
does the answer follow the format? The name comes from the Tülu 3 paper (Lambert
and colleagues, 2024).

Strength: a checker is hard to fool. Limit: it only works where answers can be checked,
so essays and advice still need human preferences.

2024

### GRPO

**Group relative policy optimization** (DeepSeekMath, Shao and colleagues, 2024).
For each problem the model writes a group of answers, say 16. Each answer is scored,
and it is rewarded for being _better than its group’s average_. No separate “critic”
model is needed, which saves memory.

DeepSeek-R1 (January 2025) used GRPO with checkable rewards to train long step-by-
step reasoning, and many open reasoning models copied the recipe.

Predict first

You want to train a model to write kinder feedback on student essays. Can you use
RLVR?

Show the answer

Not on its own. “Kind and useful” cannot be checked by a simple program the way 
a maths answer can. You would use SFT on good examples plus preference tuning (RLHF
or DPO) from teachers’ judgments. You might add small checkable rules, such as “
mentions one strength and one next step”, as part of the reward.

04 · Advanced reasoning

## Thinking longer _instead of growing bigger_

For years, the way to a smarter model was a bigger model. Since 2024 there is a 
second dial: let the model spend more computing on each question.

**Chain-of-thought.** In 2022, Google researchers showed that asking a model to 
write out its steps before the answer (“Let’s think step by step”) improved maths
and logic results. The steps give the model room to work, because each token it 
writes can be used by the tokens after it.

**Reasoning models.** OpenAI’s o1 (September 2024) and DeepSeek-R1 (January 2025)
were trained with reinforcement learning to produce long internal chains of thought:
trying an approach, checking it, backing up. Most frontier models now have a thinking
mode, often with a “thinking budget” you can set.

**Test-time scaling** means spending more computing when the question is asked (
at “test time”) rather than during training. The main ways: think longer; write 
several answers and take a vote (self-consistency); or write many answers and let
a checker pick the best (best-of-N). A 2024 study by Snell and colleagues found 
that, for some problems, extra thinking time beat a model 14 times larger. The trade-
off is that every thinking token costs money and time.

**RLMs.** The letters are used two ways. Often they simply mean **reasoning language
models**, as above. Since late 2025 they also mean **recursive language models**(
Zhang, Kraska and Khattab, MIT): instead of pasting a huge document into the prompt,
the model keeps it in a small coding workspace, writes code to look through it, 
and calls copies of itself on the pieces. The authors report handling inputs about
100 times longer than the model’s context window. Ask which one a writer means.

Where it helps

Maths, coding, science problems, planning and puzzles: jobs with a right answer 
that rewards checking your work.

Be careful

The written “thoughts” are not a reliable window into how the model reached its 
answer. Anthropic researchers reported in 2025 that reasoning models often left 
out a hint they had actually used. Check the answer, not just how convincing the
steps look.

05 · Fine-tuning and adaptation

## Teach an old model _a new job, cheaply_

Most builders never train a model from scratch. They start from an existing one 
and adapt it. Try the cheapest option first.

Cheapest first| Option | What changes | Use it when | 
 | Better prompt | Nothing in the model. You change the instructions and examples. | Always try first. See [Talking to AI](https://arqiv.si/ai/talking-to-ai/). | 
 | Retrieval (RAG) | Nothing in the model. Relevant documents are fetched and added to the prompt. | The model needs _facts_ it does not know, or that change. See [Retrieval and RAG](https://arqiv.si/ai/retrieval-and-rag/). | 
 | PEFT (LoRA, QLoRA) | A small set of new numbers is trained; the original weights stay frozen. | The model needs a new _behaviour_: a style, a format, a narrow skill, a specialist vocabulary. | 
 | Full fine-tune | Every weight can change. | Large, very different jobs, with lots of data and budget. | 
 | Distillation | A new, smaller model is trained to copy a big model’s answers. | You need a fast, cheap model for one job. See [Compute and efficiency](https://arqiv.si/ai/compute-and-efficiency/). |

**PEFT** stands for _parameter-efficient fine-tuning_, a family of methods that 
train only a tiny fraction of the numbers. The most popular is **LoRA**, low-rank
adaptation (Hu and colleagues, Microsoft, 2021). Next to a big grid of frozen weights,
LoRA adds two thin grids. If the big grid is 4,096 × 4,096 (about 16.8 million numbers),
the thin ones might be 4,096 × 8 and 8 × 4,096 (about 65,000 numbers). The 8 is 
called the **rank**. Only the thin grids are trained, and they can be saved as a
small file, swapped in and out like a plug-in. On GPT-3, the paper cut the trainable
numbers by 10,000 times and memory needs by 3 times, with similar quality.

**QLoRA** (Dettmers and colleagues, 2023) adds one trick: store the frozen model
in 4-bit numbers while training the LoRA grids. That made it possible to fine-tune
a 65-billion-parameter model on a single 48 GB graphics card.

**Synthetic data** is training data written by a model rather than by people: practice
questions, worked solutions, rewritten examples. It is now part of almost every 
training pipeline, and small models such as Microsoft’s Phi series were trained 
heavily on it. The risk: a model trained mostly on model-written text, generation
after generation, can lose the rare and unusual cases; researchers call this **model
collapse** (Shumailov and colleagues, _Nature_, 2024). Builders mix in real data
and filter carefully.

Lab 1

### How much does LoRA save?

Pick a model size and a LoRA rank. The lab counts the numbers you would train and
estimates the graphics-card memory for a full fine-tune, LoRA and QLoRA.

 A calculator in your browser. Layer counts and widths match well-known open models;
the memory rule of thumb ignores some working memory, so real numbers run higher.

1 · Model size

2 · LoRA rank

Trained by LoRA**–**

Full fine-tune memory**–**

about 16 bytes per parameter

LoRA memory**–**

16-bit frozen model plus adapters

QLoRA memory**–**

4-bit frozen model plus adapters

How it counts: LoRA is added to the four attention grids in every layer, each treated
as square (width × width). Trainable numbers = layers × 4 × 2 × rank × width. Full
fine-tuning with the common Adam optimizer needs about 16 bytes per parameter (weights,
gradients and optimizer state).

The interactive version needs JavaScript. With rank 8: a 1-billion model (16 layers,
width 2,048) trains about 2.1 million numbers; an 8-billion model (32 layers, width
4,096) about 8.4 million; a 70-billion model (80 layers, width 8,192) about 42 million,
well under 0.1% of the model. Rough memory: full fine-tune 16, 128 and 1,120 GB;
LoRA about 2, 16 and 140 GB; QLoRA about 0.5, 4 and 35 GB.

06 · Inference optimization

## Serving millions of people _without melting the budget_

Training happens once. Inference, answering questions, happens billions of times,
so most engineering effort now goes here.

Answering has two phases. In **prefill**, the model reads your whole prompt at once,
which keeps the chip busy. In **decode**, it writes the answer one token at a time,
and for each token it must read all of its weights from memory again. Decode is 
usually limited by memory speed, not by maths, which is why the tricks below work.

Memory

### KV cache

Attention compares each new token with every earlier one, using two lists of numbers
per token called **keys** and **values**. Instead of recomputing them for the whole
conversation at every step, the server keeps them: the key-value (KV) cache. It 
saves huge amounts of computing, but it grows with every token and every user, and
for long chats it can need more memory than the model itself.

Size

### Quantization

Store each weight, and often the KV cache too, with fewer bits: 8 or 4 instead of
16. Less memory to read means faster decode. Quality usually drops a little and 
must be measured. Worked examples: [Compute and efficiency](https://arqiv.si/ai/compute-and-efficiency/).

Speed

### Speculative decoding

A small, fast “draft” model guesses the next several tokens. The big model checks
all of them in a single pass and keeps the ones it agrees with (Leviathan and colleagues,
Google; Chen and colleagues, DeepMind; 2023). Because the big model has the final
say, the answer is the same as without the trick, just often two to three times 
faster.

Like a student drafting a sentence and a teacher approving several words at a glance.

Software

### vLLM and batching

**vLLM** is a free, open-source serving engine started at UC Berkeley. Its key idea,**
PagedAttention** (Kwon and colleagues, 2023), stores the KV cache in small fixed-
size pages, like a computer’s operating system manages memory, so little is wasted
and many more users fit at once. With **continuous batching**, new requests join
the batch the moment another finishes. Similar engines include SGLang and NVIDIA
TensorRT-LLM; llama.cpp and Ollama run models on laptops.

Lab 2

### How big is the KV cache?

Choose a model, a conversation length and the number of people using the server 
at once. Compare the cache with the memory for the model’s weights.

 A calculator in your browser, using the published layer and head counts of two 
open 8- and 70-billion-parameter models at 16-bit. Real servers also use tricks 
that shrink these numbers.

1 · Model

2 · Tokens in each conversation

3 · People at the same time

Cache per token**–**

KV cache in total**–**

Model weights**–**

How it counts: 2 (a key and a value) × layers × key-value heads × numbers per head
× 2 bytes. The 8-billion model has 32 layers and 8 key-value heads of 128 numbers;
the 70-billion model has 80 layers and the same heads. Sharing key-value heads between
attention heads (grouped-query attention) already makes these caches about 4 to 
8 times smaller than older designs.

The interactive version needs JavaScript. The 8-billion model needs about 128 KB
of cache per token, so one 128,000-token conversation needs about 17 GB, more than
its 16 GB of weights. The 70-billion model needs about 320 KB per token: 100 people
at 8,000 tokens each need about 260 GB of cache, nearly double its 140 GB of weights.

Predict first

Why do AI companies often charge less for input tokens that repeat a prompt they
saw a moment ago?

Show the answer

Because they can keep that prompt’s KV cache and skip the prefill work. This is 
called **prompt caching**; the next page, [AI in production](https://arqiv.si/ai/ai-in-production/#context),
shows how builders arrange prompts to get the discount.

07 · Frontier research

## Beyond _next-token prediction_

Some researchers think language models alone will not reach human-like understanding.
The main alternatives are covered on [What’s next for AI](https://arqiv.si/ai/whats-next/).

### World models

Models that learn how the world changes, for example by watching video, so they 
can predict and plan as well as describe. Important for robots. See [Robotics](https://arqiv.si/ai/robotics/).

### JEPA

Joint-embedding predictive architecture, proposed by Yann LeCun in 2022. It predicts
the _meaning_ of a hidden part of an image or video, not every pixel, as in Meta’s
I-JEPA (2023) and V-JEPA 2 (2025).

### Multimodal and self-supervised

One model that reads and writes text, images, audio and video, trained mostly by
hiding parts of its own data and predicting them, with no human labels. See [Machine learning](https://arqiv.si/ai/machine-learning/).

Common mix-ups

## What people often _get wrong about LLMs_

Four ideas that trip people up, and what to think instead.

 * **Mix-up:** Fine-tuning is how you teach a model new facts.
 * **Better:** Fine-tuning mostly teaches behaviour and style. For facts that change,
   retrieval is cheaper and easier to update.
 * **Mix-up:** A model with 600 billion parameters is always slower than one with
   70 billion.
 * **Better:** Not if it is a mixture of experts that uses only a fraction of them
   per token.
 * **Mix-up:** The thinking text shows exactly how a reasoning model decided.
 * **Better:** It helps the model, but it can leave things out. Judge the answer.
 * **Mix-up:** Speculative decoding makes answers worse to make them faster.
 * **Better:** The big model checks every drafted token, so the output matches what
   it would have written anyway.

Check yourself

## Four _quick questions_

Answer in your head or on paper before you open each one.

 1.  Put these in the usual order and say what each adds: RLVR, pre-training, SFT, 
     DPO.
 2.  Check your answer
 3.  **Pre-training** (knowledge and language from next-token prediction) → **SFT**(
     the format of helpful answers from examples) → **DPO** (prefer the answers people
     like better) → **RLVR** (practise problems a program can check, building reasoning).
     Real pipelines repeat and mix stages, but this is the logic.
 4.  A school wants a model that writes report-card comments in its house style. It
     has 2,000 old comments and a small budget. Which adaptation would you try, and
     why not a full fine-tune?
 5.  Check your answer
 6.  First a good prompt with a few examples. If that is not enough, **LoRA or QLoRA**
     on the 2,000 comments: it teaches a style cheaply, the adapter is a small file,
     and the base model stays unchanged. A full fine-tune needs far more memory and
     risks the model forgetting other skills. Remove student names from the data first.
 7.  What is test-time scaling, and what is its price?
 8.  Check your answer
 9.  Spending more computing when a question is answered: thinking longer, or writing
     several answers and picking or voting. The price is more tokens, so more cost 
     and a longer wait for each answer.
 10. A server runs out of memory when many people have long conversations, even though
     the model fits easily. What is filling the memory, and name two fixes.
 11. Check your answer
 12. The **KV cache**, which grows with every token of every conversation. Fixes: page
     it efficiently (vLLM’s PagedAttention), quantize the cache, limit or compact conversation
     length, use a model with fewer key-value heads, or add more memory.

Honest limits

## What this page _leaves out_

What this page leaves out

The maths of attention and training; how models are split across thousands of chips;
exact recipes, which companies rarely publish; long-context tricks such as rotary
position embeddings; and safety training methods such as Anthropic’s Constitutional
AI. Numbers in the labs are rules of thumb for teaching. Model names and records
change fast; this page reflects November 2026.

Where to go next

## Keep _going_

Next in Build with AI

### AI in production

Context, memory, agents, cloud, cost, monitoring, tests and security once a model
is part of a real app.

The numbers behind serving

### Compute and efficiency

Memory, GPUs, quantization and distillation, with a fit-the-model lab.

The idea behind RLHF and GRPO

### Reinforcement learning

Learning from rewards, from games to chatbots.

Sources

## Where this comes from

 1.  [DeepSeek-AI, “DeepSeek-V3 Technical Report,” 2024 (arXiv:2412.19437): mixture of experts, 671B total and 37B active parameters per token.](https://arxiv.org/abs/2412.19437)
 2.  [Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” 2023 (arXiv:2312.00752).](https://arxiv.org/abs/2312.00752)
 3.  [Long Ouyang and colleagues, “Training language models to follow instructions with human feedback” (InstructGPT), 2022 (arXiv:2203.02155).](https://arxiv.org/abs/2203.02155)
 4.  [Rafael Rafailov and colleagues, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” 2023 (arXiv:2305.18290).](https://arxiv.org/abs/2305.18290)
 5.  [Nathan Lambert and colleagues, “Tülu 3: Pushing Frontiers in Open Language Model Post-Training,” 2024 (arXiv:2411.15124): reinforcement learning with verifiable rewards.](https://arxiv.org/abs/2411.15124)
 6.  [Zhihong Shao and colleagues, “DeepSeekMath,” 2024 (arXiv:2402.03300): group relative policy optimization (GRPO).](https://arxiv.org/abs/2402.03300)
 7.  [DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025 (arXiv:2501.12948).](https://arxiv.org/abs/2501.12948)
 8.  [Jason Wei and colleagues, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” 2022 (arXiv:2201.11903).](https://arxiv.org/abs/2201.11903)
 9.  [Charlie Snell and colleagues, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters,” 2024 (arXiv:2408.03314).](https://arxiv.org/abs/2408.03314)
 10. [Alex L. Zhang, Tim Kraska and Omar Khattab, “Recursive Language Models,” 2025 (arXiv:2512.24601).](https://arxiv.org/abs/2512.24601)
 11. [Anthropic, “Reasoning models don’t always say what they think,” April 2025.](https://www.anthropic.com/research/reasoning-models-dont-say-think)
 12. [Edward Hu and colleagues, “LoRA: Low-Rank Adaptation of Large Language Models,” 2021 (arXiv:2106.09685).](https://arxiv.org/abs/2106.09685)
 13. [Tim Dettmers and colleagues, “QLoRA: Efficient Finetuning of Quantized LLMs,” 2023 (arXiv:2305.14314).](https://arxiv.org/abs/2305.14314)
 14. [Ilia Shumailov and colleagues, “AI models collapse when trained on recursively generated data,” _Nature_, 2024.](https://www.nature.com/articles/s41586-024-07566-y)
 15. [Yaniv Leviathan, Matan Kalman and Yossi Matias, “Fast Inference from Transformers via Speculative Decoding,” 2023 (arXiv:2211.17192).](https://arxiv.org/abs/2211.17192)
 16. [Woosuk Kwon and colleagues, “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM), 2023 (arXiv:2309.06180).](https://arxiv.org/abs/2309.06180)

Last checked

November 2026. Product and model names change quickly; the ideas on this page change
slowly. Spotted an error? [Tell us](https://arqiv.si/about/).

[← Previous**How AI agents are built**](https://arqiv.si/ai/how-agents-are-built/)
[Next →**AI in production**](https://arqiv.si/ai/ai-in-production/)

[arqiv](https://arqiv.si/)

A little understanding changes everything.

[LISTEN: ARQIV.FM, OUR WEEKLY SHOW ↗](https://arqiv.fm/)

Educators

 * [Start here](https://arqiv.si/teachers/)
 * [Lessons & kits](https://arqiv.si/teachers/resources/)
 * [Curriculum](https://arqiv.si/teachers/curriculum/)
 * [Labs](https://arqiv.si/teachers/labs/)
 * [AI vocabulary](https://arqiv.si/teachers/ai-vocabulary/)

For your role

 * [Counselors](https://arqiv.si/counselors/)
 * [School leaders & PD](https://arqiv.si/schools/)
 * [Speaking & workshops](https://arqiv.si/speaking/)
 * [Products](https://arqiv.si/products/)

Learn

 * [Learn AI](https://arqiv.si/ai/topics/)
 * [Games & labs](https://arqiv.si/games/)
 * [Future of work](https://arqiv.si/future-of-work/)
 * [Research](https://arqiv.si/research/)
 * [Headlines](https://arqiv.si/news/)

ARQIV

 * [About & corrections](https://arqiv.si/about/)
 * [Our approach](https://arqiv.si/research/#our-method)
 * [Newsletter](https://arqiv.si/newsletter/)
 * [Privacy](https://arqiv.si/privacy-policy/)
 * [Get in touch](https://arqiv.si/ai/inside-llms/info@arqiv.fm)

Classroom activities collect no student data.[info@arqiv.fm](https://arqiv.si/ai/inside-llms/info@arqiv.fm)
