Go deeper · Build with AI
Inside a large language model: trained, tuned and served
A chatbot answer is the last step of a long chain. First a model is given a shape (its architecture). Then it is trained in stages, from reading the internet to being rewarded for good answers. Builders adapt it to their own jobs with cheap add-ons, and finally it is served to millions of people by software designed to squeeze every drop out of the chips. This page walks the chain, naming the terms you will meet in AI news and job ads.
What you will be able to do
Tell apart dense, mixture-of-experts, state-space, diffusion and small language models; put SFT, RLHF, DPO, GRPO and RLVR in the right order and say what each adds; explain test-time scaling; estimate what LoRA saves; and explain why the KV cache, speculative decoding and vLLM make serving cheaper.
Before you start
Read first: What is a transformer? and Training vs inference. Helpful: Compute and efficiency and Reinforcement learning. This is an optional advanced page.
You have already seen this
The reply that appears a few words at a time
When you ask a chatbot something, the answer often fills the screen piece by piece instead of all at once.
The model really does write its reply one small piece at a time. Each piece is called a token: a whole word, part of a word or a bit of punctuation. Every new token depends on the ones before it.
That reply is the last step of a long chain inside a large language model. The chapters below follow the chain: the basic words, the different shapes a model can have, the stages of training that turn it from autocomplete into an assistant, how it can think longer before it answers, how builders adapt it cheaply for their own jobs, and how it is served to millions of people at once. Two labs let you work out the numbers yourself.
01 · The basics in one minute
Four words everything else builds on
If these are new, follow the links first. The rest of the page assumes them.
Tokenization
Text is cut into pieces called tokens: whole words, parts of words or punctuation. A model reads and writes tokens, and companies charge per token. In English, 1,000 tokens is roughly 750 words.
Embeddings
Each token becomes a long list of numbers, a point in a “meaning space” where related ideas sit close together. See Embeddings.
Attention
Each token looks back at the earlier tokens and decides how much each one matters for what comes next, which is how “it” gets linked to the right noun.
Transformer
The 2017 design that stacks attention and small neural networks dozens of times. Almost every large language model since is built on it. See What is a transformer?
02 · Model architectures
Not every model has the same shape
The plain transformer is expensive in two ways: every parameter works on every token, and attention gets slower as the text gets longer. Four families attack those costs differently.
| Shape | The idea | What it buys | What it costs |
|---|---|---|---|
| Dense transformer | Every parameter is used for every token. | Simple and well understood. | Cost grows with model size; attention slows down on long text. |
| Mixture of experts (MoE) | Each layer holds many small “expert” networks. A router sends each token to only a few of them. DeepSeek-V3 (2024) has 671 billion parameters but uses about 37 billion per token. | The knowledge of a huge model at the running cost of a much smaller one. | All experts must still sit in memory, and balancing the router is tricky. |
| State-space models (Mamba) | Instead of looking back at every earlier token, the model carries a fixed-size running summary that it updates token by token (Gu and Dao, 2023). | Cost grows in a straight line with length, and memory stays fixed, so very long inputs are cheap. | A fixed summary can lose exact details. Many 2025–2026 models are hybrids that mix Mamba layers with some attention layers. |
| Diffusion language models | Borrowed from image generators: start with a rough or masked draft of the whole answer and refine all of it over several passes, instead of writing one token at a time. | Many tokens are produced in parallel, so they can be very fast. | Newer and less proven on long, careful reasoning. Examples: Inception’s Mercury and Google’s Gemini Diffusion research model (2025). |
| Small language models (SLMs) | Models with a few billion parameters or fewer, often trained on carefully chosen or synthetic data, or distilled from a big model. | Run on a phone or laptop, privately and cheaply. Good for narrow jobs. | Less general knowledge; more likely to fail on unusual questions. |
Predict first
A mixture-of-experts model has 600 billion parameters but uses 30 billion per token. Does it need the memory of a 30-billion model or a 600-billion model?
Show the answer
The memory of the 600-billion model. Different tokens go to different experts, so all of them must be ready. What gets smaller is the computing per token, which is what makes each answer faster and cheaper. Use the numbers on Compute and efficiency to see how much memory that means.
03 · Training and alignment
From autocomplete to assistant in four stages
A freshly pre-trained model only continues text. Each later stage teaches it something the previous one could not.
- Pre-trainingPredict the next token across trillions of tokens of text. No labels are needed (self-supervised), and this is where almost all of the knowledge and cost goes.
- SFTSupervised fine-tuning: thousands to millions of example prompts with good answers written or checked by people. The model learns the format of being helpful.
- Preference tuningShow two answers, learn which one people prefer. Done with RLHF and a reward model, or more directly with DPO.
- RL with checkable rewardsRLVR: practise on problems whose answers a program can check, such as maths and code with tests, often with GRPO. This builds reasoning.
RLHF and reward models
Reinforcement learning from human feedback. People rank several answers to the same prompt. A separate reward model is trained to predict those rankings, then the chatbot is trained with reinforcement learning to score highly on it. OpenAI’s InstructGPT paper (2022) showed this made a small model preferred over a model 100 times bigger.
Weak spot: reward hacking. The model learns what pleases the reward model, such as long, confident answers, rather than what is actually good.
DPO
Direct preference optimization (Rafailov and colleagues, 2023) skips the reward model and the reinforcement-learning loop. It uses the same “answer A beats answer B” pairs and adjusts the model directly with an ordinary training step that makes preferred answers more likely.
Why it spread: simpler, cheaper and more stable to run, so open-model builders adopted it widely.
RLVR
Reinforcement learning with verifiable rewards. Instead of a human or a reward model judging, a program does: is the final number right, do the unit tests pass, does the answer follow the format? The name comes from the Tülu 3 paper (Lambert and colleagues, 2024).
Strength: a checker is hard to fool. Limit: it only works where answers can be checked, so essays and advice still need human preferences.
GRPO
Group relative policy optimization (DeepSeekMath, Shao and colleagues, 2024). For each problem the model writes a group of answers, say 16. Each answer is scored, and it is rewarded for being better than its group’s average. No separate “critic” model is needed, which saves memory.
DeepSeek-R1 (January 2025) used GRPO with checkable rewards to train long step-by-step reasoning, and many open reasoning models copied the recipe.
Predict first
You want to train a model to write kinder feedback on student essays. Can you use RLVR?
Show the answer
Not on its own. “Kind and useful” cannot be checked by a simple program the way a maths answer can. You would use SFT on good examples plus preference tuning (RLHF or DPO) from teachers’ judgments. You might add small checkable rules, such as “mentions one strength and one next step”, as part of the reward.
04 · Advanced reasoning
Thinking longer instead of growing bigger
For years, the way to a smarter model was a bigger model. Since 2024 there is a second dial: let the model spend more computing on each question.
Chain-of-thought. In 2022, Google researchers showed that asking a model to write out its steps before the answer (“Let’s think step by step”) improved maths and logic results. The steps give the model room to work, because each token it writes can be used by the tokens after it.
Reasoning models. OpenAI’s o1 (September 2024) and DeepSeek-R1 (January 2025) were trained with reinforcement learning to produce long internal chains of thought: trying an approach, checking it, backing up. Most frontier models now have a thinking mode, often with a “thinking budget” you can set.
Test-time scaling means spending more computing when the question is asked (at “test time”) rather than during training. The main ways: think longer; write several answers and take a vote (self-consistency); or write many answers and let a checker pick the best (best-of-N). A 2024 study by Snell and colleagues found that, for some problems, extra thinking time beat a model 14 times larger. The trade-off is that every thinking token costs money and time.
RLMs. The letters are used two ways. Often they simply mean reasoning language models, as above. Since late 2025 they also mean recursive language models (Zhang, Kraska and Khattab, MIT): instead of pasting a huge document into the prompt, the model keeps it in a small coding workspace, writes code to look through it, and calls copies of itself on the pieces. The authors report handling inputs about 100 times longer than the model’s context window. Ask which one a writer means.
Where it helps
Maths, coding, science problems, planning and puzzles: jobs with a right answer that rewards checking your work.
Be careful
The written “thoughts” are not a reliable window into how the model reached its answer. Anthropic researchers reported in 2025 that reasoning models often left out a hint they had actually used. Check the answer, not just how convincing the steps look.
05 · Fine-tuning and adaptation
Teach an old model a new job, cheaply
Most builders never train a model from scratch. They start from an existing one and adapt it. Try the cheapest option first.
| Option | What changes | Use it when |
|---|---|---|
| Better prompt | Nothing in the model. You change the instructions and examples. | Always try first. See Talking to AI. |
| Retrieval (RAG) | Nothing in the model. Relevant documents are fetched and added to the prompt. | The model needs facts it does not know, or that change. See Retrieval and RAG. |
| PEFT (LoRA, QLoRA) | A small set of new numbers is trained; the original weights stay frozen. | The model needs a new behaviour: a style, a format, a narrow skill, a specialist vocabulary. |
| Full fine-tune | Every weight can change. | Large, very different jobs, with lots of data and budget. |
| Distillation | A new, smaller model is trained to copy a big model’s answers. | You need a fast, cheap model for one job. See Compute and efficiency. |
PEFT stands for parameter-efficient fine-tuning, a family of methods that train only a tiny fraction of the numbers. The most popular is LoRA, low-rank adaptation (Hu and colleagues, Microsoft, 2021). Next to a big grid of frozen weights, LoRA adds two thin grids. If the big grid is 4,096 × 4,096 (about 16.8 million numbers), the thin ones might be 4,096 × 8 and 8 × 4,096 (about 65,000 numbers). The 8 is called the rank. Only the thin grids are trained, and they can be saved as a small file, swapped in and out like a plug-in. On GPT-3, the paper cut the trainable numbers by 10,000 times and memory needs by 3 times, with similar quality.
QLoRA (Dettmers and colleagues, 2023) adds one trick: store the frozen model in 4-bit numbers while training the LoRA grids. That made it possible to fine-tune a 65-billion-parameter model on a single 48 GB graphics card.
Synthetic data is training data written by a model rather than by people: practice questions, worked solutions, rewritten examples. It is now part of almost every training pipeline, and small models such as Microsoft’s Phi series were trained heavily on it. The risk: a model trained mostly on model-written text, generation after generation, can lose the rare and unusual cases; researchers call this model collapse (Shumailov and colleagues, Nature, 2024). Builders mix in real data and filter carefully.
Lab 1
How much does LoRA save?
Pick a model size and a LoRA rank. The lab counts the numbers you would train and estimates the graphics-card memory for a full fine-tune, LoRA and QLoRA.
A calculator in your browser. Layer counts and widths match well-known open models; the memory rule of thumb ignores some working memory, so real numbers run higher.
1 · Model size
2 · LoRA rank
about 16 bytes per parameter
16-bit frozen model plus adapters
4-bit frozen model plus adapters
How it counts: LoRA is added to the four attention grids in every layer, each treated as square (width × width). Trainable numbers = layers × 4 × 2 × rank × width. Full fine-tuning with the common Adam optimizer needs about 16 bytes per parameter (weights, gradients and optimizer state).
The interactive version needs JavaScript. With rank 8: a 1-billion model (16 layers, width 2,048) trains about 2.1 million numbers; an 8-billion model (32 layers, width 4,096) about 8.4 million; a 70-billion model (80 layers, width 8,192) about 42 million, well under 0.1% of the model. Rough memory: full fine-tune 16, 128 and 1,120 GB; LoRA about 2, 16 and 140 GB; QLoRA about 0.5, 4 and 35 GB.
06 · Inference optimization
Serving millions of people without melting the budget
Training happens once. Inference, answering questions, happens billions of times, so most engineering effort now goes here.
Answering has two phases. In prefill, the model reads your whole prompt at once, which keeps the chip busy. In decode, it writes the answer one token at a time, and for each token it must read all of its weights from memory again. Decode is usually limited by memory speed, not by maths, which is why the tricks below work.
KV cache
Attention compares each new token with every earlier one, using two lists of numbers per token called keys and values. Instead of recomputing them for the whole conversation at every step, the server keeps them: the key-value (KV) cache. It saves huge amounts of computing, but it grows with every token and every user, and for long chats it can need more memory than the model itself.
Quantization
Store each weight, and often the KV cache too, with fewer bits: 8 or 4 instead of 16. Less memory to read means faster decode. Quality usually drops a little and must be measured. Worked examples: Compute and efficiency.
Speculative decoding
A small, fast “draft” model guesses the next several tokens. The big model checks all of them in a single pass and keeps the ones it agrees with (Leviathan and colleagues, Google; Chen and colleagues, DeepMind; 2023). Because the big model has the final say, the answer is the same as without the trick, just often two to three times faster.
Like a student drafting a sentence and a teacher approving several words at a glance.
vLLM and batching
vLLM is a free, open-source serving engine started at UC Berkeley. Its key idea, PagedAttention (Kwon and colleagues, 2023), stores the KV cache in small fixed-size pages, like a computer’s operating system manages memory, so little is wasted and many more users fit at once. With continuous batching, new requests join the batch the moment another finishes. Similar engines include SGLang and NVIDIA TensorRT-LLM; llama.cpp and Ollama run models on laptops.
Lab 2
How big is the KV cache?
Choose a model, a conversation length and the number of people using the server at once. Compare the cache with the memory for the model’s weights.
A calculator in your browser, using the published layer and head counts of two open 8- and 70-billion-parameter models at 16-bit. Real servers also use tricks that shrink these numbers.
1 · Model
2 · Tokens in each conversation
3 · People at the same time
How it counts: 2 (a key and a value) × layers × key-value heads × numbers per head × 2 bytes. The 8-billion model has 32 layers and 8 key-value heads of 128 numbers; the 70-billion model has 80 layers and the same heads. Sharing key-value heads between attention heads (grouped-query attention) already makes these caches about 4 to 8 times smaller than older designs.
The interactive version needs JavaScript. The 8-billion model needs about 128 KB of cache per token, so one 128,000-token conversation needs about 17 GB, more than its 16 GB of weights. The 70-billion model needs about 320 KB per token: 100 people at 8,000 tokens each need about 260 GB of cache, nearly double its 140 GB of weights.
Predict first
Why do AI companies often charge less for input tokens that repeat a prompt they saw a moment ago?
Show the answer
Because they can keep that prompt’s KV cache and skip the prefill work. This is called prompt caching; the next page, AI in production, shows how builders arrange prompts to get the discount.
07 · Frontier research
Beyond next-token prediction
Some researchers think language models alone will not reach human-like understanding. The main alternatives are covered on What’s next for AI.
World models
Models that learn how the world changes, for example by watching video, so they can predict and plan as well as describe. Important for robots. See Robotics.
JEPA
Joint-embedding predictive architecture, proposed by Yann LeCun in 2022. It predicts the meaning of a hidden part of an image or video, not every pixel, as in Meta’s I-JEPA (2023) and V-JEPA 2 (2025).
Multimodal and self-supervised
One model that reads and writes text, images, audio and video, trained mostly by hiding parts of its own data and predicting them, with no human labels. See Machine learning.
Common mix-ups
What people often get wrong about LLMs
Four ideas that trip people up, and what to think instead.
Mix-up: Fine-tuning is how you teach a model new facts.
Better: Fine-tuning mostly teaches behaviour and style. For facts that change, retrieval is cheaper and easier to update.
Mix-up: A model with 600 billion parameters is always slower than one with 70 billion.
Better: Not if it is a mixture of experts that uses only a fraction of them per token.
Mix-up: The thinking text shows exactly how a reasoning model decided.
Better: It helps the model, but it can leave things out. Judge the answer.
Mix-up: Speculative decoding makes answers worse to make them faster.
Better: The big model checks every drafted token, so the output matches what it would have written anyway.
Check yourself
Four quick questions
Answer in your head or on paper before you open each one.
Put these in the usual order and say what each adds: RLVR, pre-training, SFT, DPO.
Check your answer
Pre-training (knowledge and language from next-token prediction) → SFT (the format of helpful answers from examples) → DPO (prefer the answers people like better) → RLVR (practise problems a program can check, building reasoning). Real pipelines repeat and mix stages, but this is the logic.
A school wants a model that writes report-card comments in its house style. It has 2,000 old comments and a small budget. Which adaptation would you try, and why not a full fine-tune?
Check your answer
First a good prompt with a few examples. If that is not enough, LoRA or QLoRA on the 2,000 comments: it teaches a style cheaply, the adapter is a small file, and the base model stays unchanged. A full fine-tune needs far more memory and risks the model forgetting other skills. Remove student names from the data first.
What is test-time scaling, and what is its price?
Check your answer
Spending more computing when a question is answered: thinking longer, or writing several answers and picking or voting. The price is more tokens, so more cost and a longer wait for each answer.
A server runs out of memory when many people have long conversations, even though the model fits easily. What is filling the memory, and name two fixes.
Check your answer
The KV cache, which grows with every token of every conversation. Fixes: page it efficiently (vLLM’s PagedAttention), quantize the cache, limit or compact conversation length, use a model with fewer key-value heads, or add more memory.
Honest limits
What this page leaves out
What this page leaves out
The maths of attention and training; how models are split across thousands of chips; exact recipes, which companies rarely publish; long-context tricks such as rotary position embeddings; and safety training methods such as Anthropic’s Constitutional AI. Numbers in the labs are rules of thumb for teaching. Model names and records change fast; this page reflects November 2026.
Where to go next
Keep going
Next in Build with AI
AI in production
Context, memory, agents, cloud, cost, monitoring, tests and security once a model is part of a real app.
The numbers behind serving
Compute and efficiency
Memory, GPUs, quantization and distillation, with a fit-the-model lab.
The idea behind RLHF and GRPO
Reinforcement learning
Learning from rewards, from games to chatbots.
Sources
Where this comes from
- DeepSeek-AI, “DeepSeek-V3 Technical Report,” 2024 (arXiv:2412.19437): mixture of experts, 671B total and 37B active parameters per token.
- Albert Gu and Tri Dao, “Mamba: Linear-Time Sequence Modeling with Selective State Spaces,” 2023 (arXiv:2312.00752).
- Long Ouyang and colleagues, “Training language models to follow instructions with human feedback” (InstructGPT), 2022 (arXiv:2203.02155).
- Rafael Rafailov and colleagues, “Direct Preference Optimization: Your Language Model is Secretly a Reward Model,” 2023 (arXiv:2305.18290).
- Nathan Lambert and colleagues, “Tülu 3: Pushing Frontiers in Open Language Model Post-Training,” 2024 (arXiv:2411.15124): reinforcement learning with verifiable rewards.
- Zhihong Shao and colleagues, “DeepSeekMath,” 2024 (arXiv:2402.03300): group relative policy optimization (GRPO).
- DeepSeek-AI, “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning,” 2025 (arXiv:2501.12948).
- Jason Wei and colleagues, “Chain-of-Thought Prompting Elicits Reasoning in Large Language Models,” 2022 (arXiv:2201.11903).
- Charlie Snell and colleagues, “Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters,” 2024 (arXiv:2408.03314).
- Alex L. Zhang, Tim Kraska and Omar Khattab, “Recursive Language Models,” 2025 (arXiv:2512.24601).
- Anthropic, “Reasoning models don’t always say what they think,” April 2025.
- Edward Hu and colleagues, “LoRA: Low-Rank Adaptation of Large Language Models,” 2021 (arXiv:2106.09685).
- Tim Dettmers and colleagues, “QLoRA: Efficient Finetuning of Quantized LLMs,” 2023 (arXiv:2305.14314).
- Ilia Shumailov and colleagues, “AI models collapse when trained on recursively generated data,” Nature, 2024.
- Yaniv Leviathan, Matan Kalman and Yossi Matias, “Fast Inference from Transformers via Speculative Decoding,” 2023 (arXiv:2211.17192).
- Woosuk Kwon and colleagues, “Efficient Memory Management for Large Language Model Serving with PagedAttention” (vLLM), 2023 (arXiv:2309.06180).
Last checked
November 2026. Product and model names change quickly; the ideas on this page change slowly. Spotted an error? Tell us.