Go deeper · Advanced track
How do different generators make something new?
They learn the patterns in many examples, then produce a new example that fits them. They differ in how: one piece at a time, by cleaning up noise, through a contest between two networks, by decoding a short code, or with reversible steps. Each trades speed, quality, variety and control differently, and real systems often combine them.
This is an optional advanced page. You do not need it for the main foundations route.
What you will be able to do
Describe how five generator families make a sample, multiply step probabilities to score a short sequence, and pick a family for a job that needs speed, quality, variety or control.
Before you start
Read first: Why are there different kinds of neural networks?, How sure should an AI be? and How can meaning become a place on a map? Helpful: next-word prediction and creative AI tools.
1 · You have already seen this
Text and pictures nobody made by hand
Many people have asked a chatbot to write something, or typed a sentence into an app and got back a picture.
Ask a chatbot for a short poem and the words appear a few at a time. Type “a city skyline at night” into an image tool and a picture comes back that nobody drew. Each one learned from a huge pile of examples, and each made something new that fits the patterns in that pile.
Programs like these are generative models. They learn which results are likely and then draw a new one, and there are five main ways to do the drawing, from building one piece at a time to cleaning up random static. This page follows them through a game studio’s skylines. It shows how they work, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From many examples to a new one
Making something new in four steps. The next section goes deeper into steps 1 and 2, and the two after it show two different ways to do steps 3 and 4. After the lab, a section covers three more ways.
How a generator works
Four steps
- 1 · Collect many examplesThe model needs a large pile of the kind of thing it should make, such as a few thousand skylines drawn by artists.
- 2 · Learn what is likelyTraining teaches the model which results are typical of the pile and which are not. Nobody has to label the examples.
- 3 · Start from chanceMaking something new begins with random numbers, such as a dice roll for the first piece or a screen of pure static.
- 4 · Build the resultSome models add one piece at a time and never go back. Others clean up the whole thing a little at every step, and some make it in a single pass.
Going deeper · steps 1 and 2
One goal, five routes
Before any family names: what does it mean for a computer to make something new?
A game studio has a few thousand skylines drawn by artists. It wants endless new ones that look as if they belong in the same pile, not copies.
A generative model learns which skylines are likely and which are not: in effect, a probability distribution. Making a new one is sampling: drawing one example from that distribution, like drawing a card from a deck. Randomness is the point: without it, every skyline would be the same.
The five families below do this differently. A family name says how sampling works, not how the layers are wired: most use the network families you already know inside.
Family 1 · going deeper into steps 3 and 4
One piece at a time
An autoregressive model predicts the next piece from the pieces so far, adds it, and repeats.
This is how a chatbot writes: it scores every possible next token, picks one, adds it and predicts again (see next-word prediction). It works for pixels, notes or buildings too.
So a whole result’s probability is the step probabilities multiplied together, which the model can compute exactly. And it needs one step per piece and never goes back: 1,000 tokens take 1,000 steps, and an early mistake stays.
Worked example
Multiplying along a 4-building skyline
- The modelMade-up probabilities: building 1 is 3 floors with chance 0.4. After that, the same height has chance 0.5, one floor up 0.25, one floor down 0.25.
- Skyline 3, 4, 4, 50.4 × 0.25 × 0.5 × 0.25 = 0.0125, or 1 in 80.
- Skyline 3, 3, 3, 30.4 × 0.5 × 0.5 × 0.5 = 0.05, or 1 in 20. No 4-building skyline is more likely.
- Check the totalOnly 95 of the 5 × 5 × 5 × 5 = 625 height lists are allowed, and their probabilities add up to exactly 1.
- Go to 8 buildingsAlways taking the most likely next building gives eight 3s: 0.4 × 0.57 = 0.003125, or 1 in 320. Yet that is not the most likely whole skyline: eight 1s or eight 5s each have about 1 in 75, because an edge height repeats with chance 0.75. Locally best choices need not make the best whole sequence. With 5,275 skylines allowed, a sampler almost never produces the greedy “best” one.
Made-up numbers, checked in Python. Real models multiply the same way.
Family 2 · going deeper into steps 3 and 4
Start from noise, clean it step by step
A diffusion model learns to remove noise, then makes something new by removing noise from pure static.
Training: add a known amount of random noise to a real example and teach a network, the denoiser, to guess the clean version (or the noise). Repeat at every noise level, up to pure static.
Generating runs backward: start from pure noise, ask the denoiser for its guess, move a little toward it, and repeat with less noise. Every step can change any part of the output. The 2020 DDPM paper used 1,000 steps; later methods use far fewer, but each step is a pass through a big network.
In the toy below, the noise 7, 0, 4 around building 2 averages 11 ÷ 3 ≈ 3.67, so the guess is 4 and the building rises from 0 to 1.
3 · Try it
Build a skyline two ways
Same 8 buildings, two generators. Watch what changes at each step.
Predict first
In “one building at a time” mode, what skyline do you get if the model always takes the most likely option instead of rolling dice? Is it a good skyline?
Show the answer
A flat line of 3s (1 in 320), and the dullest. It is not even the most likely whole skyline: eight 1s or eight 5s are each about 1 in 75. Taking the top option at every step does not guarantee the most likely sequence. This is why chatbots sample instead of always taking the top word, with a temperature setting that makes the dice more or less adventurous.
Lab
Two generators, one skyline
Choose a generator and a seed, then press “Step.” Autoregressive mode places one building per step and never changes it. Diffusion mode starts from noise and nudges every building at every step. Try all three seeds in each mode: the randomness is where variety comes from.
Made-up probabilities and a hand-written denoiser, computed ahead of time. No trained model runs here.
A seed fixes the random numbers (the dice rolls or the starting noise), so you can repeat a run exactly.
The lab needs JavaScript, so here are the same runs as tables. The model’s probabilities (made up): building 1 is 3 floors 40% of the time, 2 or 4 floors 20% each, 1 or 5 floors 10% each. After that, each building is the same height as the one before 50% of the time, one floor taller 25% and one floor shorter 25%. At 1 or 5 floors it cannot go further, so “same” gets 75%. A dice roll from 0 to 99 picks the option.
| Building | Options (floors: chance) | Dice roll | Chosen | Factor |
|---|---|---|---|---|
| Building 1 | 1: 10%, 2: 20%, 3: 40%, 4: 20%, 5: 10% | 63 | 3 | 0.4 |
| Building 2 | 2: 25%, 3: 50%, 4: 25% | 12 | 2 | 0.25 |
| Building 3 | 1: 25%, 2: 50%, 3: 25% | 88 | 3 | 0.25 |
| Building 4 | 2: 25%, 3: 50%, 4: 25% | 41 | 3 | 0.5 |
| Building 5 | 2: 25%, 3: 50%, 4: 25% | 95 | 4 | 0.25 |
| Building 6 | 3: 25%, 4: 50%, 5: 25% | 30 | 4 | 0.5 |
| Building 7 | 3: 25%, 4: 50%, 5: 25% | 7 | 3 | 0.25 |
| Building 8 | 2: 25%, 3: 50%, 4: 25% | 52 | 3 | 0.5 |
Multiply the factors: 0.4 × 0.25 × 0.25 × 0.5 × 0.25 × 0.5 × 0.25 × 0.5 = 0.0001953125, or 1 in 5,120. Seed B gives 2, 2, 1, 1, 2, 1, 1, 2, seed C gives 4, 4, 3, 4, 4, 5, 5, 4, and “always the most likely” gives 3, 3, 3, 3, 3, 3, 3, 3 (1 in 320).
The denoiser (a hand-written stand-in for a trained network): for each building, average it with its neighbors and round to a whole floor (halves round up), keeping the guess between 1 and 5. Then every building moves one floor toward its guess. It “knows” only two things about skylines: neighbors are similar, and heights are whole floors from 1 to 5.
| Step | Skyline | Denoiser’s guess |
|---|---|---|
| 0 | 7, 0, 4, 9, 2, 6, 1, 8 | 4, 4, 4, 5, 5, 3, 5, 5 |
| 1 | 6, 1, 4, 8, 3, 5, 2, 7 | 4, 4, 4, 5, 5, 3, 5, 5 |
| 2 | 5, 2, 4, 7, 4, 4, 3, 6 | 4, 4, 4, 5, 5, 4, 4, 5 |
| 3 | 4, 3, 4, 6, 5, 4, 4, 5 | 4, 4, 4, 5, 5, 4, 4, 5 |
| 4 | 4, 4, 4, 5, 5, 4, 4, 5 | 4, 4, 4, 5, 5, 4, 4, 5 |
Seed B ends as 5, 5, 5, 4, 4, 4, 4, 4 after 5 steps; seed C ends as 4, 4, 5, 5, 4, 4, 3, 3 after 4 steps. Different noise, different skyline.
Autoregressive: 8 steps for 8 buildings. Diffusion: 4 or 5 steps, each touching all 8. Which is faster depends on output length and the cost of each step.
Families 3 to 5
A contest, a code and a reversible path
These three make a sample in a single pass, but each is trained very differently.
GAN (generative adversarial network). A generator turns random numbers into a sample; a discriminator tries to tell real examples from generated ones, like counterfeiters and police (the 2014 paper’s comparison). Samples come out fast and sharp, but training can swing back and forth, or the generator makes only the few outputs that fool the checker: mode collapse.
VAE (variational autoencoder). An encoder squeezes each example into a short code, much like an embedding; a decoder rebuilds the example. Training pushes the codes into a simple cloud without gaps, so any fresh point decodes to something new, and walking between two codes blends two examples. Classic VAEs are fast but often blurry on photos.
Flow model. Steps that stretch and squeeze noise into data, like clay that never tears, each undoable exactly. So a flow can turn an example back into its code and give its exact probability, but every layer must stay reversible.
| Family | How one sample is made | Speed | Quality | Variety | Control | Probability of a given example |
|---|---|---|---|---|---|---|
| Autoregressive | One piece after another, each picked from probabilities given the pieces so far | One step per piece, so long outputs are slow | Very strong for text and code | Comes from sampling; always taking the top choice gets dull | Steer with the start (a prompt); hard to change the middle | Exact (multiply) |
| Diffusion | Start from pure noise; remove a little at a time, the whole output at every step | Many steps (1,000 in the 2020 DDPM paper; later methods use far fewer) | Very high for images, audio and video | High: new noise, new result | Strong: guide with text, fill in a masked area, edit part of a picture | Estimated |
| GAN | A generator turns a random code into a sample; trained against a discriminator | Fast: one pass | Sharp, especially in narrow subjects | Can miss whole kinds of examples (mode collapse) | Move the random code or add a label | None |
| VAE | A decoder turns a short code, drawn from a simple cloud, into a sample | Fast: one pass | Often blurry for photos | Good coverage | Smooth code space: blend or slide between examples | Estimated (a lower bound) |
| Flow | Reversible steps reshape simple noise into data | One pass, but large networks | Usually behind the best diffusion models for images | Good coverage | Exactly reversible: example to code and back | Exact |
In real systems
Families are ingredients
Products rarely use one family alone.
The 2022 “latent diffusion” design mixes three families. An autoencoder shrinks pictures to a small grid of numbers, so diffusion can run in that small space with far less computing, and a transformer guides it with your prompt. The autoencoder was even trained partly with a GAN-style checker.
Other systems turn pictures into tokens and generate them one at a time. When a product “uses diffusion,” ask which part.
One image request, step by step
- A transformer turns the prompt into numbers.
- Random noise fills a small code space.
- A denoiser cleans it in many steps, guided by the prompt.
- A decoder turns the code into pixels.
Simplified from Rombach and colleagues, 2022.
4 · Common mix-ups
What people get wrong about generators
Three ideas that trip people up, and what to think instead.
Mix-up: A diffusion model stores pictures and collages pieces of them.
Better: It runs a denoising network; no picture library is searched while it generates. How often generators reproduce close copies of training examples is an active research question.
Mix-up: The newest family replaced the older ones.
Better: All five are still used, often inside the same system, as the latent diffusion example shows.
Mix-up: The most likely output is the best output.
Better: In the lab, always taking the most likely next building gives a dull flat line, and it is not even the most likely whole skyline. Sampling trades a little probability for variety, and a likely output can still be false.
5 · Check yourself
Three quick questions
Answer on paper before you open each one.
Using the lab’s autoregressive probabilities, what is the chance that a skyline starts 2, 1, 1? Show the multiplication.
Check your answer
Building 1 is 2 floors with chance 0.2. From 2, one floor down has chance 0.25. From 1, staying at 1 has chance 0.75. So 0.2 × 0.25 × 0.75 = 0.0375, about 1 in 27.
One team needs thousands of images per second on a small server and can accept less variety. Another must know exactly how likely each new measurement is, to flag unusual ones. Which families would each look at first?
Check your answer
The first team: a one-pass generator such as a GAN (or a VAE), then a careful check for mode collapse. The second team: a flow or an autoregressive model, because both give an exact probability for any example. Both teams should test on their own data.
Why does a diffusion model start from random noise, and what happens if you reuse the same noise and settings?
Check your answer
The noise is the source of variety: different noise leads the denoiser to a different result, as seeds A, B and C showed. With the same noise, the same random numbers at every step and the same settings, you usually get the same result. That is why some image tools let you reuse a seed.
In the real world
An application, a limit and a link
Application
Text, pictures, sound
Chatbots write with autoregressive transformers; many image, audio and video tools use diffusion. See Creative AI.
Limitation
Likely is not true
Every family learns what looks typical, not what is true, so generators can make convincing fake photos and voices (deepfakes) and confident wrong sentences. See security and provenance.
For younger learners
A grades 6–8 version
Both main ideas work with a die, graph paper and no family names.
20 minutes · pairs
Dice skyline vs noise cleanup
- Roll a die for building 1: 1–2 means 2 floors, 3–4 means 3, 5–6 means 4. Color it on graph paper.
- For each next building, roll again: 1–2 one floor shorter, 3–4 the same, 5–6 one floor taller (never below 1 or above 5).
- Now draw 8 random heights from 0 to 9. In each round, move every building one floor toward the average of itself and its neighbors. Stop when nothing changes.
- Compare: which method could change a building after drawing it?
Exit question
Say it in one sentence
“How can a computer make a new picture?” A good answer: “It learns what pictures usually look like, then builds one bit by bit or cleans up random static until it looks right.” Younger classes can skip GANs, VAEs and flows.
Honest limits
What this page leaves out
What this page leaves out
The training math (loss functions, the VAE’s evidence lower bound, noise schedules), how diffusion is steered by text (guidance), newer families such as flow matching and consistency models, energy-based models, how generated samples are scored, and questions of copyright, consent and labeling, which belong on Responsible AI.
Where to go next
Follow the thread
Applications
Creative AI
What image, video, music and voice generators are used for, and the questions they raise.
Autoregressive up close
What is a transformer?
Tokens, embeddings, attention and next-word prediction, in five steps.
Next in the advanced track
Can AI know why?
Knowledge graphs, Bayesian reasoning and the difference between seeing and doing.
Sources
Where this comes from
- Ian Goodfellow and colleagues, “Generative Adversarial Nets,” 2014: the generator, the discriminator and the counterfeiter analogy.
- Diederik Kingma and Max Welling, “Auto-Encoding Variational Bayes,” 2013: the variational autoencoder.
- Jonathan Ho, Ajay Jain and Pieter Abbeel, “Denoising Diffusion Probabilistic Models,” 2020: adding and removing noise over 1,000 steps.
- Laurent Dinh, Jascha Sohl-Dickstein and Samy Bengio, “Density estimation using Real NVP,” 2016: a flow model with exact probabilities and reversible steps.
- Robin Rombach and colleagues, “High-Resolution Image Synthesis with Latent Diffusion Models,” 2022: diffusion in a compressed code space, guided by a text encoder.
- Ian Goodfellow, Yoshua Bengio and Aaron Courville, Deep Learning (MIT Press, 2016), free online: chapter 20, “Deep Generative Models,” an overview of the families.
AI words on this page
Generative AI, Autoregressive model, Next-word prediction, Diffusion model, GAN, VAE, Flow model, Probability distribution, Embedding, Temperature, Image generator, Deepfake
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.