Go deeper · Advanced track

How do different generators make something new?

They learn the patterns in many examples, then produce a new example that fits them. They differ in how: one piece at a time, by cleaning up noise, through a contest between two networks, by decoding a short code, or with reversible steps. Each trades speed, quality, variety and control differently, and real systems often combine them.

Grades 9+25 minutesMultiply probabilitiesYounger version for grades 6–8

This is an optional advanced page. You do not need it for the main foundations route.

What you will be able to do

Describe how five generator families make a sample, multiply step probabilities to score a short sequence, and pick a family for a job that needs speed, quality, variety or control.

Before you start

Read first: Why are there different kinds of neural networks?, How sure should an AI be? and How can meaning become a place on a map? Helpful: next-word prediction and creative AI tools.

1 · You have already seen this

Text and pictures nobody made by hand

Many people have asked a chatbot to write something, or typed a sentence into an app and got back a picture.

Ask a chatbot for a short poem and the words appear a few at a time. Type “a city skyline at night” into an image tool and a picture comes back that nobody drew. Each one learned from a huge pile of examples, and each made something new that fits the patterns in that pile.

Programs like these are generative models. They learn which results are likely and then draw a new one, and there are five main ways to do the drawing, from building one piece at a time to cleaning up random static. This page follows them through a game studio’s skylines. It shows how they work, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From many examples to a new one

Making something new in four steps. The next section goes deeper into steps 1 and 2, and the two after it show two different ways to do steps 3 and 4. After the lab, a section covers three more ways.

How a generator works

Four steps

  1. 1 · Collect many examplesThe model needs a large pile of the kind of thing it should make, such as a few thousand skylines drawn by artists.
  2. 2 · Learn what is likelyTraining teaches the model which results are typical of the pile and which are not. Nobody has to label the examples.
  3. 3 · Start from chanceMaking something new begins with random numbers, such as a dice roll for the first piece or a screen of pure static.
  4. 4 · Build the resultSome models add one piece at a time and never go back. Others clean up the whole thing a little at every step, and some make it in a single pass.

Going deeper · steps 1 and 2

One goal, five routes

Before any family names: what does it mean for a computer to make something new?

A game studio has a few thousand skylines drawn by artists. It wants endless new ones that look as if they belong in the same pile, not copies.

A generative model learns which skylines are likely and which are not: in effect, a probability distribution. Making a new one is sampling: drawing one example from that distribution, like drawing a card from a deck. Randomness is the point: without it, every skyline would be the same.

The five families below do this differently. A family name says how sampling works, not how the layers are wired: most use the network families you already know inside.

Family 1 · going deeper into steps 3 and 4

One piece at a time

An autoregressive model predicts the next piece from the pieces so far, adds it, and repeats.

This is how a chatbot writes: it scores every possible next token, picks one, adds it and predicts again (see next-word prediction). It works for pixels, notes or buildings too.

So a whole result’s probability is the step probabilities multiplied together, which the model can compute exactly. And it needs one step per piece and never goes back: 1,000 tokens take 1,000 steps, and an early mistake stays.

Worked example

Multiplying along a 4-building skyline

  1. The modelMade-up probabilities: building 1 is 3 floors with chance 0.4. After that, the same height has chance 0.5, one floor up 0.25, one floor down 0.25.
  2. Skyline 3, 4, 4, 50.4 × 0.25 × 0.5 × 0.25 = 0.0125, or 1 in 80.
  3. Skyline 3, 3, 3, 30.4 × 0.5 × 0.5 × 0.5 = 0.05, or 1 in 20. No 4-building skyline is more likely.
  4. Check the totalOnly 95 of the 5 × 5 × 5 × 5 = 625 height lists are allowed, and their probabilities add up to exactly 1.
  5. Go to 8 buildingsAlways taking the most likely next building gives eight 3s: 0.4 × 0.57 = 0.003125, or 1 in 320. Yet that is not the most likely whole skyline: eight 1s or eight 5s each have about 1 in 75, because an edge height repeats with chance 0.75. Locally best choices need not make the best whole sequence. With 5,275 skylines allowed, a sampler almost never produces the greedy “best” one.

Made-up numbers, checked in Python. Real models multiply the same way.

Family 2 · going deeper into steps 3 and 4

Start from noise, clean it step by step

A diffusion model learns to remove noise, then makes something new by removing noise from pure static.

Training: add a known amount of random noise to a real example and teach a network, the denoiser, to guess the clean version (or the noise). Repeat at every noise level, up to pure static.

Generating runs backward: start from pure noise, ask the denoiser for its guess, move a little toward it, and repeat with less noise. Every step can change any part of the output. The 2020 DDPM paper used 1,000 steps; later methods use far fewer, but each step is a pass through a big network.

In the toy below, the noise 7, 0, 4 around building 2 averages 11 ÷ 3 ≈ 3.67, so the guess is 4 and the building rises from 0 to 1.

3 · Try it

Build a skyline two ways

Same 8 buildings, two generators. Watch what changes at each step.

Predict first

In “one building at a time” mode, what skyline do you get if the model always takes the most likely option instead of rolling dice? Is it a good skyline?

Show the answer

A flat line of 3s (1 in 320), and the dullest. It is not even the most likely whole skyline: eight 1s or eight 5s are each about 1 in 75. Taking the top option at every step does not guarantee the most likely sequence. This is why chatbots sample instead of always taking the top word, with a temperature setting that makes the dice more or less adventurous.

Lab

Two generators, one skyline

Choose a generator and a seed, then press “Step.” Autoregressive mode places one building per step and never changes it. Diffusion mode starts from noise and nudges every building at every step. Try all three seeds in each mode: the randomness is where variety comes from.

Made-up probabilities and a hand-written denoiser, computed ahead of time. No trained model runs here.

1 · Pick a way to generate
2 · Pick the random numbers

A seed fixes the random numbers (the dice rolls or the starting noise), so you can repeat a run exactly.

3 · Run it

The lab needs JavaScript, so here are the same runs as tables. The model’s probabilities (made up): building 1 is 3 floors 40% of the time, 2 or 4 floors 20% each, 1 or 5 floors 10% each. After that, each building is the same height as the one before 50% of the time, one floor taller 25% and one floor shorter 25%. At 1 or 5 floors it cannot go further, so “same” gets 75%. A dice roll from 0 to 99 picks the option.

One building at a time, seed A (made-up probabilities)
BuildingOptions (floors: chance)Dice rollChosenFactor
Building 11: 10%, 2: 20%, 3: 40%, 4: 20%, 5: 10%6330.4
Building 22: 25%, 3: 50%, 4: 25%1220.25
Building 31: 25%, 2: 50%, 3: 25%8830.25
Building 42: 25%, 3: 50%, 4: 25%4130.5
Building 52: 25%, 3: 50%, 4: 25%9540.25
Building 63: 25%, 4: 50%, 5: 25%3040.5
Building 73: 25%, 4: 50%, 5: 25%730.25
Building 82: 25%, 3: 50%, 4: 25%5230.5

Multiply the factors: 0.4 × 0.25 × 0.25 × 0.5 × 0.25 × 0.5 × 0.25 × 0.5 = 0.0001953125, or 1 in 5,120. Seed B gives 2, 2, 1, 1, 2, 1, 1, 2, seed C gives 4, 4, 3, 4, 4, 5, 5, 4, and “always the most likely” gives 3, 3, 3, 3, 3, 3, 3, 3 (1 in 320).

The denoiser (a hand-written stand-in for a trained network): for each building, average it with its neighbors and round to a whole floor (halves round up), keeping the guess between 1 and 5. Then every building moves one floor toward its guess. It “knows” only two things about skylines: neighbors are similar, and heights are whole floors from 1 to 5.

Clean up noise, seed A (step 0 is pure noise)
StepSkylineDenoiser’s guess
07, 0, 4, 9, 2, 6, 1, 84, 4, 4, 5, 5, 3, 5, 5
16, 1, 4, 8, 3, 5, 2, 74, 4, 4, 5, 5, 3, 5, 5
25, 2, 4, 7, 4, 4, 3, 64, 4, 4, 5, 5, 4, 4, 5
34, 3, 4, 6, 5, 4, 4, 54, 4, 4, 5, 5, 4, 4, 5
44, 4, 4, 5, 5, 4, 4, 54, 4, 4, 5, 5, 4, 4, 5

Seed B ends as 5, 5, 5, 4, 4, 4, 4, 4 after 5 steps; seed C ends as 4, 4, 5, 5, 4, 4, 3, 3 after 4 steps. Different noise, different skyline.

Autoregressive: 8 steps for 8 buildings. Diffusion: 4 or 5 steps, each touching all 8. Which is faster depends on output length and the cost of each step.

Families 3 to 5

A contest, a code and a reversible path

These three make a sample in a single pass, but each is trained very differently.

GAN (generative adversarial network). A generator turns random numbers into a sample; a discriminator tries to tell real examples from generated ones, like counterfeiters and police (the 2014 paper’s comparison). Samples come out fast and sharp, but training can swing back and forth, or the generator makes only the few outputs that fool the checker: mode collapse.

VAE (variational autoencoder). An encoder squeezes each example into a short code, much like an embedding; a decoder rebuilds the example. Training pushes the codes into a simple cloud without gaps, so any fresh point decodes to something new, and walking between two codes blends two examples. Classic VAEs are fast but often blurry on photos.

Flow model. Steps that stretch and squeeze noise into data, like clay that never tears, each undoable exactly. So a flow can turn an example back into its code and give its exact probability, but every layer must stay reversible.

Typical trade-offs, simplified. Research keeps moving these; read them as tendencies, not rules.
FamilyHow one sample is madeSpeedQualityVarietyControlProbability of a given example
AutoregressiveOne piece after another, each picked from probabilities given the pieces so farOne step per piece, so long outputs are slowVery strong for text and codeComes from sampling; always taking the top choice gets dullSteer with the start (a prompt); hard to change the middleExact (multiply)
DiffusionStart from pure noise; remove a little at a time, the whole output at every stepMany steps (1,000 in the 2020 DDPM paper; later methods use far fewer)Very high for images, audio and videoHigh: new noise, new resultStrong: guide with text, fill in a masked area, edit part of a pictureEstimated
GANA generator turns a random code into a sample; trained against a discriminatorFast: one passSharp, especially in narrow subjectsCan miss whole kinds of examples (mode collapse)Move the random code or add a labelNone
VAEA decoder turns a short code, drawn from a simple cloud, into a sampleFast: one passOften blurry for photosGood coverageSmooth code space: blend or slide between examplesEstimated (a lower bound)
FlowReversible steps reshape simple noise into dataOne pass, but large networksUsually behind the best diffusion models for imagesGood coverageExactly reversible: example to code and backExact

In real systems

Families are ingredients

Products rarely use one family alone.

The 2022 “latent diffusion” design mixes three families. An autoencoder shrinks pictures to a small grid of numbers, so diffusion can run in that small space with far less computing, and a transformer guides it with your prompt. The autoencoder was even trained partly with a GAN-style checker.

Other systems turn pictures into tokens and generate them one at a time. When a product “uses diffusion,” ask which part.

One image request, step by step

  1. A transformer turns the prompt into numbers.
  2. Random noise fills a small code space.
  3. A denoiser cleans it in many steps, guided by the prompt.
  4. A decoder turns the code into pixels.

Simplified from Rombach and colleagues, 2022.

4 · Common mix-ups

What people get wrong about generators

Three ideas that trip people up, and what to think instead.

  • Mix-up: A diffusion model stores pictures and collages pieces of them.

    Better: It runs a denoising network; no picture library is searched while it generates. How often generators reproduce close copies of training examples is an active research question.

  • Mix-up: The newest family replaced the older ones.

    Better: All five are still used, often inside the same system, as the latent diffusion example shows.

  • Mix-up: The most likely output is the best output.

    Better: In the lab, always taking the most likely next building gives a dull flat line, and it is not even the most likely whole skyline. Sampling trades a little probability for variety, and a likely output can still be false.

5 · Check yourself

Three quick questions

Answer on paper before you open each one.

  1. Using the lab’s autoregressive probabilities, what is the chance that a skyline starts 2, 1, 1? Show the multiplication.

    Check your answer

    Building 1 is 2 floors with chance 0.2. From 2, one floor down has chance 0.25. From 1, staying at 1 has chance 0.75. So 0.2 × 0.25 × 0.75 = 0.0375, about 1 in 27.

  2. One team needs thousands of images per second on a small server and can accept less variety. Another must know exactly how likely each new measurement is, to flag unusual ones. Which families would each look at first?

    Check your answer

    The first team: a one-pass generator such as a GAN (or a VAE), then a careful check for mode collapse. The second team: a flow or an autoregressive model, because both give an exact probability for any example. Both teams should test on their own data.

  3. Why does a diffusion model start from random noise, and what happens if you reuse the same noise and settings?

    Check your answer

    The noise is the source of variety: different noise leads the denoiser to a different result, as seeds A, B and C showed. With the same noise, the same random numbers at every step and the same settings, you usually get the same result. That is why some image tools let you reuse a seed.

In the real world

An application, a limit and a link

Application

Text, pictures, sound

Chatbots write with autoregressive transformers; many image, audio and video tools use diffusion. See Creative AI.

Limitation

Likely is not true

Every family learns what looks typical, not what is true, so generators can make convincing fake photos and voices (deepfakes) and confident wrong sentences. See security and provenance.

Connection

Learning without labels

Generators learn from unlabeled examples, like finding groups.

For younger learners

A grades 6–8 version

Both main ideas work with a die, graph paper and no family names.

20 minutes · pairs

Dice skyline vs noise cleanup

  1. Roll a die for building 1: 1–2 means 2 floors, 3–4 means 3, 5–6 means 4. Color it on graph paper.
  2. For each next building, roll again: 1–2 one floor shorter, 3–4 the same, 5–6 one floor taller (never below 1 or above 5).
  3. Now draw 8 random heights from 0 to 9. In each round, move every building one floor toward the average of itself and its neighbors. Stop when nothing changes.
  4. Compare: which method could change a building after drawing it?

Exit question

Say it in one sentence

“How can a computer make a new picture?” A good answer: “It learns what pictures usually look like, then builds one bit by bit or cleans up random static until it looks right.” Younger classes can skip GANs, VAEs and flows.

Honest limits

What this page leaves out

What this page leaves out

The training math (loss functions, the VAE’s evidence lower bound, noise schedules), how diffusion is steered by text (guidance), newer families such as flow matching and consistency models, energy-based models, how generated samples are scored, and questions of copyright, consent and labeling, which belong on Responsible AI.