Go deeper · How machines learn

How can meaning become a place on a map?

A model can turn a word, a sentence or a picture into a list of numbers called an embedding. Training arranges these lists so that things used in similar ways get similar numbers, like nearby places on a map. Then “find related things” becomes “find the nearest points,” which is how semantic search works. The map is useful but imperfect: nearby does not mean the same meaning.

Grades 9+20 minutesMultiply, add, square rootsYounger version for grades 6–8

What you will be able to do

Compute the cosine similarity of two small vectors, find a query’s nearest neighbors, and explain why a flat 2-D picture of embeddings distorts distances and why a high similarity score does not guarantee the same meaning.

Before you start

Before this: Turn words into places on a map (step 2 of the transformer page) and How does a picture become numbers? (vectors and the dot product). Helpful: How can an AI answer from documents it was given?

1 · You have already seen this

Search that finds what you meant

You type one thing and get back a page that says it in different words.

Search a library website for “late fees” and a page about “overdue fines” can come back near the top, even though the two phrases share no words. Apps that suggest “more like this” work the same way: they look for items that are close in meaning to the one you picked.

Both rely on embeddings. A model turns every phrase into a list of numbers, and phrases that are used in similar ways get similar numbers. Finding related things then becomes a matter of finding the nearest numbers. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.

2 · What happens, step by step

From a question to the nearest answer

Semantic search in four steps. The two sections after this one go deeper into steps 1 and 3.

How semantic search works

Four steps

  1. 1 · Embed the libraryAhead of time, an embedding model turns every passage into one list of numbers (many popular models use 768) and stores the lists.
  2. 2 · Embed the questionWhen you search, your question goes through the same model and gets a list of its own.
  3. 3 · Compare directionsEach list is treated as an arrow. The system scores how closely your question’s arrow points the same way as each stored arrow. That score is called cosine similarity, and 1 means the same direction.
  4. 4 · Return the nearestThe passages with the highest scores come back first. When a chatbot answers from documents, these are the passages it reads before it writes its answer.

Going deeper · step 1

A vector for every meaning

The transformer page drew words on a flat map. Real embeddings are the same idea with far more directions.

On the transformer page, “cat,” “dog” and “kitten” sit close together on a two-dimensional map, and food words sit somewhere else. Each word’s position is a vector: a list of numbers. Real models use hundreds or thousands of numbers per item. Many popular sentence-embedding models, for example, give every sentence a list of 768 numbers.

Two kinds are worth telling apart. Inside a chatbot, every token starts as an embedding that the layers then refine. Separately, embedding models turn a whole sentence or passage into one vector, built for comparing texts. Search systems use the second kind.

Nobody writes these numbers by hand. Training nudges them so that items appearing in similar contexts end up with similar vectors. As a result, no single number means “animal” or “food.” Meaning is spread across many numbers at once. The lab below labels its four numbers only so you can see what is going on.

Going deeper · step 3

Cosine similarity: same direction?

Picture each vector as an arrow from the origin. Cosine similarity asks how closely two arrows point the same way, ignoring how long they are.

Worked example

Three made-up 3-D vectors

  1. VectorsIllustrative only: cat = [2, 1, 0], kitten = [3, 1, 1], car = [0, 1, 3].
  2. Dot productMultiply matching numbers and add: cat · kitten = 2×3 + 1×1 + 0×1 = 7.
  3. Lengths|cat| = √(2² + 1² + 0²) = √5 ≈ 2.236. |kitten| = √(9 + 1 + 1) = √11 ≈ 3.317.
  4. Dividecosine = 7 ÷ (2.236 × 3.317) = 7 ÷ 7.416 ≈ 0.94. The arrows point almost the same way.
  5. Comparecat · car = 0 + 1 + 0 = 1, and |car| = √10. cosine = 1 ÷ (√5 × √10) = 1 ÷ √50 ≈ 1 ÷ 7.071 ≈ 0.14. Far less similar.
  6. Length ignoredA longer arrow in the same direction, [4, 2, 0], has cosine 1.00 with cat: twice as long, same direction.

Cosine similarity runs from −1 (opposite directions) through 0 (unrelated, at right angles) to 1 (same direction). Scores from different embedding models are not on the same scale, so compare scores only within one model.

Predict first

Before you open the lab: a thunderstorm and a sunny afternoon describe opposite weather. Will their embeddings be far apart?

Show the answer

Probably not. Both are about weather, appear in similar sentences (“the forecast says…”), and so tend to get similar vectors. In the lab’s made-up numbers their cosine is 0.95, the closest pair for “a thunderstorm.” Embeddings capture how words are used, which is not the same as what they mean. Opposites like “hot” and “cold” often sit close together for the same reason.

3 · Try it

Nearest neighbors, then flatten

Pick a phrase as your search query and see which phrases are closest. Then switch to the flat map and watch the neighbors change.

Lab

Semantic search on eleven phrases

Choose a query. The list ranks every other phrase by cosine similarity using all four numbers. Then switch to the flat 2-D map, which keeps only the first two numbers, the way a chart on paper has only two directions. Which neighbors survive?

Hand-made, illustrative 4-number vectors. Real embeddings are learned and much longer. Computed in your browser.

1 · Pick a query
2 · How should “close” be measured?

Nearest neighbors

    The flat map: animals across, food up

    Scatter plot of the eleven phrases using only their first two numbers

    The lab needs JavaScript, so here are its numbers. Each phrase has a made-up vector of four numbers, labeled (animals, food, sports, weather) only for teaching. The third column lists the three nearest phrases by cosine similarity using all four numbers (higher is closer). The last column lists the three nearest dots on a flat map that keeps only the first two numbers (distance on the map, lower is closer).

    Illustrative vectors and nearest neighbors
    PhraseVectorNearest with all 4 numbers (cosine)Nearest on the flat 2-D map (distance)
    a puppy[0.9, 0.1, 0.1, 0.0]a sleepy kitten (0.92), a bag of dog food (0.73), a hot dog with mustard (0.33)a sleepy kitten (0.14), a bag of dog food (0.67), a thunderstorm (0.71)
    a sleepy kitten[0.8, 0.0, 0.0, 0.3]a puppy (0.92), a bag of dog food (0.61), a thunderstorm (0.55)a puppy (0.14), a thunderstorm (0.60), a match stopped by rain (0.70)
    a bag of dog food[0.6, 0.7, 0.0, 0.0]a hot dog with mustard (0.88), a slice of pizza (0.74), a puppy (0.73)a hot dog with mustard (0.45), a picnic lunch in the park (0.51), a slice of pizza (0.63)
    a hot dog with mustard[0.2, 0.9, 0.1, 0.0]a slice of pizza (0.97), a bag of dog food (0.88), a picnic lunch in the park (0.71)a slice of pizza (0.20), a picnic lunch in the park (0.32), a bag of dog food (0.45)
    a slice of pizza[0.0, 0.9, 0.2, 0.0]a hot dog with mustard (0.97), a bag of dog food (0.74), a picnic lunch in the park (0.71)a hot dog with mustard (0.20), a picnic lunch in the park (0.32), a bag of dog food (0.63)
    a picnic lunch in the park[0.1, 0.6, 0.1, 0.6]a sunny afternoon (0.86), a hot dog with mustard (0.71), a slice of pizza (0.71)a hot dog with mustard (0.32), a slice of pizza (0.32), a sunny afternoon (0.40)
    a football match[0.0, 0.1, 0.9, 0.2]a tennis racket (0.97), a match stopped by rain (0.80), a sunny afternoon (0.35)a tennis racket (0.11), a match stopped by rain (0.11), a sunny afternoon (0.11)
    a tennis racket[0.1, 0.15, 0.9, 0.0]a football match (0.97), a match stopped by rain (0.65), a slice of pizza (0.37)a sunny afternoon (0.07), a match stopped by rain (0.10), a football match (0.11)
    a match stopped by rain[0.1, 0.05, 0.6, 0.7]a sunny afternoon (0.82), a football match (0.80), a thunderstorm (0.76)a tennis racket (0.10), a football match (0.11), a thunderstorm (0.11)
    a thunderstorm[0.2, 0.0, 0.0, 0.9]a sunny afternoon (0.95), a match stopped by rain (0.76), a picnic lunch in the park (0.71)a match stopped by rain (0.11), a tennis racket (0.18), a football match (0.22)
    a sunny afternoon[0.05, 0.2, 0.1, 0.8]a thunderstorm (0.95), a picnic lunch in the park (0.86), a match stopped by rain (0.82)a tennis racket (0.07), a football match (0.11), a match stopped by rain (0.16)

    Compare the two right-hand columns for “a thunderstorm”: with all four numbers its neighbors are about weather; on the flat map, sports items crowd next to it, because the map threw away the sports and weather numbers.

    With all four numbers, “a thunderstorm” is nearest to “a sunny afternoon” (0.95), “a match stopped by rain” and “a picnic lunch in the park”: all have weather in them. On the flat map it sits next to “a tennis racket” and “a football match,” whose cosine with it in the full space is about 0.02 and 0.21. Those items are not similar. The map simply dropped the two numbers that tell sports and weather apart, so they all piled up in one corner.

    4 · Common mix-ups

    Similar is not the same

    An embedding measures how things are used together in the training data. That is powerful, and it has blind spots.

    Opposites and negation. “Cheap tickets” and “expensive tickets,” or “the bus was late” and “the bus was not late,” share almost every word and context. Embedding models can score such pairs as very similar, even though for a reader they mean opposite things.

    Surface overlap. In the lab, “a hot dog with mustard” has a little “animals” in its made-up vector because of the word “dog,” so its cosine with “a bag of dog food” is 0.88. Real models can make the same kind of slip.

    Bias. Embeddings learn from human writing, including its stereotypes. In 2016, Tolga Bolukbasi and colleagues showed that word embeddings trained on news articles completed the analogy “man is to computer programmer as woman is to …” with “homemaker.” A 2017 study by Aylin Caliskan and colleagues found that word embeddings reproduced many biases that psychologists had measured in people. If a search or a hiring tool ranks by similarity, those patterns can quietly shape what people are shown.

    Four more to watch for

    • Mix-up: Each number in an embedding stands for one human idea, like “animal.”

      Better: The lab labels its numbers for teaching. In a real embedding, meaning is spread across many numbers and no single number has a clean label.

    • Mix-up: If two items look close on a 2-D chart, they are similar.

      Better: The chart squashed hundreds of directions into two. Check the similarity computed with all the numbers.

    • Mix-up: A similarity of 0.95 means the two texts say the same thing.

      Better: It means they are used in similar contexts. Opposites, negations and word overlaps can score high.

    • Mix-up: An embedding is a definition the model looked up.

      Better: It is a learned list of numbers. It records patterns of use in the training data, including its biases.

    Mix-up · trusting the picture

    Why a flat map lies a little

    You cannot draw 768 directions on a page. Every 2-D picture of an embedding space has to throw information away.

    The lab’s flat map just dropped two numbers. Real charts are made with smarter dimensionality reduction methods, such as PCA, t-SNE or UMAP, which try to keep as much structure as possible (see Can a computer find groups nobody labeled?). They still cannot keep everything. Squashing 768 numbers into 2 is like photographing a city from one angle: some buildings that are far apart line up and look close.

    Methods like t-SNE mainly try to keep each point’s nearest neighbors nearby. The sizes of clusters and the gaps between them on the picture can mean very little, and different settings can produce quite different pictures of the same data. So a chart of embeddings is a sketch for asking questions, not a measurement. To decide whether two items are similar, compute their similarity in the full space.

    Rule of thumb

    Trust the numbers, question the picture

    Close on a 2-D chart: maybe similar. Far on a 2-D chart: maybe not different. A similarity score computed with all the numbers is the better evidence, and even that is only as good as the model that made the vectors.

    5 · Check yourself

    Three quick questions

    Work them out on paper before you open each one.

    1. What is the cosine similarity of [1, 2, 2] and [2, 4, 4]? And of [1, 2, 2] and [2, −1, 0]?

      Check your answer

      The first pair points the same way (the second is exactly twice the first), so cosine = 1. Check: dot product 2 + 8 + 8 = 18, lengths 3 and 6, and 18 ÷ (3 × 6) = 1. For the second pair the dot product is 2 − 2 + 0 = 0, so cosine = 0: the arrows are at right angles.

    2. A chart in an article shows “river” and “bank” far apart on a 2-D map of word embeddings. Can you conclude the model sees them as unrelated?

      Check your answer

      No. The chart is a 2-D picture of a space with hundreds of directions, so distances on it are distorted. Compute the similarity with all the numbers before concluding anything. (“Bank” also has more than one meaning, which a single word vector blends together.)

    3. A semantic search for “cheap train tickets” returns a page about “expensive train tickets” near the top. Why, and what could the search system add?

      Check your answer

      The two phrases share words and contexts, so their embeddings are close even though the meaning is opposite. The system could combine embeddings with keyword or price filters, rerank the top results with a more careful model, and show the source so a person can check.

    In the real world

    An application, a limit and a link

    Application

    Semantic search

    Embed every passage once and store the vectors in a vector database. When a question arrives, embed it and return the passages with the highest similarity. That is the retrieval step in retrieval-augmented generation (RAG), and it finds “overdue fines” when you asked about “late fees.”

    Limitation

    Related is not relevant

    The nearest passage is the most similar one, not necessarily the correct or most useful one. Many systems mix in keyword search, rerank results, and show sources so people can check the passage that was used.

    Connection

    Nearest neighbors, again

    Finding the closest vectors is the same idea as the k-nearest-neighbors model on Classical models. Apps that suggest “more like this” use it too: see Why does the app think I want that?

    For younger learners

    A grades 6–8 version

    No square roots: a word map on the classroom floor.

    20 minutes · whole class

    Human word map

    1. Tape two lines on the floor: “how much is it about animals?” and “how much is it about food?”
    2. Give students cards: puppy, pizza, hot dog, football, thunderstorm, tennis. Each stands where their card belongs.
    3. Ask: who is standing close to whom? Do football and thunderstorm belong together?
    4. Add a third question, “how much is it about weather?”, by raising hands. Now who is really close?

    Exit question

    Say it in one sentence

    “Why can a map of meaning fool you?” A good answer: “It only shows a few directions, so things that differ in a hidden direction can look close.”

    What we skipped

    What this page leaves out

    What this page leaves out

    How embedding models are trained (for example by pulling matching pairs of sentences together and pushing others apart), other distance measures such as Euclidean distance, how vector databases find approximate nearest neighbors quickly among millions of vectors, embeddings for pictures, sound and products, and methods that try to measure or reduce bias in embeddings.