Go deeper · How machines learn
How can meaning become a place on a map?
A model can turn a word, a sentence or a picture into a list of numbers called an embedding. Training arranges these lists so that things used in similar ways get similar numbers, like nearby places on a map. Then “find related things” becomes “find the nearest points,” which is how semantic search works. The map is useful but imperfect: nearby does not mean the same meaning.
What you will be able to do
Compute the cosine similarity of two small vectors, find a query’s nearest neighbors, and explain why a flat 2-D picture of embeddings distorts distances and why a high similarity score does not guarantee the same meaning.
Before you start
Before this: Turn words into places on a map (step 2 of the transformer page) and How does a picture become numbers? (vectors and the dot product). Helpful: How can an AI answer from documents it was given?
1 · You have already seen this
Search that finds what you meant
You type one thing and get back a page that says it in different words.
Search a library website for “late fees” and a page about “overdue fines” can come back near the top, even though the two phrases share no words. Apps that suggest “more like this” work the same way: they look for items that are close in meaning to the one you picked.
Both rely on embeddings. A model turns every phrase into a list of numbers, and phrases that are used in similar ways get similar numbers. Finding related things then becomes a matter of finding the nearest numbers. The rest of this page shows how that works, lets you try it, and shows where it goes wrong.
2 · What happens, step by step
From a question to the nearest answer
Semantic search in four steps. The two sections after this one go deeper into steps 1 and 3.
How semantic search works
Four steps
- 1 · Embed the libraryAhead of time, an embedding model turns every passage into one list of numbers (many popular models use 768) and stores the lists.
- 2 · Embed the questionWhen you search, your question goes through the same model and gets a list of its own.
- 3 · Compare directionsEach list is treated as an arrow. The system scores how closely your question’s arrow points the same way as each stored arrow. That score is called cosine similarity, and 1 means the same direction.
- 4 · Return the nearestThe passages with the highest scores come back first. When a chatbot answers from documents, these are the passages it reads before it writes its answer.
Going deeper · step 1
A vector for every meaning
The transformer page drew words on a flat map. Real embeddings are the same idea with far more directions.
On the transformer page, “cat,” “dog” and “kitten” sit close together on a two-dimensional map, and food words sit somewhere else. Each word’s position is a vector: a list of numbers. Real models use hundreds or thousands of numbers per item. Many popular sentence-embedding models, for example, give every sentence a list of 768 numbers.
Two kinds are worth telling apart. Inside a chatbot, every token starts as an embedding that the layers then refine. Separately, embedding models turn a whole sentence or passage into one vector, built for comparing texts. Search systems use the second kind.
Nobody writes these numbers by hand. Training nudges them so that items appearing in similar contexts end up with similar vectors. As a result, no single number means “animal” or “food.” Meaning is spread across many numbers at once. The lab below labels its four numbers only so you can see what is going on.
Going deeper · step 3
Cosine similarity: same direction?
Picture each vector as an arrow from the origin. Cosine similarity asks how closely two arrows point the same way, ignoring how long they are.
Worked example
Three made-up 3-D vectors
- VectorsIllustrative only: cat = [2, 1, 0], kitten = [3, 1, 1], car = [0, 1, 3].
- Dot productMultiply matching numbers and add: cat · kitten = 2×3 + 1×1 + 0×1 = 7.
- Lengths|cat| = √(2² + 1² + 0²) = √5 ≈ 2.236. |kitten| = √(9 + 1 + 1) = √11 ≈ 3.317.
- Dividecosine = 7 ÷ (2.236 × 3.317) = 7 ÷ 7.416 ≈ 0.94. The arrows point almost the same way.
- Comparecat · car = 0 + 1 + 0 = 1, and |car| = √10. cosine = 1 ÷ (√5 × √10) = 1 ÷ √50 ≈ 1 ÷ 7.071 ≈ 0.14. Far less similar.
- Length ignoredA longer arrow in the same direction, [4, 2, 0], has cosine 1.00 with cat: twice as long, same direction.
Cosine similarity runs from −1 (opposite directions) through 0 (unrelated, at right angles) to 1 (same direction). Scores from different embedding models are not on the same scale, so compare scores only within one model.
Predict first
Before you open the lab: a thunderstorm and a sunny afternoon describe opposite weather. Will their embeddings be far apart?
Show the answer
Probably not. Both are about weather, appear in similar sentences (“the forecast says…”), and so tend to get similar vectors. In the lab’s made-up numbers their cosine is 0.95, the closest pair for “a thunderstorm.” Embeddings capture how words are used, which is not the same as what they mean. Opposites like “hot” and “cold” often sit close together for the same reason.
3 · Try it
Nearest neighbors, then flatten
Pick a phrase as your search query and see which phrases are closest. Then switch to the flat map and watch the neighbors change.
Lab
Semantic search on eleven phrases
Choose a query. The list ranks every other phrase by cosine similarity using all four numbers. Then switch to the flat 2-D map, which keeps only the first two numbers, the way a chart on paper has only two directions. Which neighbors survive?
Hand-made, illustrative 4-number vectors. Real embeddings are learned and much longer. Computed in your browser.
Nearest neighbors
The flat map: animals across, food up
The lab needs JavaScript, so here are its numbers. Each phrase has a made-up vector of four numbers, labeled (animals, food, sports, weather) only for teaching. The third column lists the three nearest phrases by cosine similarity using all four numbers (higher is closer). The last column lists the three nearest dots on a flat map that keeps only the first two numbers (distance on the map, lower is closer).
| Phrase | Vector | Nearest with all 4 numbers (cosine) | Nearest on the flat 2-D map (distance) |
|---|---|---|---|
| a puppy | [0.9, 0.1, 0.1, 0.0] | a sleepy kitten (0.92), a bag of dog food (0.73), a hot dog with mustard (0.33) | a sleepy kitten (0.14), a bag of dog food (0.67), a thunderstorm (0.71) |
| a sleepy kitten | [0.8, 0.0, 0.0, 0.3] | a puppy (0.92), a bag of dog food (0.61), a thunderstorm (0.55) | a puppy (0.14), a thunderstorm (0.60), a match stopped by rain (0.70) |
| a bag of dog food | [0.6, 0.7, 0.0, 0.0] | a hot dog with mustard (0.88), a slice of pizza (0.74), a puppy (0.73) | a hot dog with mustard (0.45), a picnic lunch in the park (0.51), a slice of pizza (0.63) |
| a hot dog with mustard | [0.2, 0.9, 0.1, 0.0] | a slice of pizza (0.97), a bag of dog food (0.88), a picnic lunch in the park (0.71) | a slice of pizza (0.20), a picnic lunch in the park (0.32), a bag of dog food (0.45) |
| a slice of pizza | [0.0, 0.9, 0.2, 0.0] | a hot dog with mustard (0.97), a bag of dog food (0.74), a picnic lunch in the park (0.71) | a hot dog with mustard (0.20), a picnic lunch in the park (0.32), a bag of dog food (0.63) |
| a picnic lunch in the park | [0.1, 0.6, 0.1, 0.6] | a sunny afternoon (0.86), a hot dog with mustard (0.71), a slice of pizza (0.71) | a hot dog with mustard (0.32), a slice of pizza (0.32), a sunny afternoon (0.40) |
| a football match | [0.0, 0.1, 0.9, 0.2] | a tennis racket (0.97), a match stopped by rain (0.80), a sunny afternoon (0.35) | a tennis racket (0.11), a match stopped by rain (0.11), a sunny afternoon (0.11) |
| a tennis racket | [0.1, 0.15, 0.9, 0.0] | a football match (0.97), a match stopped by rain (0.65), a slice of pizza (0.37) | a sunny afternoon (0.07), a match stopped by rain (0.10), a football match (0.11) |
| a match stopped by rain | [0.1, 0.05, 0.6, 0.7] | a sunny afternoon (0.82), a football match (0.80), a thunderstorm (0.76) | a tennis racket (0.10), a football match (0.11), a thunderstorm (0.11) |
| a thunderstorm | [0.2, 0.0, 0.0, 0.9] | a sunny afternoon (0.95), a match stopped by rain (0.76), a picnic lunch in the park (0.71) | a match stopped by rain (0.11), a tennis racket (0.18), a football match (0.22) |
| a sunny afternoon | [0.05, 0.2, 0.1, 0.8] | a thunderstorm (0.95), a picnic lunch in the park (0.86), a match stopped by rain (0.82) | a tennis racket (0.07), a football match (0.11), a match stopped by rain (0.16) |
Compare the two right-hand columns for “a thunderstorm”: with all four numbers its neighbors are about weather; on the flat map, sports items crowd next to it, because the map threw away the sports and weather numbers.
With all four numbers, “a thunderstorm” is nearest to “a sunny afternoon” (0.95), “a match stopped by rain” and “a picnic lunch in the park”: all have weather in them. On the flat map it sits next to “a tennis racket” and “a football match,” whose cosine with it in the full space is about 0.02 and 0.21. Those items are not similar. The map simply dropped the two numbers that tell sports and weather apart, so they all piled up in one corner.
4 · Common mix-ups
Similar is not the same
An embedding measures how things are used together in the training data. That is powerful, and it has blind spots.
Opposites and negation. “Cheap tickets” and “expensive tickets,” or “the bus was late” and “the bus was not late,” share almost every word and context. Embedding models can score such pairs as very similar, even though for a reader they mean opposite things.
Surface overlap. In the lab, “a hot dog with mustard” has a little “animals” in its made-up vector because of the word “dog,” so its cosine with “a bag of dog food” is 0.88. Real models can make the same kind of slip.
Bias. Embeddings learn from human writing, including its stereotypes. In 2016, Tolga Bolukbasi and colleagues showed that word embeddings trained on news articles completed the analogy “man is to computer programmer as woman is to …” with “homemaker.” A 2017 study by Aylin Caliskan and colleagues found that word embeddings reproduced many biases that psychologists had measured in people. If a search or a hiring tool ranks by similarity, those patterns can quietly shape what people are shown.
Four more to watch for
Mix-up: Each number in an embedding stands for one human idea, like “animal.”
Better: The lab labels its numbers for teaching. In a real embedding, meaning is spread across many numbers and no single number has a clean label.
Mix-up: If two items look close on a 2-D chart, they are similar.
Better: The chart squashed hundreds of directions into two. Check the similarity computed with all the numbers.
Mix-up: A similarity of 0.95 means the two texts say the same thing.
Better: It means they are used in similar contexts. Opposites, negations and word overlaps can score high.
Mix-up: An embedding is a definition the model looked up.
Better: It is a learned list of numbers. It records patterns of use in the training data, including its biases.
Mix-up · trusting the picture
Why a flat map lies a little
You cannot draw 768 directions on a page. Every 2-D picture of an embedding space has to throw information away.
The lab’s flat map just dropped two numbers. Real charts are made with smarter dimensionality reduction methods, such as PCA, t-SNE or UMAP, which try to keep as much structure as possible (see Can a computer find groups nobody labeled?). They still cannot keep everything. Squashing 768 numbers into 2 is like photographing a city from one angle: some buildings that are far apart line up and look close.
Methods like t-SNE mainly try to keep each point’s nearest neighbors nearby. The sizes of clusters and the gaps between them on the picture can mean very little, and different settings can produce quite different pictures of the same data. So a chart of embeddings is a sketch for asking questions, not a measurement. To decide whether two items are similar, compute their similarity in the full space.
Rule of thumb
Trust the numbers, question the picture
Close on a 2-D chart: maybe similar. Far on a 2-D chart: maybe not different. A similarity score computed with all the numbers is the better evidence, and even that is only as good as the model that made the vectors.
5 · Check yourself
Three quick questions
Work them out on paper before you open each one.
What is the cosine similarity of [1, 2, 2] and [2, 4, 4]? And of [1, 2, 2] and [2, −1, 0]?
Check your answer
The first pair points the same way (the second is exactly twice the first), so cosine = 1. Check: dot product 2 + 8 + 8 = 18, lengths 3 and 6, and 18 ÷ (3 × 6) = 1. For the second pair the dot product is 2 − 2 + 0 = 0, so cosine = 0: the arrows are at right angles.
A chart in an article shows “river” and “bank” far apart on a 2-D map of word embeddings. Can you conclude the model sees them as unrelated?
Check your answer
No. The chart is a 2-D picture of a space with hundreds of directions, so distances on it are distorted. Compute the similarity with all the numbers before concluding anything. (“Bank” also has more than one meaning, which a single word vector blends together.)
A semantic search for “cheap train tickets” returns a page about “expensive train tickets” near the top. Why, and what could the search system add?
Check your answer
The two phrases share words and contexts, so their embeddings are close even though the meaning is opposite. The system could combine embeddings with keyword or price filters, rerank the top results with a more careful model, and show the source so a person can check.
In the real world
An application, a limit and a link
Application
Semantic search
Embed every passage once and store the vectors in a vector database. When a question arrives, embed it and return the passages with the highest similarity. That is the retrieval step in retrieval-augmented generation (RAG), and it finds “overdue fines” when you asked about “late fees.”
Limitation
Related is not relevant
The nearest passage is the most similar one, not necessarily the correct or most useful one. Many systems mix in keyword search, rerank results, and show sources so people can check the passage that was used.
Connection
Nearest neighbors, again
Finding the closest vectors is the same idea as the k-nearest-neighbors model on Classical models. Apps that suggest “more like this” use it too: see Why does the app think I want that?
For younger learners
A grades 6–8 version
No square roots: a word map on the classroom floor.
20 minutes · whole class
Human word map
- Tape two lines on the floor: “how much is it about animals?” and “how much is it about food?”
- Give students cards: puppy, pizza, hot dog, football, thunderstorm, tennis. Each stands where their card belongs.
- Ask: who is standing close to whom? Do football and thunderstorm belong together?
- Add a third question, “how much is it about weather?”, by raising hands. Now who is really close?
Exit question
Say it in one sentence
“Why can a map of meaning fool you?” A good answer: “It only shows a few directions, so things that differ in a hidden direction can look close.”
What we skipped
What this page leaves out
What this page leaves out
How embedding models are trained (for example by pulling matching pairs of sentences together and pushing others apart), other distance measures such as Euclidean distance, how vector databases find approximate nearest neighbors quickly among millions of vectors, embeddings for pictures, sound and products, and methods that try to measure or reduce bias in embeddings.
Where to go next
Put similarity to work
Search with evidence
How can an AI answer from documents?
Retrieve passages, then answer from them and cite them.
Where it starts
Words as places on a map
The transformer’s first step turns every token into a vector.
The arithmetic
How does a picture become numbers?
Vectors, matrices and the dot product, step by step.
Sources
Where this comes from
- Google for Developers, Machine Learning Crash Course: Embeddings (embedding spaces, similarity and how embeddings are learned).
- Tomas Mikolov and colleagues, “Efficient Estimation of Word Representations in Vector Space,” 2013 (word2vec).
- Nils Reimers and Iryna Gurevych, “Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks,” 2019: one vector per sentence, compared with cosine similarity.
- Martin Wattenberg, Fernanda Viégas and Ian Johnson, “How to Use t-SNE Effectively,” Distill, 2016: why cluster sizes and distances in 2-D pictures can mislead.
- Tolga Bolukbasi and colleagues, “Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings,” 2016.
- Aylin Caliskan, Joanna Bryson and Arvind Narayanan, “Semantics derived automatically from language corpora contain human-like biases,” Science, 2017.
AI words on this page
Embedding, Vector, Dot product, Semantic search, Vector database, RAG, Dimensionality reduction, k-nearest neighbors, Token, Transformer, Bias
All definitions live in one place: AI words in plain language →
Review status
Written by ARQIV. Last checked October 2026. Independent subject review and classroom pilots are still to come, so treat the activities as drafts and tell us about anything that is wrong or unclear.