RAG and search engineeringMiddleLesson 2086 min read

Embeddings and similarity search

An embedding turns text into a list of numbers, so that texts with similar meaning land close together. Learn how similarity is measured and how search stays fast.

✦ Ask the tutor 🧪 Lab · 25 min 📝 Lesson test

Speed

Lesson in motion

In 60 seconds

Embeddings and similarity search

An embedding turns text into a list of numbers, so that texts with similar meaning land close together. Learn how similarity is measured and how search stays fast.

1/3

In simple words

Think of a map of a city. Shops that sell similar things are often close together: all the phone shops in one lane, all the sweet shops in another. An embedding puts every sentence on a kind of map, so that sentences with similar meaning end up near each other. To search, you find where your question lands and look at its neighbours.

An embedding model reads a piece of text and returns a vector: a fixed-length list of numbers. Real models return hundreds or thousands of numbers per text. We cannot read the numbers one by one, but the whole list captures meaning. "My money was cut but the payment failed" and "Refund for a failed UPI transaction" get vectors that point in similar directions, even though they share few words.

Measuring closeness: cosine similarity

Cosine similarity measures the angle between two vectors. It is 1 when they point the same way, near 0 when they are unrelated, and negative when they point in opposite directions. It ignores the length of the vectors and only looks at direction.

Many systems normalise every vector to length 1 when they store it. Then cosine similarity is just the dot product: multiply matching numbers and add them up. That is fast.

Python: cosine similarity with made-up vectorspython

import numpy as np

# Made-up 4-number "embeddings", only to show the maths.
# Real embedding models return hundreds or thousands of numbers.
docs = {
    "How to reset your UPI PIN":       np.array([0.9, 0.1, 0.0, 0.2]),
    "Refund for a failed UPI payment": np.array([0.7, 0.7, 0.1, 0.0]),
    "Monsoon sowing tips for paddy":   np.array([0.0, 0.1, 0.9, 0.3]),
}
# Pretend this is the vector for "money was cut but the payment failed".
query = np.array([0.6, 0.8, 0.0, 0.1])

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

ranked = sorted(docs, key=lambda title: cosine(query, docs[title]), reverse=True)
for title in ranked:
    print(f"{cosine(query, docs[title]):.3f}  {title}")

Run it and the refund document comes first, the PIN document second, and the farming tips last. A real system does exactly this, only with real vectors from an embedding model and many more documents.

Making search fast: approximate nearest neighbour

Comparing the question with every stored vector is called exact or brute-force search. It is simple and perfectly accurate, and for a few thousand chunks it is often fast enough. For millions of chunks it gets slow.

Approximate nearest neighbour (ANN) indexes trade a little accuracy for a lot of speed. A popular one is HNSW (Hierarchical Navigable Small World). It builds a graph in layers: the top layers have few points and long links for big jumps, the lower layers have more points and short links for fine steps. A search starts at the top, jumps close to the right area, and then walks down to the nearest neighbours.

Choice	Good for	Watch out for
Brute-force (exact)	Small collections, tests, checking ANN results	Slow at large scale
HNSW graph	Fast search on large collections	Uses more memory; may miss some true neighbours; settings trade speed for recall
Other ANN methods (for example clustering or compression)	Very large collections, lower memory	Usually lose more accuracy; need tuning

Note

"Approximate" means the index can miss a few true nearest neighbours. Measure this: run the same questions through brute-force search and through the index, and see how many of the exact top results the index also returns. Index settings usually let you trade speed for this recall. Check your database's current docs for the names of those settings.

Things that trip people up

Use the same model for documents and questions. Vectors from two different embedding models live on different "maps" and cannot be compared.
Some models expect a prefix or a mode. Some embedding models want you to mark text as a query or a document. Read the model card.
Similar is not the same as correct. "Refund allowed within 7 days" and "Refund not allowed after 7 days" can be very close in vector space. Similarity finds candidates; the answer step must read carefully.
Exact words can get lost. Product codes, error numbers and rare names are often matched badly by vectors alone. The hybrid search lesson fixes this.
Language coverage varies. Some embedding models handle Hindi, Tamil or Bengali well, others poorly. Test with your users' real questions.

Try this. Change the query vector in the code to [0.0, 0.2, 0.8, 0.4]. Which document comes first now, and why?

Watch and read more

Hugging Face: models and documentationHugging Face · reading

Lab

A working brute-force vector search over made-up vectors, and a note on what each result shows.

~25 min

The problem

You need Python 3 and numpy (pip install numpy). No API key is needed; all vectors are made up.

Run the starter code and note the ranking.
Add two more documents with your own made-up vectors, for example about bus passes or exam results. Choose the numbers so that one of them should match the query.
Change search to return only results with similarity above a threshold. Try 0.5 and 0.9 and write what changes.
Write two sentences on why a real threshold must be chosen by testing, not guessed.

Practice lab

Edit the code and run it. Python runs in your browser — nothing is sent to our server.

Ctrl + Enter to run

Output

No output. Add print(...) to see values.

You are done when

The code runs and prints a ranked list with scores
The two new documents have vectors chosen on purpose, and the expected one ranks high
The threshold version returns fewer results at 0.9 than at 0.5
The note explains that scores depend on the model and the data

Hard questions

Try to answer before you reveal. If you can answer these, you understood the lesson.

Q1Why does the starter normalise vectors before building the matrix?Reveal

After normalising every vector to length 1, the dot product of two vectors equals their cosine similarity. That lets the code score all documents with one fast matrix multiplication, instead of dividing by the lengths for every pair. It also means long and short vectors are compared only by direction.

Q2A vector search returns "Refund not allowed after 7 days" as the top result for "Can I get a refund within 7 days?". Is the search broken?Reveal

Not necessarily. The two texts are about the same topic and share most of their meaning, so they sit close together in vector space. Retrieval's job is to find relevant candidates; the answer step must read the chunk carefully and answer correctly. A re-ranker that reads the question and chunk together can also help order such cases better.

Please sign in to continue.

Questions people ask

How do I choose an embedding model?

Make a small test set of real questions with known right chunks, and try two or three candidate models on it. Look at quality in your languages, vector size (bigger means more storage), speed, cost and licence. Public leaderboards are a starting point, but your own test matters more.

Should I use cosine similarity or dot product?

If vectors are normalised to length 1, they give the same ranking. Many embedding models recommend one; follow the model card and use the same choice when building the index and when searching.

Can someone read my documents from the vectors?

Treat embeddings as being as sensitive as the text they came from. Researchers have shown partial text recovery in some settings. See Vector stores and embedding leaks for controls.

What score means "relevant"?

There is no universal cut-off. Scores depend on the model and the data. If you want a threshold, choose it by testing on your own labelled questions, and check it again whenever you change the model.

Lesson test

5 questions. Get 3 right (60%) to pass and complete this lesson.