Re-ranking and query rewriting
The first search casts a wide net. A re-ranker reads each candidate closely and puts the best on top. Query rewriting fixes questions that are vague, short or in a different language or script.
In 60 seconds
Re-ranking and query rewriting
The first search casts a wide net. A re-ranker reads each candidate closely and puts the best on top. Query rewriting fixes questions that are vague, short or in a different…
Two stages: retrieve, then re-rank
| Bi-encoder (embeddings) | Cross-encoder (re-ranker) | |
|---|---|---|
| Reads | Question and chunk separately | Question and chunk together |
| Speed | Fast; chunk vectors are pre-computed | Slower; one model call per question-chunk pair |
| Use it for | Searching the whole collection | Re-ordering the top candidates, for example 20 to 100 |
| Quality | Good at finding candidates | Often better at choosing the very best few |
- 1
Wide first search
Hybrid search returns, say, the top 50 candidates. - 2
Re-rank
The cross-encoder scores each question-chunk pair. - 3
Cut
Keep the best few, for example 5, for the prompt. - 4
Measure
Check that re-ranking really improves results on your test set, and what it adds to latency.
Query rewriting
- Make follow-ups standalone. "And for contract staff?" becomes "casual leave rules for contract staff", using the chat history.
- Multi-query. Ask a model for 3–4 different phrasings, search with each, and fuse the results with RRF. This helps when documents use different words from users.
- Expand terms. Add known synonyms and abbreviations: "PF" and "provident fund", "DOJ" and "date of joining".
- Hypothetical answer. Ask a model to write a short imagined answer and search with that text, because it may look more like the documents than the question does. This is sometimes called HyDE. The imagined answer is only for searching; never show it as the answer.
def call_model(prompt: str) -> str: ... # plug in your provider
def hybrid_search(query: str, k: int = 20) -> list[str]: ... # your search, returns chunk ids
def rrf(rankings, k=60):
scores = {}
for ranking in rankings:
for rank, doc_id in enumerate(ranking, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=lambda d: (-scores[d], d))
def multi_query_search(question: str, n: int = 3) -> list[str]:
prompt = (
f"Write {n} different search queries for this question, one per line. "
"Keep names, numbers and codes exactly as written.\n\nQuestion: " + question
)
rewrites = [q.strip() for q in call_model(prompt).splitlines() if q.strip()][:n]
queries = [question] + rewrites # always keep the original
return rrf([hybrid_search(q) for q in queries])Indian-language and transliterated queries
| Problem | What helps |
|---|---|
| Keyword search misses because the query and documents use different scripts | Transliterate the query into the documents' script, or translate it, and search with both versions |
| Many spellings of the same romanised word | Normalise common spellings; rely more on vector search; add a synonym list from real logs |
| Embedding model is weak in the user's language | Test multilingual embedding models on your real queries; translate the query to the documents' language as an extra search |
| Mixed language in one query (code-mixing) | Keep the English terms such as "order" and "refund" as keywords; run a translated version too; fuse with RRF |
| Answer should be in the user's language | Search in the documents' language, then ask the model to answer in the user's language, citing the original |
Watch and read more
Lab
A multilingual query test sheet and a written rewriting plan for a support bot.
The problem
Imagine a support bot for an online grocery app used across India. Its help pages are in English and Hindi.
- Write 10 real-looking user questions: at least 3 in romanised Hindi or another romanised Indian language, 2 in a native script, 2 code-mixed, 1 follow-up question, and 1 with an order id.
- For each, write the query or queries you would actually search with after rewriting (standalone, translated, transliterated, or unchanged).
- Mark which questions must not be rewritten freely, and why (for example, the order id).
- Write three rules for your rewriting prompt that stop drift.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Multi-query rewriting made your answers better on average but much worse for questions with product codes. What is the likely cause, and the fix?Reveal
The rewriter probably changed, dropped or "corrected" the codes, and the rewritten queries then dominated the fused list. Fixes: keep the original query as one of the searches, instruct the rewriter to copy codes and names exactly, and skip rewriting when the query contains an id pattern. Then measure code-heavy questions separately in your test set.
Q2Your re-ranker improved English results but made Hindi results worse. What would you do?Reveal
Check the re-ranker's training languages; it may simply be weak in Hindi. Options: try a multilingual re-ranker, re-rank a translated version of the question and chunk, or use the re-ranker only for languages where it helps. Keep separate test scores per language so a gain in one language cannot hide a loss in another.
Questions people ask
How many candidates should I re-rank?
Enough that the right chunk is usually among them, and few enough that latency is acceptable. Measure recall at your candidate count: if the right chunk is often at position 80, re-ranking only the top 20 cannot help.
Does re-ranking always help?
Often, but not always. A re-ranker trained mostly on English web text may do poorly on your Tamil support tickets. Compare with and without it on your own test set.
Is query rewriting worth the extra model call?
For short, vague and follow-up questions it often is. For precise searches such as a code lookup, it adds cost and risk. Some systems rewrite only when the question looks like a follow-up or when the first search finds little.
Should I translate all my documents into English instead?
It is one option, but translation can lose meaning, and citations should point to the original text. Many teams keep the originals, and translate or transliterate the query instead, then answer in the user's language.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.