Keeping the index fresh
Documents change, get deleted and get new versions. Your index must follow, or the bot will confidently quote rules that no longer exist.
In 60 seconds
Keeping the index fresh
Documents change, get deleted and get new versions. Your index must follow, or the bot will confidently quote rules that no longer exist.
Incremental updates instead of rebuilding everything
import hashlib
def fingerprint(text: str) -> str:
return hashlib.sha256(text.encode("utf-8")).hexdigest()
def plan_sync(source_docs: dict[str, str], indexed: dict[str, str]) -> dict:
"""source_docs: doc_id -> current text. indexed: doc_id -> stored fingerprint."""
add, update = [], []
for doc_id, text in source_docs.items():
fp = fingerprint(text)
if doc_id not in indexed:
add.append(doc_id)
elif indexed[doc_id] != fp:
update.append(doc_id)
delete = [doc_id for doc_id in indexed if doc_id not in source_docs]
return {"add": add, "update": update, "delete": delete}
source = {"leave-policy": "12 days casual leave", "travel-policy": "Rs 2,000 per day"}
indexed = {"leave-policy": fingerprint("10 days casual leave"), "old-memo": "abc"}
print(plan_sync(source, indexed))
# {'add': ['travel-policy'], 'update': ['leave-policy'], 'delete': ['old-memo']}doc_id.Deletions are the step people forget
- Detect them. A document that disappears from the source must disappear from the index. A sync that only adds and updates will keep deleted documents searchable for ever.
- Delete everything derived from it. Chunks, vectors, keyword index entries, cached answers, and any summaries made from it.
- Check it worked. Some indexes mark items as deleted and clean up later. After a deletion, search for the document and confirm it no longer appears.
- Permissions are changes too. If a user loses access to a folder, the index filters must update quickly. See Securing RAG: permissions for retrieved documents.
- Personal data requests. Deletion may be a legal duty, not just tidiness. See Vector stores and embedding leaks for erasure and backups.
Versions: old and new side by side
| Approach | When it fits |
|---|---|
| Index only the latest version | Most policy and FAQ bots; simplest and safest |
| Index all versions with an "effective from" date and a "current" flag | When people ask "what was the rule in March?"; filter to current by default |
| Mark replaced documents clearly in the chunk text | So even if an old version is retrieved, the model sees "replaced by circular 14/2025" |
Re-embedding when the model changes
- 1
Build a new index
Embed all chunks with the new model into a separate index. The old one keeps serving users. - 2
Evaluate
Run your golden question set on both indexes. Compare recall and answer quality. - 3
Switch
Point the app to the new index in one step, for example by changing an alias or a setting. - 4
Keep the old one briefly
So you can switch back quickly if something goes wrong. - 5
Clean up
Delete the old index and record which model version the new one uses.
- Rebuilt by hand "when someone remembers"
- Only adds new files, never removes old ones
- Old and new embedding models mixed in one index
- No record of which model or settings made it
- Scheduled or event-driven sync with add, update and delete
- Deletions checked by searching after they run
- Full re-embed into a new index, tested, then switched
- Model, settings and date stored with the index
Watch and read more
Lab
A working sync planner with deletion checks, plus a written re-embedding plan.
The problem
Use the plan_sync code from the lesson (Python 3, standard library).
- Create a small in-memory "index": a dictionary from chunk id to
{"doc_id": ..., "text": ...}, built from three made-up documents. - Write
apply_sync(plan, source_docs, index)that deletes all chunks of deleted and updated documents, and adds chunks for new and updated ones. - Change one document, delete one and add one. Run the sync. Print the index and confirm no chunk of the deleted document remains.
- Write a five-step plan for moving the index to a new embedding model without downtime.
You are done when
Hard questions
Try to answer before you reveal. If you can answer these, you understood the lesson.
Q1Your source system cannot tell you which files were deleted; it only lists current files. How do you still detect deletions?Reveal
Compare lists. Keep a record of every doc_id in the index. On each sync, fetch the full list of current doc_ids from the source; anything in the index but not in the source was deleted. This is what plan_sync does. For very large sources, run this full comparison on a slower schedule, and use change events for fast updates in between.
Q2After you delete a document, a user still gets an answer quoting it. Name three places the text could still be hiding.Reveal
An answer cache that stored an earlier response; a second index (for example the keyword index) that was not updated; and derived data such as summaries or a "related questions" list made from that document. Other candidates: an index that marks deletions but has not cleaned up yet, a replica that has not synced, or a backup that was restored. Checks after deletion should search every path the app uses.
Questions people ask
How often should the index sync?
As often as the documents change and users need freshness. A policy library might sync hourly or when a document is published. A news-like source may need minutes. Show the "last updated" time so users know how current answers are.
Can I update only the changed chunk instead of the whole document?
It is possible with careful chunk ids, but error-prone, because a small edit can shift every later boundary. Replacing all chunks of a changed document is simpler and safer, and the cost is usually small.
What about answer caches?
A cache must be cleared, or keyed by index version, when documents change. Otherwise the cache keeps serving answers built from old or deleted text.
Do I need to re-embed if I only change chunk size?
Yes. New chunk boundaries mean new chunk texts, and each needs a new vector. Treat it like a model change: build a new index, evaluate, then switch.
Lesson test
5 questions. Get 3 right (60%) to pass and complete this lesson.