When documents change or are deleted, the index doesn't keep up on its own. A sturdier incremental sync uses a content hash to find which documents changed and redoes only those; deletion has to reach every place a document is stored, including the full-text index and the vector store. Many tools don't handle deletion by default, and deletes and updates can take a while to take effect. Why clean up thoroughly: one preprint put a single outdated document into the context, and two small models flipped three to four in ten medical questions they had answered correctly; when told to answer from the documents, that rose to between two-thirds and three-quarters.
Ask AI indexing runs in two production stages: source-hash changes update D1, post chunks, and FTS5 first; embedding checkpoints and a delete queue then let Vectorize catch up asynchronously. The two stores do not share one transaction.
Private-corpus sync is not periodic refetching. It requires stable canonical IDs, source versions plus checksums for change detection, idempotent upserts, and tombstones that propagate deletion through every index.
Chunks too large and retrieval loses precision; too small and you lose context; hit a table and retrieval falls apart entirely. Chunking is the most underrated part of RAG — pick the wrong strategy and no amount of downstream optimization will save you.
When you split a document into chunks, each chunk loses its place in the original document. Contextual Retrieval solves the isolated-chunk problem by generating a per-chunk context from the whole document and prepending it at index time.
A RAG system needs data to answer questions, but data only accumulates as the system gets used. Cold-start strategy is what bridges the gap from empty to useful.