Small chunks give precise embeddings but lack context; big chunks have complete context but diluted embeddings. Hierarchical Chunking builds multi-level indexes (2048→512→128 tokens) with an Auto-Merge algorithm: leaf nodes match precisely, and when hit density exceeds a threshold the parent node is returned to the LLM instead. HiChunk shows a 12.7% evidence recall improvement; LlamaIndex and Haystack have it built in.
Markdown-KV format achieves 60.7% LLM comprehension accuracy vs 44.3% for CSV — a 16-point gap from format alone. But retrieval and comprehension have different optimal formats: metadata prepend + row-wise key-value is the current best combination for table RAG.
Ask AI indexing runs in two production stages: source-hash changes update D1, post chunks, and FTS5 first; embedding checkpoints and a delete queue then let Vectorize catch up asynchronously. The two stores do not share one transaction.
Anthropic Contextual Retrieval uses an LLM to prefix each chunk with 50-100 tokens, cutting failure rate from 5.7% to 1.9% with rerank at ~$1.02/1M tokens; Late Chunking encodes the full 32K-window document first then mean-pools by chunk boundaries for zero extra LLM cost — the trade-off is window, latency, and update shape.
Traditional RAG splits documents into small chunks for retrieval, but this causes information fragmentation. LongRAG leverages 100K+ token long-context models to retrieve larger document segments (entire sections or even whole documents), reducing fragmentation while maintaining retrieval efficiency.
Chunks too large and retrieval loses precision; too small and you lose context; hit a table and retrieval falls apart entirely. Chunking is the most underrated part of RAG — pick the wrong strategy and no amount of downstream optimization will save you.
When you split a document into chunks, each chunk loses its place in the original document. Contextual Retrieval solves the isolated-chunk problem by generating a per-chunk context from the whole document and prepending it at index time.