What's Different When AI Reads Your Knowledge Base: Write the Test Questions First
A wiki that people understand can still trip up an AI: what it sees is a few chunks cut out of the pages, without the navigation, heading hierarchy or surrounding text. So before building RAG, write a set of test questions: each with its answer and its source passage, plus a few deliberately unanswerable ones. Azure's RAG guide does this in parallel with content analysis; one study found that evaluating only with AI-generated questions overstated retrieval performance by 29 to 37 percentage points, so mix in real users' questions. If your knowledge base is under about 500 pages, you may not need RAG at all: put the whole thing in the prompt.