How to Evaluate RAG Pipelines Without Fooling Yourself: A 5-Step Checklist
Fluency is not accuracy. Use this five-step checklist to test ingestion, gold-query recall, token-bounded packing, and honest refusals.
It is easy to build a Retrieval-Augmented Generation (RAG) demo that sounds confident. It is much harder to build one that cites the exact file and line number—and admits when the answer isn't in the corpus.
Step 1: Verify Your Ingestion Manifest
Before testing embeddings, assert that your indexer actually discovers all expected documents, splits them on sensible section boundaries, and records stable file paths and line ranges.
Step 2: Grade Gold-Query Recall Separately from Generation
Build a small table of 15–30 "gold" questions where you know the exact target file. Check whether your retriever surfaces that file in the top k results before you spend a single token grading the LLM's prose.
Step 3: Test Token-Bounded Context Packing
More context isn't always better. Use a greedy packer that admits the highest-scoring reranked chunks until it hits a hard token budget, preserving metadata headers on every chunk.
Step 4: Require Verbatim Path and Line Citations
Instruct the model to cite the exact path:line_start-line_end from the admitted chunks, and verify in post-processing that every cited path actually appeared in the retrieved context.
Step 5: Include Unanswerable "Trap" Queries
At least 25% of your eval suite should ask about topics that do not exist in your corpus. A passing grade means the system abstains cleanly and never invents a plausible-sounding filename.