How to Evaluate RAG Pipelines Without Fooling Yourself: A 5-Step Checklist

Fluency is not accuracy. Use this five-step checklist to test ingestion, gold-query recall, token-bounded packing, and honest refusals.

Abstract editorial still life of loose sheets passing through a sequence of lenses and focusing onto a small block

It is easy to build a Retrieval-Augmented Generation (RAG) demo that sounds confident. It is much harder to build one that cites the exact file and line number—and admits when the answer isn't in the corpus.

Step 1: Verify Your Ingestion Manifest

Before testing embeddings, assert that your indexer actually discovers all expected documents, splits them on sensible section boundaries, and records stable file paths and line ranges.

Step 2: Grade Gold-Query Recall Separately from Generation

Build a small table of 15–30 "gold" questions where you know the exact target file. Check whether your retriever surfaces that file in the top k results before you spend a single token grading the LLM's prose.

Step 3: Test Token-Bounded Context Packing

More context isn't always better. Use a greedy packer that admits the highest-scoring reranked chunks until it hits a hard token budget, preserving metadata headers on every chunk.

Step 4: Require Verbatim Path and Line Citations

Instruct the model to cite the exact path:line_start-line_end from the admitted chunks, and verify in post-processing that every cited path actually appeared in the retrieved context.

Step 5: Include Unanswerable "Trap" Queries

At least 25% of your eval suite should ask about topics that do not exist in your corpus. A passing grade means the system abstains cleanly and never invents a plausible-sounding filename.

More stories