Evaluate RAG Overview 1. Do error analysis on end-to-end traces first. Determine whether failures come from retrieval, generation, or both. 2. Build a retrieval evaluation dataset: queries paired with relevant document chunks. 3. Measure retrieval quality with Recall@k (most important for first-pass retrieval). 4. Evaluate generation separately: faithfulness (grounded in context?) and relevance (answers the query?). 5. If retrieval is the bottleneck, optimize chunking via grid search before tuning generation. Prerequisites Complete error analysis on RAG pipeline traces before selecting metric…