Evaluating RAG Systems: Metrics That Catch Real Failures

Most RAG systems are tuned by vibes: someone asks a few questions, the answers "look good," and it ships. Then it quietly fails in production on the queries nobody tried. The fix is the same as any other engineering discipline — measure it — but RAG needs metrics tailored to its two-stage nature: retrieval and generation.

Why generic accuracy isn't enough

A RAG answer can be wrong in two very different ways:

• Retrieval failed — the right information was never fetched, so the model couldn't possibly answer.
• Generation failed — the right context was retrieved, but the model ignored it, misread it, or added unsupported claims.

A single "is the answer correct?" score can't tell these apart — and the fixes are opposite (improve chunking/retrieval vs. improve the prompt/model). You need to score the stages separately.

The four metrics that matter

1. Faithfulness — is the answer grounded in the retrieved context, or did the model invent things? This is your hallucination detector. Low faithfulness = generation problem.

2. Answer relevance — does the answer actually address the question (vs. drifting or padding)? Catches on-topic-but-useless responses.

3. Context precision — of the chunks retrieved, how many were actually relevant? Low precision = noisy retrieval diluting the context.

4. Context recall — of the information needed to answer, how much did retrieval actually fetch? Low recall = the answer was doomed before generation. This is usually where RAG quality is won or lost.

Frameworks like RAGAS compute these automatically (often using an LLM as judge), so you can score hundreds of examples without manual grading.

Build the eval loop

1. Golden set. Assemble 30–100 representative question/answer pairs from real or expected queries — include the hard and edge cases, not just easy ones.
2. Baseline. Run the current system, score the four metrics, record the numbers.
3. Change one thing. Adjust chunking, top-k, the prompt, or the model — one variable at a time.
4. Re-score and compare. Did context recall improve? Did faithfulness hold? Let the metrics, not vibes, decide.
5. Gate it in CI. Run the eval on every change to the retrieval/prompt path and fail the build if a metric regresses below threshold.

Reading the signals

• Low context recall → fix retrieval first: chunking, embeddings, top-k, hybrid search. No prompt change helps if the data isn't fetched.
• Good recall, low faithfulness → generation problem: tighten the prompt ("answer only from the context"), lower temperature, or change model.
• Good faithfulness, low answer relevance → the right facts, wrong framing — improve the instruction.

Wrap-up

RAG is a measurable system, not a black box. Score retrieval and generation separately with faithfulness, answer relevance, and context precision/recall; optimize the weakest stage; and gate changes in CI. That loop turns "looks good" into "provably better."

Related reading

• RAG Systems Explained — the pipeline these metrics evaluate.
• RAG Chunking Strategies — the biggest lever on context recall.
• Testing AI Applications — wiring evals into your CI/CD quality gate.