AI Engineer Roadmap All lessons Interactive Quiz GitHub

RAG Evaluation (RAGAS / DeepEval)

Advanced · 11 min read · Updated · By Saurabh Pathak

In short: RAG evaluation measures, with numbers, whether your system retrieves the right context and writes answers that are correct, grounded in that context, and on-topic.

Play the interactive lesson: The evaluation loopRead on GitHub
On this page
  1. Explain it like I'm new
  2. Watch it
  3. How it works
  4. Code example
  5. When to use it / when not to
  6. Common mistakes
  7. Recap
  8. Interview questions
  9. Go deeper

Level: Advanced · Reading time: 11 min · Phase: 2 — Advanced RAG

Before you start: What is RAG and Reranking. You should know the retrieve-then-generate loop and what "top results" means.

Explain it like I'm new

A restaurant critic doesn't just say "nice". They score the ingredients, the cooking, and the service separately. If the dish is bad, they can tell you why: stale ingredients or a bad cook.

A RAG system has two parts that can fail independently: the retriever (the ingredients) and the generator (the cook). If you only look at the final answer, you can't tell which part broke. Good evaluation scores each part.

You start with a golden set (also called a test set): realistic questions, ideally with a reference answer (what a correct answer says) and the sources that should be found. You run every question through the pipeline and compute metrics. Then every change — new chunk size, a reranker, a new prompt — becomes a measured decision instead of a guess.

RAGAS and DeepEval are two popular open-source Python libraries that implement these metrics, mostly using an LLM-as-judge: a language model that reads the question, contexts and answer and makes yes/no judgements, like "is this claim supported by the context?".

Words to know:

Watch it

The evaluation loop: Measure, change one thing, measure again.
The evaluation loop: Measure, change one thing, measure again.

It plays on a loop. Read the caption in the box, one step at a time. Want to go at your own pace? Step through it with Previous and Next.

How it works

The four core RAG metrics (names as popularised by RAGAS; DeepEval has close equivalents such as faithfulness, answer relevancy, contextual precision and contextual recall):

MetricPart testedQuestion it answersNeeds reference?
Context precisionRetrieverAre the relevant chunks ranked at the top?Usually
Context recallRetrieverDid we retrieve everything needed for the reference answer?Yes
FaithfulnessGeneratorIs every claim in the answer supported by the retrieved context?No
Answer relevancyGeneratorDoes the answer actually address the question?No

Simplified formulas, with a tiny worked example:

  1. Context precision — average of precision@k at each relevant position. Retrieved relevance [1, 0, 1] → (1/1 + 2/3) / 2 = 0.83. If the relevant chunks were first, [1, 1, 0] → 1.0.
  2. Context recall — split the reference answer into claims; share supported by the contexts. 2 of 2 claims found → 1.0.
  3. Faithfulness — split the generated answer into claims; share supported by the contexts. "Stored in an httpOnly cookie" (supported) + "expire after exactly 30 days" (not in context) → 1/2 = 0.5. That is a hallucination caught.
  4. Answer relevancy — RAGAS generates questions from the answer and compares them to the real question by embedding similarity. An evasive or off-topic answer scores low.

Worked example

The sample in the code below asks "Where should refresh tokens be stored?". The retriever returned three chunks: (1) refresh tokens go in an httpOnly cookie and should be rotated, (2) a Docker Compose sentence, (3) "access tokens should be short lived". A person marks them relevant, irrelevant, relevant: [1, 0, 1]. The answer was "Refresh tokens should be stored in an httpOnly cookie. They expire after exactly 30 days."

Reading the four together: retrieval was decent (small ranking flaw), but the generator invented a fact. You would fix the prompt or generator, not the retriever. These match the real output of the demo.

flowchart LR
    G[Golden set:<br/>question, reference] --> P[Your RAG pipeline]
    P --> C[Retrieved contexts]
    P --> A[Generated answer]
    C --> CP[Context precision]
    C --> CR[Context recall]
    G --> CR
    A --> F[Faithfulness]
    C --> F
    A --> AR[Answer relevancy]
    G --> AR

Also keep the simple retrieval metrics from this phase's harness: hit@k and MRR (see eval_harness.py). They are deterministic, free and fast.

Code example

adv_rag_metrics.py implements toy versions of all four metrics, replacing the LLM judge with sentence splitting and word overlap so the formulas are visible:

from adv_rag_metrics import SAMPLE, evaluate

print(evaluate(SAMPLE))
# {'context_precision': 0.833, 'context_recall': 1.0, 'faithfulness': 0.5, 'answer_relevancy': 1.0}

The faithfulness core:

def faithfulness(answer, contexts, threshold=0.6):
    ans = claims(answer)                 # an LLM splits claims in real tools
    if not ans:
        return 0.0
    return sum(supported(c, contexts, threshold) for c in ans) / len(ans)

In real projects use RAGAS or DeepEval. Both build an evaluation dataset of samples (question, retrieved contexts, answer, optional reference), choose metrics, and call an evaluate function that uses a judge LLM you configure. Their APIs have changed across major versions (for example, field names and metric classes), so copy the current quick-start from the official docs rather than older blog posts:

When to use it / when not to

Common mistakes

Recap

If you remember only three things:

Interview questions

1. Explain faithfulness vs answer relevancy. <details><summary>Answer</summary> Faithfulness checks that every claim in the answer is supported by the retrieved context (no hallucination). Answer relevancy checks that the answer addresses the question. An answer can be fully faithful but irrelevant (quotes the context, misses the question), or relevant but unfaithful (on-topic but invented). </details>

2. Context recall is low but faithfulness is high. What does that tell you? <details><summary>Answer</summary> The generator is behaving (only uses what it gets), but the retriever is missing needed information. Work on retrieval: chunking, hybrid search, query transformation, larger k, or reranking. </details>

3. How would you set up RAG evaluation in CI? <details><summary>Answer</summary> Keep a versioned golden set. On every PR run cheap deterministic checks (hit@k, MRR, schema validity) and fail on regressions beyond a threshold. Run LLM-judged metrics (faithfulness, relevancy, context precision/recall) nightly or before release with a pinned judge model, and track trends over time. </details>

Go deeper

More interview answers: Interview prep — evaluation and production.

Prev: Streaming Retrieval · Next: LLM-as-a-Judge · Back to phase: Phase 2