Measuring Retrieval

Building the labelled set

"The answer was wrong" is not a retrieval metric. It mixes extraction, chunking, ranking and generation, and it cannot tell you which layer to fix. A retrieval metric needs a question, the passages that answer it, and a ranked list to compare. The retrieval contract introduced Harbourline's set of about 400 real questions; this section covers how to build one that holds up.

Sample from logs, by slice. A random sample of 400 over-represents the common questions and leaves the exclusions and identifier slices with a handful of items. Stratify: for each of the five slices (limits and deductibles, exclusions, procedures, definitions, identifiers) draw a target count, around 80 each, and weight the report back to traffic where you need a single headline number. Remove duplicates and near-duplicates, and strip personal data before a human or a model sees the log.

Judge relevance. Binary judgments (this passage answers the question or it does not) are quick and enough for recall and MRR. Graded judgments (2 fully answers, 1 supports or partly answers, 0 irrelevant) are needed for nDCG and matter when several passages are partial, as with a limit in one chunk and its exception in another. Many questions have more than one relevant passage, so label all of them: a set that records only the first passage you found punishes a retriever for returning an equally good one. Also record the document version, because "the deductible" in the 2024 wording is the wrong answer to a 2025 policyholder.

Use a model to pre-label and a person to audit. Have a strong model propose relevant passages from the top 50 of several retrievers, then have a domain reviewer confirm or reject each. This finds the passages your current retriever would not rank, so the set does not simply echo today's system. Measure inter-annotator agreement on a sample of 50 that two reviewers label independently, for example with Cohen's kappa. Low agreement means the relevance definition is vague, and no metric built on it can be trusted until it is tightened. Audit a slice of the model's labels against the human ones and report the disagreement rate.

Size sets the differences you can see. With 80 questions in a slice and a recall near 0.8, the standard error of one measurement is about 0.045, so an unpaired comparison cannot resolve a gap of a few points. Compare two systems on the same questions and use a paired bootstrap, which is far tighter than that figure suggests. Even so, treat a difference smaller than the interval as noise. If a decision hinges on a two-point gain on a slice, you need more questions on that slice, not a closer look at the table.

The metrics and what each hides

Choose by the decision. To pick k, a chunk size or an embedding model, report recall@k at the k you will pass to the model, because the generator cannot use what is not in the prompt. To compare a reranker or fusion setting, report nDCG@10 alongside recall@5, because ordering is what changed. MRR suits the identifiers slice, where there is usually one right passage. Always report per slice. A single average lets a gain on the large slice cover a regression on the small one that carries the regulatory risk.

Computing them

The function below takes a ranked list of chunk ids and a dictionary of graded labels. It was run on the example at the bottom, which prints recall@3 of 0.5, precision@3 of 0.33, MRR of 0.5 and nDCG@5 of 0.64.

import math

def recall_at_k(ranked, rel, k):
    """rel: dict chunk_id -> grade (0 = not relevant)."""
    gold = {c for c, g in rel.items() if g > 0}
    return len(gold & set(ranked[:k])) / len(gold) if gold else None

def precision_at_k(ranked, rel, k):
    return sum(rel.get(c, 0) > 0 for c in ranked[:k]) / k

def mrr(ranked, rel):
    for i, c in enumerate(ranked, start=1):
        if rel.get(c, 0) > 0:
            return 1.0 / i
    return 0.0

def ndcg_at_k(ranked, rel, k):
    dcg = sum((2 ** rel.get(c, 0) - 1) / math.log2(i + 1)
              for i, c in enumerate(ranked[:k], start=1))
    ideal = sorted(rel.values(), reverse=True)[:k]
    idcg = sum((2 ** g - 1) / math.log2(i + 1) for i, g in enumerate(ideal, start=1))
    return dcg / idcg if idcg else None

rel = {"c2": 2, "c9": 1}                 # c2 answers fully, c9 supports
ranked = ["c7", "c2", "c4", "c9", "c1"]
print(recall_at_k(ranked, rel, 3), precision_at_k(ranked, rel, 3),
      mrr(ranked, rel), round(ndcg_at_k(ranked, rel, 5), 3))

Running a sweep as an experiment

A sweep is an experiment, so give it an experiment's rules. Fix the question set, the corpus snapshot and the index build. Change one thing: chunk size, the embedding model, the fusion constant, the reranker depth. Record the configuration with the result so a number can be reproduced. Report a table with one row per configuration and one column per slice, with the paired difference from the baseline and its interval, not the raw value alone.

Read the table by slice, then by cost. A row that moves the average but drops the exclusions column below the baseline fails, because an exclusion misread to a customer during a claim is the costly error. Run each configuration once to find candidates, then confirm the winner on a held-out part of the set that you did not look at while tuning. Without that, you are fitting the set, and the gain will not show up in production. See the chunk size interview chapter for a worked version.

Keeping the set alive

The set rots. Harbourline publishes several wording versions per product per year, so a gold passage ID can vanish or change. Anchor labels to a stable identifier for the clause (document, section, version) and not to a chunk ID, then map to chunks whenever chunking changes. Re-sample the logs each quarter to catch new question types, and add every confirmed production failure from the debugging chapter as a permanent item. Keep a frozen core for trend lines and a rolling part for realism, and note which one a number came from. When a label is overturned, record the change and re-run the last baseline, or comparisons across the change are not like for like.

✓ Check yourself

Answer from memory before opening each one.

1. Support says the assistant "got worse after the last release". You have only thumbs-down counts. Why can they not tell you whether retrieval regressed?

   Show answer

   A wrong answer can come from extraction, chunking, ranking or generation, and thumbs-down mixes them. A retrieval metric needs the question, the labelled passages and the ranked list, so you can fix the model and corpus and watch only retrieval move. See Building the labelled set.

2. You pass 6 chunks to the model. Which of recall@5, recall@6 and MRR do you report to decide between two embedding models, and why?

   Show answer

   Recall@6, because the generator can use anything inside the six and nothing outside it. MRR rewards rank 1 over rank 6, which does not matter to a reader who sees all six. See The metrics and what each hides.

3. A new retriever scores recall@10 of 0.74 against 0.80 for the old one, but when you read the misses many returned passages look correct. What went wrong, and what do you do?

   Show answer

   The labels are incomplete: they came from the old system, so correct passages that only the new one finds count as misses. Re-label by pooling the top results of both systems and re-run. Do not reject the new retriever on the first number.

4. A sweep shows a two-point recall gain on the exclusions slice of 80 questions. The change adds a day of work to every re-index. How do you decide?

   Show answer

   First get the paired interval on those 80 questions. If it includes zero, the gain is noise, and the choice is to add questions to that slice and re-run. If it holds, weigh the gain against the cost on the slice with the highest risk. A small, real gain on exclusions can justify a day, while the same gain on definitions may not.

5. Two reviewers agree on only 60% of relevance labels for the limits slice. What do you do before running any sweep on it?

   Show answer

   Fix the definition: decide whether a chunk with the limit but not its exception counts as relevant, write that rule down, and relabel. Metrics computed on labels that two people dispute measure the disagreement, not the retriever.

Takeaway

• A retrieval metric needs a question, labelled passages and a ranked list, which is why "the answer was wrong" cannot stand in for one.
• Stratify the labelled set by slice, label every relevant passage, and anchor labels to clauses so chunking changes do not break them.
• Model pre-labels with a human audit and a measured agreement rate give a set that is cheap to build and still trustworthy.
• Report recall@k at the k you pass to the model, and nDCG@10 when ordering is what you changed.
• A sweep changes one thing, uses paired comparison with an interval, reports per slice, and confirms the winner on held-out questions.
• Labels pooled from several systems stop a better retriever from being scored as a miss.

With retrieval measured, the next question is what the model does with the passages it gets. → Context Assembly and Grounding

Further reading

• BEIR — Thakur et al., 2021. A model for evaluating retrievers across datasets rather than one.
• Passage Re-ranking with BERT — Nogueira and Cho, 2019. An example of reporting MRR@10 on a labelled passage benchmark.

See also Debugging a wrong RAG answer for the same bisection from the interviewer's side.