The Retrieval Contract

A wrong answer, a right index

A Harbourline customer asks what the wind and hail deductible is on their homeowners policy. The assistant answers $1,000. The policy wording sets it at 2.5% of the dwelling limit. The engineer on call opens the index and finds the right page there, correctly extracted, with the deductible table intact. The model is not hallucinating in the usual sense. It read a number from somewhere and reported it faithfully.

That "somewhere" is the whole subject of this guide. The generator can only use what retrieval hands it, so each failure of this kind is a breach of an agreement nobody wrote down. Here the table that got through was last year's version of the wording, ranked above this year's because the two are almost the same text. Retrieval returned a passage that answered the question and was out of date. It looked like success on every metric that checks only for the presence of an answering passage.

At roughly 13,000 assistant sessions a day, a wrong limit in 2% of the sessions that ask about limits is a few hundred bad answers a week, each one a possible complaint and refund. The fix is to say what retrieval owes the generator in terms precise enough to test, then to test each term separately. This chapter writes that contract, shows where each layer of the pipeline can break it, and introduces the labelled question set that the rest of the guide uses to measure every change.

Seven clauses

Think of the generator as a reader with a strict rule: it will treat everything in its context as the evidence. It cannot go and fetch the missing page, and it cannot tell that a passage is stale unless the passage says so. What it needs falls into seven clauses. Each one fails differently, and each one is checked by a different measurement.

Coverage is the clause everyone measures, so it gets the least explanation here. A question with two answering passages, for example a limit in the schedule and its exception in an endorsement, needs both. Recall that counts only the first is flattering the system.

Precision matters because the generator does not weigh passages by how they were ranked. Six near-duplicate sections from six policy versions leave it with a vote, and it will often pick the majority. Precision is a constraint on the answer, and a cost issue only second.

Shape is the clause most often missed, because no standard metric covers it. A table row reading "Wind/hail: 2.5%" is a different fact depending on whether "of Coverage A" and "deductible" travel with it. You find shape breaches by reading fifty gold chunks, not by computing anything. The next chapter is largely about keeping this clause.

Provenance is what makes the answer checkable. If the context carries document id, version, section path and page for each chunk, then the assistant can cite, an auditor can follow the citation, and a debugging session can find the source. Without it, citation checking has nothing to compare against.

Budget ties retrieval to the generator's window and to cost. The budget is not just a ceiling. The study by Liu and colleagues found that models do best when the relevant text sits at the start or end of a long input and worse in the middle, so a context that technically fits can still waste the evidence. Chapter 07 turns this into ordering rules.

Freshness and authorisation are contract clauses that no similarity score can express. Whether a wording applies depends on a date, and whether a passage may be shown depends on who is asking. Both are enforced as filters before ranking, and both are the subject of chapter 08. They are listed here because a system that scores well on the first five clauses can still breach these two, and the consequences, a wrong refund or a disclosure, are the expensive ones.

The chain of contracts

The pipeline is a sequence of stages, and each stage hands the next one something it can no longer repair. A parser that merges two columns cannot be corrected by a better embedding model. A chunker that cuts a table in half cannot be corrected by a reranker. That is why the contract has to be stated per stage: when the final context breaches a clause, you want to know which stage first broke it.

The useful property of this view is that a breach is almost always visible one or two stages later than it happened. A wrong deductible shows up at generation, but the cause may sit at the chunker, which split the table. Debugging means walking the chain backwards until the clause that was true at stage n is false at stage n+1. Chapter 09 turns that walk into a procedure, and the interview-prep version of it shows the same bisection in the form an interviewer asks for.

The contract also tells you what to log. For each request, record the query, the candidates with scores, the reranked list and the assembled context with token counts. Without those, "which clause broke" is a guess. With them, most investigations take minutes.

The labelled set

A contract is only useful if you can run it. For Harbourline that is a set of about 400 real questions sampled from the assistant's logs, each mapped to the passage or passages that answer it. The set is the instrument for the whole guide: every chunking change, model swap and reranker in the next chapters is judged by what it does to this set.

Each record carries the question, a slice label, the gold passages with their document version, and whether the question is answerable at all.

id: q-0142
slice: limits-deductibles
question: "What is my wind and hail deductible on an HO-3 policy?"
as_of: 2026-03-01              # the date the answer must be true for
gold:
  - doc: HO3-wording
    version: 2026-01
    section: "Section I - Deductibles"
    page: 14
    role: answer                # the table row
  - doc: HO3-wording
    version: 2026-01
    section: "Section I - Deductibles"
    page: 14
    role: context               # the column header for that row
answerable: true

Three decisions in that record matter. The as_of date and the version give the freshness clause something to fail against. The role field records that a row without its header is not a complete answer, so the shape clause becomes testable. The answerable flag lets you keep questions the documents cannot answer, which is how you later measure abstention in chapter 07.

Sampling is the part people rush. Take questions from logs, not from what the team thinks customers ask, and sample inside each slice so the rare, high-stakes types are not drowned by the common ones. Harbourline uses five slices: limits and deductibles, exclusions, procedures, definitions, and identifiers such as policy and form numbers. How to label, judge relevance, check agreement between labellers and keep the set current is the subject of chapter 06. For now the shape of the set is what matters.

Reading it per slice

A single recall number averages over question types that fail for different reasons. Suppose the first run of the 400 questions gives this, at k = 10. The figures are illustrative, and they are the baseline the following chapters improve on.

The overall figure of 0.79 is the question-weighted mean of the rows. It tells you nothing about where to work. The per-slice view says two things at once: the slice that carries the most money, limits and deductibles, is at 0.71, and the identifiers slice is at 0.52 for an entirely different reason. A fix aimed at one will not move the other. Chunking work in chapter 02 goes after the first. The second is a matter for lexical retrieval, because an exact form code is the kind of query an embedding model handles worst.

Slices also set a limit on what you can conclude. The identifiers slice has 60 questions. At a recall of 0.52 the standard error of that proportion is the square root of 0.52 × 0.48 / 60, about 0.065, so a 95% interval is roughly plus or minus 0.13. If a change moves that slice from 0.52 to 0.60, that is within the noise of a single run. Running both versions on the same 60 questions and comparing which questions flipped is more sensitive than comparing two means, but it does not remove the problem. Either grow the slice, or accept that you can only detect large gains there. Chapter 06 returns to how big the set must be for the differences you care about.

Why "the answer was wrong" is not a retrieval metric

It is tempting to grade the system by comparing final answers to a reference. That conflates at least four outcomes into one number. A wrong answer may follow from retrieval that missed, or from retrieval that was complete while the model ignored it or misread it. A right answer may follow from retrieval that worked, or from the model's own knowledge, which for general insurance terms is often good enough to be right by accident. Accuracy over those four cases cannot say which layer to change, and it drifts when you update the model even if retrieval is untouched.

The RAGAS paper by Es and colleagues makes the same separation in its framing: it scores whether the retriever finds relevant, focused context, whether the model stays faithful to that context, and the quality of the answer, as different things. The retrieval metrics in this guide stop at the edge of the context. They ask whether the gold passages are present, whether they are ranked where the budget will keep them, and whether they are whole, all before any model is called. That makes them cheap, deterministic and repeatable, so a sweep over nine chunking settings is nine index builds and nine scoring passes, with no generation spend and no judge variance.

The bisection is the bridge back to answers. Take a question the system got wrong, put the gold passage into the context by hand, and rerun. If the answer is now right, retrieval was at fault and the retrieval metrics should show a miss. If it is still wrong, the fault is in generation or in the prompt, and no retrieval change will help. A system that fails this test on the same question with the gold passage present is a different project, covered in chapter 07.

Treat answer-level evaluation as the final check, not the tuning loop. Run it on a sample before release, to confirm that better retrieval produced better answers, and keep the retrieval metrics as the thing you optimise.

When the frame bends

The contract assumes one query, one retrieval, one context. That fits the help-centre assistant. It fits less well where the model issues several searches in sequence, as an agent does when it reads a claim file, looks up the wording, then checks a procedure. There, coverage is a property of the whole session: no single call has to return everything, but the sequence has to reach every needed passage, and a bad first query can trap the loop. You then measure the fraction of tasks that reached all gold passages, in addition to per-call recall, and you add a clause for the cost of searching again.

Two other cases flip the emphasis. When the corpus is small enough to fit in the window, for example a single policy of fifty pages, passing the whole document may satisfy coverage, shape and provenance more cheaply than any pipeline, at the cost of budget and the position effect. And when a wrong answer is far more expensive than no answer, as with an exclusion read to a customer during a claim, precision and abstention outrank coverage, and you should accept a lower recall that comes with a clear "I could not find this" over a higher recall that sometimes brings in the wrong passage. The compliance guide's treatment of RAG covers that risk posture for regulated work.

✓ Check yourself

Answer from memory before opening each one.

1. Overall recall@10 on the 400-question set has sat at 0.79 for a month, yet complaints about quoted deductibles have risen. Name two ways both can be true.

   Show answer

   The mean can hide a slice moving against another: limits and deductibles may have fallen while identifiers rose, so the weighted figure stays flat. Or the set may not test the failing behaviour at all, for instance if it has no superseded wordings, so a stale-version breach never counts as a miss. See reading it per slice and the clauses.

2. The gold page is in the context and the model still quotes last year's deductible. Which clause was breached, and what do you look for in the logged context?

   Show answer

   Freshness, and probably precision. Look for two versions of the same wording in the assembled context, or a missing version field that would have let a filter or the model choose. Coverage passed, which is why recall alone looked healthy. See seven clauses.

3. A colleague proposes tuning the retriever by maximising agreement between final answers and reference answers. Which two different failures does that score merge, and why does it mislead after a model upgrade?

   Show answer

   Retrieval misses and generation faults both produce a wrong answer, and a right answer can come from the model's own knowledge with no useful retrieval. A model upgrade changes the generator's behaviour, so the score moves even when retrieval is unchanged. Score the context directly and bisect to attribute the rest. See why "wrong answer" is not a metric.

4. After a change, recall on the 60-question identifiers slice goes from 0.52 to 0.60. Do you ship on that evidence? What would you do next?

   Show answer

   Not on that alone. Each figure carries roughly plus or minus 0.13 at 95%, so the gain is within noise. Compare the two runs question by question to see which flipped and whether they share a cause, then add questions to the slice or accept that only larger gains are detectable. See reading it per slice.

5. A team wants to fix missing coverage by raising k from 5 to 20 with 600-token chunks. Argue for and against, and say what you would measure to decide.

   Show answer

   For: recall rises monotonically with k, and a tolerant generator can ignore extras. Against: the context grows from about 3,000 to 12,000 tokens, precision falls, near-duplicate versions enter, and evidence in the middle is used less reliably. Measure recall and precision at both k, the tokens per request, and answer faithfulness on a sample. A reranker that keeps a smaller k is often the better route. See budget and precision.

6. The claims-intake agent runs four searches per task. Why is per-call recall@k the wrong single number for it?

   Show answer

   No single call has to return every needed passage. What matters is whether the sequence reaches all of them and how many calls that takes, so measure task-level coverage and search count alongside per-call recall. See when the frame bends.

Takeaway

• Retrieval owes the generator seven things: coverage, precision, shape, provenance, budget, freshness and authorisation. A system can satisfy the first and still produce a regulatory problem.
• Each pipeline stage hands the next something it cannot repair, so a breach appears downstream of its cause and debugging walks the chain backwards.
• Shape has no standard metric. You check it by reading gold chunks, and the table-without-header case is the commonest breach.
• A labelled set of real questions, each mapped to gold passages with version and date, is the instrument for every later decision in this guide.
• Report recall per slice. The mean hides opposite movements, and a 60-question slice cannot confirm a gain of a few points.
• Final-answer accuracy mixes retrieval and generation faults. Tune on context metrics, and use the bisection and an answer-level sample to confirm.

The first breach in the chain is usually the earliest one, in the way documents are read and cut. → Parsing and Chunking

Further reading

• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023. Evidence that position in the context changes how well relevant text is used; the basis for the budget clause.
• RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023. Separates retrieval quality, faithfulness and answer quality; good for seeing why one score is not enough.