Debugging a RAG System

The trace: what one request must leave behind

A support lead at Harbourline forwards a screenshot: the assistant told a customer her water-damage deductible was $500, and the wording says $1,000. Without a record of that request you can only rerun it today, against today's index, today's prompt and a sampling temperature that may not repeat the mistake. Debugging a RAG system is forensic work on a request that already happened, so the first decision is what every request writes down. The rule is that the log must let you replay each stage in isolation, not just read the answer.

Three fields are routinely missing and each one hides a failure class. Candidates with scores, because without them you cannot tell "the gold chunk was never retrieved" from "it was retrieved and the reranker buried it". The assembled context in prompt order with token counts, because truncation and position (see chapter 07) change what the model can use. And the document version, because Harbourline keeps several versions of each product wording per year and a perfectly retrieved chunk from the 2024 wording is a wrong answer for a 2026 policy. Store chunk ids and hashes rather than full text if volume or privacy is a concern, and keep the text in the immutable index snapshot so the ids resolve later. The retrieval contract that decides what a chunk id means is in chapter 01.

The bisection: retrieval or generation

Every wrong answer has one of two broad causes: the model did not receive the evidence, or it received it and failed to use it. One experiment separates them. Take the trace, replace the retrieved context with the gold passage (the one your labelled set maps to this question), keep the prompt and model identical, and ask again. If the answer is now right, the fault is upstream of generation. If it is still wrong, retrieval is not the problem and no amount of chunk tuning will fix it.

def bisect(trace, gold_chunks, ask, judge):
    """Replay a failed request with oracle context. `ask` and `judge` are yours."""
    oracle = ask(trace.query, context=gold_chunks, prompt=trace.prompt_version)
    if judge(oracle, trace.expected_answer):
        # generation is fine when given the evidence: look upstream
        gold_ids = {c.id for c in gold_chunks}
        retrieved = {c.id for c in trace.candidates}
        in_final = {c.id for c in trace.context}
        if not gold_ids & retrieved:
            return "not retrieved"      # extraction, chunking, embedding, or filter
        if not gold_ids & in_final:
            return "ranked below cutoff"
        return "retrieved but not used: check position and truncation"
    return "generation"                 # model ignored or overrode correct context

The second cut happens inside the retrieval branch. Walk the evidence outward from the answer: is the gold passage in the assembled context, in the reranked list, in the 100 candidates, in the index at all, in the parsed text at all? The first stage where it is absent is the stage that failed. This is the same bisection as the one in the interview-prep chapter, here with the measurements attached.

The six failure classes and the one test for each

Each class below leaves a different signature in the trace, and each has a test that takes minutes and returns a yes or no. Run them in this order, cheapest first, because an excluded filter or an unparsed table wastes less time than a retuned embedding model.

Two of these are easy to misread. Embedding mismatch and ranking below k both show the gold chunk "not in the final context", so people retune the reranker for a problem that sits in the first-stage retrieval. The distinguishing number is where the gold chunk sits in the 100 candidates: absent means first stage, present but low means ranking. The other is the generation class, where "the model hallucinated" is often really a position effect. Liu et al. found that accuracy on multi-document question answering falls when the relevant passage sits in the middle of a long context rather than at the start or end, so the one-line test of moving the gold chunk is worth running before blaming the model.

For Harbourline's $500 versus $1,000 case the traces usually land in one of three places. The wording for 2024 and 2026 both match the query and the older one outranks the newer (a filter or dedupe fault). The deductible sits in a table whose header row was parsed into a different chunk (extraction or chunking). Or the right table was retrieved and the model quoted a neighbouring row (generation). The bisection takes the three apart in one pass.

The regression dashboard

Individual bug reports tell you what is broken for people who complained. The dashboard tells you what changed for everyone else, and it runs on the same ~400 labelled questions used throughout this guide. Four panels, run nightly and on every change to the index, prompt or model.

• Recall@k per slice. The five question types (limits and deductibles, exclusions, procedures, definitions, identifiers) each get their own line, measured as in chapter 06. The aggregate hides a regression that touches one slice, and identifiers are exactly where a change to the embedding model or tokenizer hurts first.
• Faithfulness sample. A judge model scores whether each claim in the answer is supported by the retrieved context, as in the RAGAS metrics, on a rotating sample of live traffic. Treat the judge as a noisy instrument and spot-check a handful of its verdicts by hand every week.
• Abstention rate. The share of answers that decline. A rise means retrieval is thinning out or the grounding prompt got stricter. A fall can mean the model has started answering things it should refuse, which at Harbourline is the dangerous direction.
• Latency per stage. p50 and p95 for rewrite, retrieve, rerank and generate separately. A p95 jump in retrieve with flat recall usually means an index or filter-plan change.

Mind the sample size. With roughly 80 questions in a slice and recall near 0.90, the 95% confidence interval on a single measurement is about plus or minus 6.5 points (1.96 times the square root of 0.9 x 0.1 / 80). A drop from 0.90 to 0.86 in one night is inside the noise. Compare runs on the same questions as paired differences, alert on drops that persist for several runs or exceed what the paired difference supports, and keep a fixed holdout that you never tune against so improvements are not just overfitting to the dashboard.

Runbook: a user reported a wrong answer

Write this down once so the on-call engineer does not improvise. It assumes the trace exists.

1. Pull the trace by request id. Confirm the index snapshot and prompt version are still available. If not, say so in the ticket, because the rest is guesswork.
2. Establish the correct answer from the source document, with page and version, before you touch the system. Write the gold passage into the ticket.
3. Run the oracle-context replay. Wrong with the gold passage: go to step 6.
4. Right with the gold passage: find the first stage where the gold chunk is missing (context, reranked list, candidates, index, parsed text). Run the matching test from the table.
5. Apply the fix to the cause, not the symptom. Do not add a prompt line saying "the water-damage deductible is $1,000".
6. For a generation fault, test position, context size and grounding wording, one change at a time, on this question.
7. Add the question and its gold passage to the labelled set under the right slice. Rerun the dashboard and confirm the slice did not regress elsewhere.
8. If the wrong answer reached a customer who may have acted on it, tell the owner of complaints and refunds the same day. That decision is not an engineering one.

Step 7 is the one that compounds. Each reported failure that enters the labelled set becomes a permanent regression test, so the same bug cannot return unnoticed. A related failure type, where retrieval and answer are right but the attack came through a document, is the subject of the next chapter; the compliance guide covers the audit trail side in RAG applied.

✓ Check yourself

Answer from memory before opening each one.

1. A user reports a wrong exclusion. You replay the request with the gold passage placed in the prompt and the answer is still wrong. A colleague wants to try a larger chunk size. What do you tell them, and what do you test instead?

   Show answer

   Chunk size cannot matter: the model was handed the right evidence and still failed, so the fault is in generation. Test the position of the gold chunk, the amount of other context, and the grounding instruction one at a time. See the bisection.

2. Gold chunk is rank 37 among 100 candidates and absent from the 5 chunks in the prompt. Another question has the gold chunk absent from all 100. Do they need the same fix?

   Show answer

   No. The first is a ranking loss, addressed by reranking or a larger k. The second is a first-stage failure (embedding or query mismatch, filter, chunking or extraction), and a better reranker cannot find what was never a candidate. See the failure classes.

3. The aggregate recall@5 on your labelled set is flat at 0.88 after an embedding model upgrade, but support complains about policy-number lookups. What do you check, and why did the dashboard stay green?

   Show answer

   Check the identifiers slice. Dense models often lose exact-match behaviour on codes, and gains on other slices can offset it in the average. A per-slice view exposes this. See the dashboard.

4. A slice of 80 questions drops from 0.90 to 0.86 recall overnight. Should the on-call page the team?

   Show answer

   Not on that alone. The interval on one measurement is about plus or minus 6.5 points, so the drop is within noise. Look at the paired difference on the same questions, and page if it persists or specific questions that used to pass now fail. See the dashboard.

5. An engineer fixes a reported wrong deductible by adding the correct figure to the system prompt. It works. Why is that the wrong fix at Harbourline, and what should have happened?

   Show answer

   It patches one product's one limit, goes stale when the wording changes, and leaves the real cause (parsing, ranking, versioning) in place for every other limit. The question and gold passage should have gone into the labelled set after a root-cause fix. See the runbook.

Takeaway

• A request that leaves no replayable trace (candidates with scores, ordered context with token counts, index and prompt versions) cannot be debugged, only guessed at.
• Replacing the retrieved context with the gold passage splits wrong answers into retrieval faults and generation faults in one experiment.
• Within retrieval, the first stage at which the gold chunk goes missing names the failure class: filter, extraction, chunking, first-stage mismatch or ranking.
• Test the cheap, mechanical causes first, and rerun the bisection after each fix because a second fault is often hiding behind the first.
• Recall per slice on a fixed labelled set catches regressions that an average hides, but small slices carry wide error bars, so use paired comparisons.
• Every reported failure should end as a new labelled question, not as a prompt patch.

Retrieval is also an input channel an attacker can write to. → Injection Through Retrieved Text

Further reading

• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023. The evidence for position effects in multi-document question answering.
• Ragas: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023. Reference-free metrics for context relevance, faithfulness and answer quality, the basis for the faithfulness panel.