Start Here
Most RAG systems are tuned by swapping parts until the demo looks better. This guide takes the pipeline apart layer by layer so you can predict what a change will do before you make it, and measure it afterwards on one labelled set. Start here for the frame, the set, and the running example.
Why a deep dive on RAG
Retrieval-augmented generation is the most common shape of production LLM system and the least understood by the teams running it. The pattern is simple enough to build in an afternoon, which is the problem: the afternoon version works on the ten questions you tried, and the failures arrive later, one user complaint at a time, with no obvious place to look. Teams respond by changing the embedding model, or the chunk size, or the prompt, and judge the change by whether the complaint stops. That is not engineering; it is superstition with a deploy pipeline.
Every layer of the pipeline has a mechanism: something it computes, over some input, with a cost, and a specific way it loses information. Once you know the mechanism you can say in advance that a larger chunk will help recall on procedures and hurt it on definitions, that a reranker will fix rank-eleven failures and do nothing for parsing failures, that a filter will exclude the right passage as readily as the wrong one. This guide explains each layer at that level, then gives you the measurement that shows whether your system has the problem.
It is written for people who already run a RAG system. If you need the interview-shaped version of this material, the AI Engineer Interview Prep guide covers the same ground in that format and links back here for depth.
The pipeline as a chain of contracts
The frame the whole guide uses: each layer owes the next something specific, and every failure is a breach of one of those obligations. Debugging is finding the breach; tuning is strengthening the layer that keeps breaching.
Two things follow from drawing it this way. First, the index being "complete" proves almost nothing about the answer: six obligations still have to hold after ingestion. Second, a wrong answer with the right document in the index is not one bug but a family of six, and they leave different evidence. Chapter 09 turns this into a procedure.
Take a failing question, find the passage that should have answered it, and run the generator again with that passage forced into the context. If the answer flips to correct, the breach is upstream of the prompt. If it stays wrong, the breach is in assembly or generation. Every chapter in this guide assumes you have done this before you start tuning anything.
The one labelled set
Nothing in this guide is judged by whether an answer "looks better". Every layer is measured against the same artefact: a labelled set of real questions, each mapped to the passage or passages that answer it, sliced by the kind of question it is. For Harbourline that set is about 400 questions sampled from the help-centre logs, in five slices:
| Slice | Example question | What usually breaks it |
|---|---|---|
| Limits and deductibles | "What is my deductible for water damage?" | Tables flattened at parse time; the wrong policy year outranking the right one |
| Exclusions | "Is damage from a sump-pump failure covered?" | Negation; paraphrase between customer language and policy language |
| Procedures | "How do I add a driver to my policy?" | Steps split across chunks; internal and customer-facing procedures confused |
| Definitions | "What counts as a named storm?" | Definition sections chunked away from the terms they define |
| Identifiers | "What does form HL-204 cover?" | Dense retrieval treating a form code as noise |
With this set you can compute recall@k (is the answering passage in the top k?), MRR and nDCG for retrieval, and faithfulness for generation, per slice, before and after any change. A change that lifts the aggregate while dropping the identifiers slice is a regression you would otherwise ship. Chapter 06 explains how to build the set and what each metric hides; chapters 02 through 05 each end by measuring their layer against it.
The running example: Harbourline Insurance
Harbourline is a fictional mid-size property-and-casualty insurer, about two million policies across the US and UK, shared with the other AI Engineering guides on this site. The system under study is its help-centre assistant: retrieval over roughly 12,000 documents, policy wordings of 20 to 300 pages in PDF and HTML with tables of limits and deductibles, several versions per product per year, plus internal procedures that agents use. Around 40,000 chat sessions a day, a third of which reach the assistant.
The cost of error is deliberately uneven. A wrong deductible quoted to a customer is a complaint and a refund. A wrong exclusion read to a customer during a claim is a regulatory issue. The same corpus feeds the claims-intake agent, so a retrieval fault can surface as a wrong coverage decision rather than a wrong chat answer. Those stakes are what justify the measurement discipline; a hobby project could skip most of it.
The chapters, in order
The order follows the pipeline. Each chapter opens on the decision or the failure, explains the mechanism, gives the numbers with their sources, shows the measurement, and ends with scenario questions to answer before you open the answers.
Section B · Building the index
| Chapter | What it settles |
|---|---|
| 01 The Retrieval Contract | What the generator needs from retrieval, stated so you can test for it |
| 02 Parsing and Chunking | Where documents lose their meaning before an embedding sees them, and how to chunk and measure |
| 03 Embeddings | What a bi-encoder can and cannot represent, and how to choose a model for your corpus |
Section C · Retrieving
| Chapter | What it settles |
|---|---|
| 04 Lexical, Dense and Hybrid Retrieval | Why BM25 and dense retrieval miss different queries, and how to fuse them |
| 05 Reranking | What a cross-encoder computes that a bi-encoder cannot, and when it earns its latency |
| 06 Measuring Retrieval | Building the labelled set, and recall@k, MRR and nDCG with what each hides |
Section D · Generating and operating
| Chapter | What it settles |
|---|---|
| 07 Context Assembly and Grounding | k, ordering, budgets, citations the model can honour, and abstention |
| 08 Freshness, Tenancy and Access Control | Keeping the index true to the source and enforcing permissions at retrieval time |
| 09 Debugging a RAG System | The trace, the bisection, the six failure classes and the regression dashboard |
| 10 Injection Through Retrieved Text | Retrieved documents as untrusted input: the attacks, the defences that hold, and how to test them |
How to read it
If you run a RAG system, read with it open. After chapter 06, build the labelled set for your own corpus; a hundred questions is enough to start. Every later chapter then gives you something to measure rather than something to believe. If you are reading for understanding only, the order still holds, because each layer's failures are explained in terms of what the previous layer promised.
The chapters assume you know what an embedding and a vector index are and have called a model API. They do not assume you have read any paper; the ones that matter are cited where they are used and listed under Further reading with a note on what each is good for.
Each chapter ends with scenario questions whose answers are folded away. They are written so that a reader who followed the mechanism can work them out and a reader who skimmed cannot. Use them as the test of whether to move on.
Takeaway
- A RAG pipeline is a chain of contracts: each layer owes the next something specific, and every failure is a breach you can locate.
- "The document is in the index" leaves six obligations still to be met before the model can use it.
- Forcing the gold passage into the context separates retrieval faults from generation faults in one model call; do it before tuning anything.
- Every change is measured against one labelled set, per slice, because aggregates hide the regressions that matter.
- Mechanism first: knowing what a layer computes lets you predict what a change will do instead of trying it and watching the complaints.
The chain begins with what the generator is owed. → The Retrieval Contract