Section B · RAG and retrieval

Chunk Size and Overlap

How to choose chunking for a corpus you have never seen: what the embedding model, the questions and the generator each constrain, and how to measure the choice instead of guessing.

The question, and what is being scored

The interviewer says: "How would you choose chunk size and overlap for this corpus?" The corpus is Harbourline's help-centre material: policy wordings of 20 to 300 pages, claims procedures, and short FAQ pages.

A weak answer gives a number: "512 tokens with 10% overlap". A strong answer treats size as a decision with constraints and a measurement. Three things are being scored:

Do you know what constrains the choice? The embedding model, the shape of the questions, the generator's budget and the citation need each push in a different direction.
Do you know there are alternatives to fixed windows? Structure-aware splitting and small-to-big retrieval change the question.
Do you test it? The honest answer is "I do not know the best size for your data until I sweep it".

Chunking is also where the most common retrieval fault from chapter 01 is born: a fact separated from the context that gives it meaning.

Four constraints on chunk size

A chunk is the unit of retrieval, of embedding, of the prompt and of the citation. Those four roles want different sizes.

1. The embedding model. An embedding model turns a chunk into one vector, usually by pooling the token representations (mean pooling or a special token). Every token in the chunk contributes. A 1,500-token chunk that holds six unrelated clauses produces a vector that sits between them and matches none of them well. A 40-token chunk can be precise, but has little for the model to work with. Each model also has a maximum input length, from 512 tokens on older BERT-style models to roughly 8,000 on newer ones; text beyond the limit is truncated silently by many clients. Check the model card for the exact limit and whether it truncates.

2. The shape of the questions. Harbourline's questions come in two kinds. "What is my water damage deductible?" is answered by one table row. "What do I do after a car accident?" needs a procedure that spans several paragraphs. Chunks sized for the first kind fragment the second, and chunks sized for the second dilute the first. Look at real questions before choosing.

3. The generator's budget. If the prompt carries five chunks, the chunk size sets the context cost: five 800-token chunks is 4,000 tokens per question. At 40,000 chats a day that is 160 million input tokens a day from retrieval alone, before the system prompt and history. Larger chunks also lengthen the prompt in which the position effect from chapter 01 operates.

4. The citation need. If the assistant must cite "Home Plus wording, section 4.2", a chunk that spans three sections cannot cite precisely. Citation granularity argues for chunks that map to a section or smaller.

Retrieval wants small chunks; generation wants whole ones

Small units match questions precisely, and large units give the model enough surrounding text to answer correctly. A single size is always a compromise between those two jobs. Most of the good techniques below exist to decouple them.

Chunking strategies

Strategy	How it splits	Strength	Failure
Fixed window	N tokens, cut anywhere	Trivial, uniform cost	Cuts mid-sentence and mid-table; context lands in a neighbour
Recursive	Try paragraph breaks, then sentences, then words until under N	Respects natural breaks; the common default	Knows nothing of headings or tables
Structure-aware	Split on headings and sections; keep a table or list whole	Chunks are units a person would cite	Section sizes vary; huge sections need a second pass
Semantic	Embed sentences, cut where similarity between neighbours drops	Follows topic shifts without headings	Extra embedding cost; gains over recursive are inconsistent, so test them
Parent-child	Index small chunks; return the parent section to the model	Precise match, full context	Larger prompts; a parent can hold several matches

For Harbourline's policy wordings I would use structure-aware splitting with parent-child retrieval. Policy documents already have a hierarchy: part, section, clause. Split on it, keep each table intact whatever its size, and cut any section over a cap (say 800 tokens) at paragraph breaks. Embed child chunks of one clause or one table row group, and when a child matches, send its parent section to the generator. The deductible question hits the table row; the model reads the whole table with its column headers.

The parent-child pattern pays for itself because the two jobs separate. The index is optimised for matching, and the prompt is optimised for understanding. Its cost is the larger prompt: if a parent is 1,200 tokens and you send five, that is 6,000 tokens, so dedupe parents when two children come from the same one.

When this flips: for short FAQ pages of under 300 tokens, do not split at all; the page is the chunk. For free-form content with no headings, such as transcribed call notes, recursive splitting at 300 to 500 tokens is a reasonable start. For a corpus that is small enough to fit the context window outright, skip retrieval on that corpus. Anthropic's 2024 contextual-retrieval post suggests this for knowledge bases under about 200,000 tokens.

Overlap: what it buys and when it only duplicates

Overlap repeats the last part of one chunk at the start of the next. It exists for one reason: a fixed-window cut can fall in the middle of an idea, and without overlap neither side holds the whole of it. With 100 tokens of overlap, a sentence cut at the boundary appears whole in at least one chunk.

That is a patch for blind cutting. If you split on sentence or section boundaries, nothing is cut mid-idea, and overlap adds only cost: more vectors, more storage, and two near-identical chunks competing for the same top-k slots, which pushes out distinct passages. The effect is worst with top-k of 5, where one duplicated pair uses 40% of the slots.

Fixed window or recursive splitting: some overlap helps; 10 to 20% of the chunk is a common starting range, to be tuned.
Structure-aware splitting: start at zero overlap. Add it only if the sweep shows gains.
Sliding windows over long unstructured text (a 40-page transcript) are where overlap earns its place.

Overlap also affects deduplication downstream. If you keep it, merge adjacent retrieved chunks before building the prompt so the repeated text appears once.

Put the context in the chunk

A clause on its own often cannot be understood: "The deductible is $1,000" does not say which policy, peril or year. Two fixes, from cheap to expensive.

Prepend metadata. Add the document title, version and section path to the text that gets embedded: Home Plus wording 2026 > Section 4 Water damage > Deductibles. It costs a few tokens per chunk and no model calls. Store the same fields as filterable metadata too, which is the cure for the wrong-year bug from chapter 01.

Generate context per chunk. Anthropic's contextual retrieval has a model write a short description of where the chunk sits in its document and prepends it before embedding and before building the BM25 index. Anthropic reported failed top-20 retrievals falling from 5.7% to 3.7% with contextual embeddings, to 2.9% when combined with contextual BM25, and to 1.9% with reranking added (as measured by them, on their datasets; yours will differ). They put the one-off cost at about $1.02 per million document tokens, using Claude 3 Haiku with prompt caching, in 2024; re-check current model prices before quoting a budget. A 50-million-token Harbourline corpus would cost around $51 by that figure, which is small next to engineer time but not zero when re-indexing happens monthly.

Start with the free fix

Metadata prepending gets most of the way for structured documents like policy wordings, because the structure already exists. Pay for generated context when the documents are flat, such as emails or call notes, where no header says what a chunk is about.

Measure, do not guess

Any chunking choice can be tested in an afternoon if you have a labelled set. Build 50 to 200 question-to-passage pairs from real questions: a sample of help-centre search logs, with a person marking the passage that answers each. Include several question kinds: single-fact, multi-paragraph, table lookups, and questions with no answer in the corpus. Keep the set fixed so results compare.

Then sweep. For each configuration, re-chunk, re-embed, and compute recall@k: the share of questions where the gold passage's text appears inside a retrieved chunk. A passage-level label survives re-chunking because you match on text, not chunk ID.

sweep.py (sketch)

from itertools import product

def covers(chunk_text, gold_text):
    return gold_text in chunk_text            # gold answer span, not a chunk ID

def recall_at_k(questions, search, k):
    hits = 0
    for q in questions:                       # q.text, q.gold_span
        top = search(q.text, k)
        hits += any(covers(c.text, q.gold_span) for c in top)
    return hits / len(questions)

for size, overlap in product([200, 400, 800], [0, 50, 100]):
    index = build_index(corpus, chunk_tokens=size, overlap=overlap)   # your code
    print(size, overlap, round(recall_at_k(questions, index.search, k=5), 3))

Report more than recall. Add the average prompt tokens per question (the cost side) and an answer-quality score from the same labelled set using a grader, because a configuration can raise recall while hurting answers if the extra text distracts the model. Chapter 01's bisect shows how to separate the two. Also count the re-index cost: a chunking change means re-embedding the whole corpus, so a sweep of nine configurations costs nine embeddings of the corpus. Run it on a sample of documents first.

Do not assert one best size. Published guidance converges on "a few hundred to about a thousand tokens", and the right value depends on the embedding model, the documents and the questions. The sweep result for your data outranks any rule of thumb.

For the same topic in a shorter, compliance-flavoured form, see RAG Applied, failure mode 2.

A worked answer

Ask about the questions and the corpus. "Are answers usually a single fact, or a procedure? Are documents structured?" For Harbourline: both, and the policies have clear headings.
State the constraints. Embedding limit and pooling, question granularity, prompt budget, citation precision.
Propose a design, not a number. Structure-aware splits on part, section and clause, tables kept whole, zero overlap to start, child chunks of about 200 to 400 tokens indexed, parent sections returned to the generator, title and section path prepended.
Say how you would check it. A labelled set of 100 questions from help-centre logs, a sweep over size and overlap, and recall@5, prompt tokens and answer quality reported together.
Name where it flips. Short FAQ pages stay whole. Unstructured transcripts get recursive splitting with overlap. A corpus under about 200,000 tokens may not need retrieval.

The sentence that scores: "I would not pick a size before I had run the sweep, but I would not start from a blind fixed window either."

Interview probes

Drill mode — hide all answers

Q1: "Why not use the biggest chunks your embedding model allows?"

"Because the vector is one summary of everything in the chunk. A 6,000-token chunk covering six clauses embeds as a blend, and a specific question matches the blend less well than it would match the one clause. It also costs more in the prompt and gives the model more text to lose the answer in. The model's maximum length is a ceiling, not a target. The exception is when questions really are broad, such as 'summarise this policy', where larger units are right."

Q2: "Your tables keep coming back as garbage. What do you do?"

"I treat tables as their own element during parsing instead of letting text splitting touch them. Extract each table as structured data, render it as Markdown or one row per line with the column headers repeated, and keep it in one chunk, even when it exceeds the normal cap. For very large tables I split by row groups and repeat the header in each. Then I add a caption from the nearest heading to the chunk, so the deductibles table says what it is."

Q3: "You change the chunk size and recall goes up but answers get worse. Explain."

"Recall@k only says the gold text is present among k chunks. If I raised the size, each chunk is bigger, so the gold text is more likely inside one, but the prompt now has more tokens and more distractors. The generator can lose the fact in the middle, or two nearby clauses can confuse it. I would check prompt tokens, where the gold passage sits and a faithfulness score. The fix might be smaller children with parent expansion, or a reranker that keeps only the best chunks."

Q4: "Re-chunking needs a re-embed of everything. How do you make that safe?"

"I build the new index beside the old one, versioned, and never edit in place. I run the labelled set against both, compare, and switch traffic with an alias, so rollback is one pointer change. I also cache embeddings by a hash of chunk text and model, so chunks that did not change are not re-embedded. Cost is proportional to corpus tokens, so I estimate it first. For Harbourline, with a corpus in the tens of millions of tokens, one embedding pass is not the expensive part. The labelling and analysis time is."

Q5: "Is semantic chunking worth it?"

"It is a hypothesis to test, not a default. It adds embedding calls at ingestion and produces variable-size chunks, and in comparisons I have seen the gain over a good recursive or structure-aware split is small and inconsistent. When documents have headings, I use the headings, because they are free and meaningful. I would try semantic splitting for long, flat text like call transcripts, and keep it only if recall@k and answer quality improve on the labelled set."

✓ Check yourself

Answer from memory before opening each one.

A colleague tests a 3,000-token chunk size and finds exact-clause questions retrieve worse than at 400 tokens. What mechanism explains this?

Show answer
The embedding is one pooled vector over all tokens, so a chunk holding many clauses sits between them and matches a specific question less well than a single clause would. See the four constraints.
You split on section headings, so no chunk is cut mid-idea. A teammate adds 100 tokens of overlap anyway. What is the likely result at top-5?

Show answer
Adjacent chunks share text, so near-duplicates occupy several of the five slots and crowd out distinct passages, while the index grows. Overlap patches arbitrary cuts, which this split does not make. See Overlap.
Five retrieved children come from two parent sections of 1,200 tokens each. What do you send the generator, and why does that matter for cost?

Show answer
Send each parent once, 2,400 tokens, not five parents at 6,000. Deduplicating parents keeps the prompt small while the model still reads full sections. See Chunking strategies.
You have 40 labelled questions, and a sweep shows 400 tokens beating 800 by two points of recall@5. Do you re-index production? What would you want first?

Show answer
Not yet. With 40 questions two points is one question, which is within noise. Grow the set toward 100 to 200, add prompt tokens and answer quality to the comparison, and then decide. See Measure, do not guess.
Harbourline adds 8,000 short, flat claims-call notes to the corpus. Why might you now spend money on generated chunk context when you did not for policy wordings?

Show answer
Policy wordings carry their context in headings, which can be prepended for free. Flat notes have no headings, so a model-written description of what each note concerns is the only way to restore the context. See Put the context in the chunk.

Takeaway

Chunk size is set by four constraints at once: how the embedding model pools, how big the questions' answers are, the generator's token budget and the citation granularity needed.
Retrieval wants small units and generation wants whole ones, so parent-child retrieval usually beats picking one size.
Split on the structure the document already has, keep tables whole, and treat fixed windows as the fallback for text with no structure.
Overlap only patches arbitrary cuts; with structure-aware splits it mostly duplicates chunks and crowds the top-k.
Prepend title, version and section path to every chunk, and spend on model-generated context only where the document has no structure to supply it.
No size is best in general: sweep size and overlap on 100 or more labelled questions, and judge by recall@k, prompt tokens and answer quality together.

Even well-cut chunks leave a gap: dense search misses exact codes, and the right chunk may rank sixth. That is the job of the next chapter. → Hybrid Search and Reranking