Context Assembly and Grounding

Choosing k and the token budget

Retrieval hands you a ranked list. Generation needs a context window you have decided how to spend. The decision between them is the one most teams leave at a default, usually "top 5" or "top 10", and it quietly sets both the cost and the ceiling on answer quality.

Think of the prompt as a budget with four lines: static instructions, the question and any chat history, the passages, and room for the answer. At Harbourline the instructions run about 900 tokens, history about 600, and the answer about 400. With a 6,000-token input target, which keeps latency and per-session cost predictable at roughly 13k assistant sessions a day, that leaves about 4,000 tokens for passages. The window size of the model is not the constraint. Cost, latency and the next section are.

Chunks are not equal in size, so fix the budget in tokens and let k fall out of it. A rule of "8 chunks" gives you 2,400 tokens one day and 6,000 the next, after a parser change alters chunk length (chapter 02). A rule of "fill 4,000 tokens in rank order, stop at the first chunk that does not fit" is stable.

The right budget comes from the labelled set (chapter 06). For each question you know the rank of the answering passage. Plot recall against cumulative tokens, not against k. The curve usually climbs steeply to a knee and then flattens. Past the knee, extra passages add cost, and they also add distractors: a limits table from the wrong product tier is a plausible-looking wrong answer. For Harbourline the limits and deductibles slice has its knee earlier than the procedures slice, because procedures often need two or three adjacent steps. That argues for a per-intent budget if you route questions by type, and for a single budget at the larger knee if you do not.

The case where this flips is questions that genuinely need breadth, such as "compare the flood terms across our three home products." Those are multi-document by nature, and a fixed 4,000-token budget starves them. Detect them (a comparison intent, or several product names in the query) and retrieve per entity with a budget each, rather than raising the global budget for everyone.

Ordering: where the model actually looks

Position inside the prompt changes what the model uses. Liu et al. (2023) placed the one relevant document at different positions among up to 20 distractors for multi-document question answering. Accuracy was highest when the answer sat at the start or end of the context and fell significantly when it sat in the middle, and the effect held for models built for long contexts. The authors tested models available in 2023, so treat the size of the dip as a property of those models. The shape of the result is the reason to measure your own.

The practical rule is to put the strongest evidence at the edges. The default of listing passages best-first already places the top passage at the start, which is the strongest position. The refinement is to put the second best last and push the weakest to the middle:

This costs nothing, but test it before adopting it, because it breaks one thing: the reading order of a procedure. If passages are consecutive steps of a claims process, shuffling them to suit attention gives the model steps out of sequence. The fix is to order after merging (next section). Merge adjacent chunks into one passage in document order, then place whole passages at the edges. The unit you place should be a passage that reads straight through.

Shorter contexts also reduce the problem. The reliable way to avoid the middle is to have fewer than ten passages in it, which returns to the budget. Measure by running the labelled set twice, best-first against edges-in, and compare answer correctness on the questions where the answering passage was ranked three or lower. If the delta is inside noise on your model, keep best-first and spend the effort elsewhere.

Deduplication, merging and provenance

The retrieved list is rarely clean. Overlapping chunks (chapter 02) return the same sentence twice. Several versions of a policy wording return near-identical paragraphs with one number changed. Adjacent chunks from one section arrive at ranks 2, 7 and 9. Assembly turns this list into a context in three steps.

1. Deduplicate. Drop a chunk whose text is contained in a higher-ranked chunk, using the chunk offsets you stored at ingestion rather than fuzzy matching. For near-duplicates across versions, do not drop by similarity at all. Resolve them by version, which is the subject of chapter 08.
2. Merge adjacent chunks. Group retrieved chunks by document and section, sort by offset, and join those that touch or sit within a small gap. One merged passage of 700 tokens reads better than three 250-token fragments, carries its heading once, and avoids the model stitching context together.
3. Attach provenance. Give every passage a short stable identifier plus the metadata the reader needs: document title, version or effective date, section heading and page.

from itertools import groupby

def assemble(chunks, budget_tokens, count_tokens):
    """chunks: ranked list of dicts with doc_id, start, end, text, meta."""
    chunks = sorted(chunks, key=lambda c: (c["doc_id"], c["start"]))
    passages = []
    for _, group in groupby(chunks, key=lambda c: c["doc_id"]):
        cur = None
        for c in group:
            if cur and c["start"] <= cur["end"] + 1:      # overlap or touching
                cur["text"] += c["text"][max(0, cur["end"] - c["start"] + 1):]
                cur["end"] = max(cur["end"], c["end"])
                cur["rank"] = min(cur["rank"], c["rank"])
            else:
                cur = {**c}
                passages.append(cur)
    passages.sort(key=lambda p: p["rank"])               # best first
    out, used = [], 0
    for i, p in enumerate(passages, 1):
        block = (f'<passage id="P{i}" doc="{p["meta"]["title"]}" '
                 f'effective="{p["meta"]["effective"]}" page="{p["meta"]["page"]}">\n'
                 f'{p["text"]}\n</passage>')
        n = count_tokens(block)
        if used + n > budget_tokens:
            break
        out.append(block); used += n
    return "\n".join(out)

The sketch assumes each chunk carries its character offsets and a rank. Putting the passage text, not the title alone, inside delimiters also matters for security: the model should be able to tell instructions from retrieved text, which chapter 10 returns to.

The effective date in the passage header is not decoration. Harbourline's worst grounding failure looked like a success: the assistant cited the right page of the right document and still quoted last year's deductible. The retriever had returned two versions of the same wording, the 2025 and 2026 editions, with the page numbers lined up. The model cited the page of the one it quoted, the citation verified, and a reviewer checking only "does the cited passage support the claim" would have passed it. Provenance has to carry the version, and assembly has to prefer the version that applied.

Citations that can be checked by a program

A citation is useful in proportion to how cheaply it can be verified. "According to your policy wording" verifies nothing. A claim followed by [P2] verifies in code: the identifier exists in the context you sent, so the answer cannot point at a document you never provided. Two stronger forms exist.

• Quote-and-locate. Ask the model to emit the exact supporting span with the identifier, then check by string match that the span occurs in that passage. This catches a citation that names a real passage but a claim that passage does not contain, at the cost of output tokens.
• Native citations. Anthropic's API can attach citations when you supply documents or search results as content blocks, and the documentation states that the cited text is extracted by the API and that the pointers are guaranteed valid. That removes a class of fabricated references but not the version problem above, and not a model that cites a real passage for a claim it only loosely supports.

Programmatic verification proves the pointer, not the reasoning. Measure the two separately. Citation validity is the share of citations whose identifier exists and whose quoted span matches, and it should be near 100%. Citation support is the share where the cited passage actually entails the sentence, which needs the faithfulness check below. A system can be perfect on the first and mediocre on the second.

Abstaining, and what it costs

When the passages do not answer the question, the model has two options: say so, or answer from its weights. The second is the dangerous one at Harbourline, because the weights hold plausible generic insurance knowledge and no knowledge of this policy. Instruct abstention explicitly: answer only from the passages, and when they do not contain the answer, say that the documents provided do not cover it and name what was searched.

Abstention has a price, so measure it as a pair. The abstention rate is the share of all questions where the assistant declines. The miss rate is the share of declines where the answering passage was in the context and the model declined anyway, plus the share of answered questions where the answer was not in the context at all. Build the second from the labelled set by splitting questions into those where the retriever surfaced the answer (check against the mapped passages) and those where it did not.

The bottom-left cell is what the instruction exists to empty, and the top-right cell is what an over-strict instruction fills. Tightening the prompt trades one for the other, so report both on every change. A rising abstention rate with a flat answer-present rate means retrieval got worse, not the prompt. The recommendation flips for low-stakes questions: for "how do I update my address" a hedged best-effort answer may serve customers better than a refusal, so set the strictness per intent, strict for limits, exclusions and anything quoted to a customer during a claim.

Faithfulness: is each claim supported?

Faithfulness asks whether every claim in the answer is supported by the provided context, independent of whether the answer is true in the world. The RAGAS paper (Es et al., 2023) defines it as claim-level: split the answer into atomic statements, check each against the context, and score the supported fraction. Two kinds of judge do the checking. A natural-language-inference model classifies each claim as entailed, contradicted or neutral against the passage. A model-graded check prompts a capable model with the passage and the claim. NLI models are cheap and deterministic but trained on short premises, so long merged passages and tables strain them. Model graders handle tables and context better but cost a call per claim and drift with the grader version.

Run a claim-level check offline on the labelled set for every change, and online on a sample of production traffic (for instance 2% of sessions) plus every answer that quotes a number. For the numbers there is a cheaper deterministic gate: extract every figure, date and percentage from the answer and require each to appear in the cited passage. That check would not have caught the 2025 deductible, because the number was in a real passage. It would catch a deductible the model invented, and it takes microseconds.

Calibrate any judge before trusting it. Hand-label 100 claim and passage pairs, run the judge, and look at the disagreement. If the judge and a human agree on only 80%, a faithfulness score moving from 0.92 to 0.94 is noise. A judge also has a bias worth testing: many are lenient toward fluent claims that paraphrase the passage loosely, which is the exact error you most want to catch on exclusions.

Caching the static part

Order the prompt so the unchanging material comes first: system instructions, abstention rules, citation format, and any few-shot examples. Retrieved passages and the question come last, because they differ on every request. Providers that cache prompt prefixes can then reuse the computation for the static head. On Anthropic's API you mark the end of the cacheable prefix with cache_control; the default lifetime is 5 minutes, refreshed on each use, with a 1-hour option at extra cost, and reads are billed at a fraction of normal input price (check the current pricing page, as of November 2026).

The mechanism is prefix matching, so a single changed token early in the prompt invalidates everything after it. Put no timestamps, user names or per-request identifiers in the system block. At 900 instruction tokens the saving is modest, and caching only pays when the prefix is above the provider's minimum cacheable length. It matters more when the static head grows, for instance a 3,000-token definitions glossary, and at Harbourline's volume the prefix will rarely go cold within five minutes. Passages themselves are rarely worth caching, except for popular documents in a conversation where the same passage returns turn after turn.

✓ Check yourself

Answer from memory before opening each one.

1. A parser upgrade makes chunks 40% longer. Your pipeline sends the top 8 chunks. What happens to cost, latency and answer quality, and how would you have designed the rule to avoid it?

   Show answer

   Prompt tokens rise by roughly 40% on the passage portion, with cost and latency following, and more distractors enter the context. A token budget filled in rank order is stable under chunk-size changes. See the budget section.

2. Recall on the labelled set is 91% at 4,000 context tokens and 92% at 8,000. A colleague proposes doubling the window "to be safe". What do you ask before agreeing?

   Show answer

   Ask what the extra tokens add besides one point of recall: cost, latency and distractors that sit in the middle where models use evidence least. Ask whether the misses are ranking failures that a reranker fixes more cheaply, and whether any question type, such as comparisons, has a different knee. See budget and ordering.

3. The assistant quotes a deductible with the right document title and page, and the citation check passes, yet a customer complains the figure is last year's. Which two assembly steps failed, and which metric would have stayed green?

   Show answer

   Version resolution before assembly failed, so two editions entered the context, and the provenance header either lacked the effective date or the model ignored it. Citation validity and faithfulness stay green, because the quoted number is in a real cited passage. The fix is to filter or prefer by effective date upstream. See assembly.

4. You tighten the abstention instruction and the abstention rate rises from 6% to 14%, while correct-answer rate on the labelled set stays flat. Is the change an improvement?

   Show answer

   Probably not. Split by whether the answering passage was in context. If the extra declines are all in the "passage absent" row, hallucinations became honest refusals, which is a gain. If they are in the "passage present" row, you built over-abstention and the flat correctness hides it behind fewer wrong answers. Judge it per intent. See abstention.

5. Your claim-level faithfulness score is 0.97 and a reviewer still finds wrong answers in the sample. Give two reasons the metric can be high while answers are wrong.

   Show answer

   Faithfulness measures support by the context, so a stale or wrong passage produces faithful wrong answers. The judge can also be lenient on loose paraphrases, so calibrate it against human labels. See faithfulness.

6. After adding the current date to the top of the system prompt, your prompt-cache hit rate collapses. Why, and where should the date go?

   Show answer

   Caching matches an exact prefix, so a changing token near the top invalidates the whole cached region. Move the date into the per-request section after the cache breakpoint, next to the question. See caching.

Takeaway

• Set the context budget in tokens and let k follow, choosing it at the knee of the recall-versus-tokens curve from your labelled set.
• Models use the start and end of a long context better than the middle, so test edges-in ordering on your own model, but merge adjacent chunks first so each placed unit reads straight through.
• Every passage needs an identifier and provenance including its version or effective date, and citations should be checked by code for validity.
• Citation validity proves a pointer exists; only a claim-level faithfulness check shows the passage supports the sentence.
• Measure abstention in the passage-present and passage-absent rows separately, and set strictness per intent.
• A faithful answer over the wrong passage is still wrong, so version resolution belongs before assembly.
• Put static instructions first and per-request text last so the prompt prefix stays cacheable.

The context is only as true as the index behind it. → Freshness, Tenancy and Access Control. For the diagnostic walk when an answer is wrong anyway, see Debugging a wrong RAG answer.

Further reading

• Lost in the Middle: How Language Models Use Long Contexts — Liu et al., 2023. The position experiments behind the ordering advice, and the protocol to repeat on your own model.
• RAGAS: Automated Evaluation of Retrieval Augmented Generation — Es et al., 2023. The claim-level definition of faithfulness and its sibling metrics.
• Anthropic citations documentation — how document and search-result blocks produce citations with valid pointers, and the constraints on combining them with structured output.
• Anthropic prompt caching documentation — breakpoints, the 5-minute and 1-hour lifetimes, and how to read cache usage in responses.