Section C · Retrieving

Reranking

Chapter 04 widened the net so the answering passage is in the candidates. A reranker reads query and passage together to put it near the top. This chapter covers what that joint reading buys, the three families, what each costs, and how to tell on your own set whether the stage earns its latency.

When a reranker pays

After hybrid retrieval the answering passage is usually in the top 50 but not in the top 5. Harbourline's assistant passes only a handful of chunks to the model, so a deductible table at rank 14 is a miss even though coverage is satisfied. A reranker closes that gap: it takes the candidate list and reorders it with a more expensive judgement than the first stage could afford. The decision is whether the reorder is worth its latency, and the labelled set from chapter 01 answers it. Compare recall@50 with recall@5 on each slice. If they are close, the first stage already ranks well and a reranker only adds delay. If recall@50 is far above recall@5, the answers are present and badly ordered, and reranking is the right tool.

What a reranker sees

A bi-encoder embeds the query and the chunk separately, so the chunk vector cannot depend on the question. A reranker reads the two together. Every query token can attend to every chunk token, so it can notice that the query asks for the water backup limit and the chunk's table row labelled "Sewer or drain backup" holds a number in the next column. No single vector can encode that for every question that might be asked of the chunk.

FIRST STAGE (bi-encoder) RERANKER (cross-encoder) query ---> [encoder] ---> q vec [query ; chunk] ---> [encoder] ---> score chunk ---> [encoder] ---> c vec joint attention over the pair score = q . c one forward pass PER candidate chunk vectors precomputed nothing precomputable

The price follows from the diagram. A cross-encoder cost scales with the number of candidates, because each pair needs its own forward pass and nothing is cached. It cannot search the corpus, only reorder a short list. Nogueira and Cho's early BERT reranker improved MRR@10 on the MS MARCO leaderboard by 27% relative over the previous best, which is the evidence that the joint view is worth paying for.

Cross-encoders, late interaction, LLM rerankers

Cross-encoders are the default. A model trained on query and passage pairs outputs one relevance score. Hosted examples include Cohere's Rerank, whose documentation lists v4 and v5 models with a 32k context and older v3.5 models with 4k, where the query and the document share that limit and longer documents are split and scored in pieces (as of November 2026). Open-weight cross-encoders run on your own GPU. Both take a list of passages and return them with scores.

Late interaction sits between the two stages. ColBERT (Khattab and Zaharia, 2020) encodes the query and the document independently, like a bi-encoder, but keeps one vector per token. The score is MaxSim: for each query token, take its best match among the document's token vectors, then sum. Document token vectors can be precomputed and indexed, so you keep much of the cross-encoder's fine-grained matching while still searching quickly. The paper's abstract reports roughly two orders of magnitude lower latency and four orders of magnitude fewer FLOPs per query than BERT-based rerankers, with competitive effectiveness. The cost is storage: one vector per token, not one per chunk, so an index is many times larger than a dense one. Use it when the corpus is stable enough to pay for that index and first-stage ranking is the problem, and skip it if you cannot afford the memory.

LLM rerankers prompt a general model. A pointwise prompt scores each candidate on its own. A listwise prompt shows the model all candidates and asks for an ordering, which lets it compare them but is limited by context length and by sensitivity to the order shown. They need no training data and can follow instructions that a trained reranker cannot, such as "prefer the wording in force on the loss date". They are also the slowest and costliest option, with every candidate's tokens billed per query. They are the right choice for a low-volume, high-stakes path like claims intake, and the wrong default for 13k chat sessions a day (about a third of 40k).

Approach	Reads query and chunk together	Precompute	Cost driver
Cross-encoder	Yes	No	Candidates per query
Late interaction	Token level, after separate encoding	Yes	Index size
LLM reranker	Yes, plus instructions	No	Tokens per query

Where it sits and what it costs

hybrid retrieval ---> top 50-100 ---> reranker ---> top 5-8 ---> prompt (cheap, wide) (costly) (narrow) candidate count n -> reranker latency grows with n cutoff m -> tokens in the prompt grow with m

Two numbers define the stage. The candidate count is how many chunks you rerank. Too few and the answer is not among them, so check recall at that depth on the set. Too many and latency and cost climb for no gain. The cutoff is how many survive into the prompt, which chapter 07 treats as a token budget. Cost per query is the candidate count times the cost per pair, plus any splitting of long chunks. Cohere's documentation counts one search unit as a query ranking up to 100 documents, and chunks beyond 500 tokens, including the query, count as extra documents (as of November 2026). Check current prices on the vendor's page, since I could not confirm a figure. Latency also varies with hardware and model size. Measure p50 and p95 for your candidate count in your region, and do not extrapolate from a vendor's marketing page.

For Harbourline the arithmetic is simple to state. Say rerank latency at 50 candidates adds 150 ms to a request. For a third of 40k sessions that is a cost in the p95 users feel and a bill. The gain must be visible on a slice that matters, such as limits and deductibles, to justify it.

Rerank the chunk the model will read

If you retrieve small chunks and expand to their parent section for the prompt, score the version the model will actually see, or at least the version that carries the heading path. A reranker that sees the table row without its header cannot tell a limit from a deductible.

Measuring the gain

Run the reranker over the same labelled set and compare before and after on nDCG@10 and recall@5, per slice. The definitions are in the next chapter. The table below shows the shape of the result you need to produce. The values are illustrative, not measured on a real Harbourline system.

Slice	recall@5 before	recall@5 after	Verdict
Limits and deductibles	0.62	0.81	Worth the latency
Exclusions	0.70	0.78	Likely worth it
Identifiers	0.93	0.92	No gain: BM25 already ranks them

The pattern to expect is a large gain on slices where passages look alike and the question picks one (tables of limits), and none on slices the first stage already wins. If the gain is uniformly small, drop the reranker. If it is large on some slices only, you can apply it conditionally, for instance skipping it when the query contains a form number and BM25 has a clear top hit. Also check what the reranker can hurt: a model trained on general web pairs may demote a terse legal clause below a chatty help page, so read the cases where the gold passage moved down. For the interview framing, see Hybrid search and reranking.

✓ Check yourself

Answer from memory before opening each one.

Why can a cross-encoder rank "sewer backup limit" correctly against two near-identical table chunks when the embedding model could not, and why can you not simply use it as the first stage?

Show answer
It attends across the query and chunk jointly, so the question shapes how the chunk is read. That requires one forward pass per query-chunk pair and nothing precomputable, so cost grows with the corpus. See What a reranker sees.
On your set, recall@50 is 0.95 and recall@5 is 0.90 for the first stage. A colleague wants to add a reranker. What do you say?

Show answer
The ceiling for any reranker is 0.95 and the current gap is 0.05, so the most it can add is a few points of recall at a fixed latency and cost. Check per slice first, since a weak slice can hide behind a strong average. See When a reranker pays.
ColBERT scores far faster than a cross-encoder. What did it give up to get there, and what new cost did it create?

Show answer
It encodes the query and document separately, so there is no full cross-attention between them, only token-level MaxSim. It stores one vector per token, so the index is much larger than a one-vector-per-chunk index. See Cross-encoders, late interaction, LLM rerankers.
The claims-intake agent must prefer the policy wording in force on the loss date, which no trained reranker knows about. Which option fits, and what stops you using it for the chat assistant?

Show answer
An LLM reranker can follow that instruction. Its per-query token cost and latency make it hard to justify at chat volume. A better chat answer is to filter by effective date, as chapter 08 describes, and keep the model for the low-volume path.
Reranking raises nDCG@10 overall but the identifiers slice drops from 0.93 to 0.85. What happened and what do you change?

Show answer
The reranker likely prefers semantically fluent passages over exact-string matches that BM25 had ranked first. Skip reranking when lexical has a clear top hit, or blend the first-stage score into the final score. See Measuring the gain.

Takeaway

A reranker helps only when the answer is in the candidates but not near the top, which you read off recall@50 against recall@5 per slice.
Cross-encoders read the query and chunk together, which is why they rank better and why they cannot be the first stage.
Late interaction keeps token-level matching with precomputed document vectors, and pays in index size.
LLM rerankers follow instructions no trained model can, and cost the most per query, so reserve them for low-volume, high-stakes paths.
Cost scales with candidate count. Choose the smallest depth whose recall you have measured.
Report the gain per slice. A reranker can lift the average while hurting the identifiers slice.

Every claim here rests on a labelled set and a metric, so next is how to build one that can detect the differences you care about. → Measuring Retrieval