Hybrid Search and Reranking

The question, and what is being scored

The interviewer says: "Our vector search misses obvious things. Would you add hybrid search, a reranker, both, or neither?"

At Harbourline, the help-centre assistant sees queries like "what does form HL-204 cover", "flood excess UK", and "is a burst pipe covered if I was away". They differ in how much they depend on exact strings versus meaning. Three things are scored:

• Can you say why each retriever fails? "Hybrid is better" is a slogan. The mechanism is the answer.
• Do you know the cost of each step? A reranker adds latency and money, and hybrid adds a second index.
• Can you prove the gain? The answer ends with before-and-after numbers on the labelled set from chapter 01 and chapter 02.

Where each retriever misses

Dense retrieval embeds query and passage into vectors and ranks by similarity. It captures meaning: "my pipe burst" finds "escape of water". It is weak where the query depends on a specific string. An identifier such as HL-204 or a policy number is an opaque token to the model, and its embedding does not reliably separate it from HL-240. Rare terms, product codes, names and acronyms all behave this way. Negation is another gap: "not covered" and "covered" are close in meaning space.

Lexical retrieval, usually BM25, scores a passage by how many query terms it contains, weighted by how rare each term is in the corpus and adjusted for passage length. It matches exact strings perfectly, and it is blind to paraphrase: "my pipe burst" shares no word with "escape of water", so BM25 scores the right passage near zero. It also needs no embedding model and no GPU, and it explains its scores, which helps debugging.

The failures barely overlap, which is the whole case for combining them. A union of the two candidate lists covers both kinds of query. The open question is how to merge two ranked lists whose scores mean different things.

Fusing two lists

BM25 returns scores such as 14.2 and the vector index returns cosine similarities such as 0.83. Adding them is meaningless, since the scales and distributions differ and shift with the query. Two answers.

Weighted score fusion normalises each score list (min-max or z-score) and takes a weighted sum, alpha * dense + (1 - alpha) * lexical. It uses score magnitude, which is information, but the normalisation is fragile: one outlier score squashes the rest, and the best alpha varies by corpus and query type, so it needs tuning on labelled data.

Reciprocal rank fusion (RRF) ignores scores and uses only ranks. Each document's fused score is the sum over lists of 1 / (k + rank). Cormack, Clarke and Büttcher introduced it in 2009 and found it beat both Condorcet fusion and learned ranking methods on their test collections. Elasticsearch documents a default k of 60 and notes RRF needs no tuning, and that the combined signals need not be related.

def rrf(rankings, k=60):
    """rankings: list of ranked lists of doc ids, best first."""
    scores = {}
    for ranking in rankings:
        for rank, doc in enumerate(ranking, start=1):
            scores[doc] = scores.get(doc, 0.0) + 1.0 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

bm25  = ["form_hl204", "water_2026", "flood", "theft"]
dense = ["water_2026", "flood", "theft", "form_hl204"]
print(rrf([bm25, dense]))   # ['water_2026', 'form_hl204', 'flood', 'theft']

Run it: the form HL-204 document, first in BM25 and last in the dense list, still lands second overall, while the document both retrievers like ranks first. A document that only one retriever finds is not lost, which is the point.

RRF is my default because it works with no labelled data and no tuning. It flips when you have a labelled set and a corpus where one retriever is clearly stronger: a tuned weighted fusion can beat it, and it uses scores that RRF throws away. Most vector databases offer hybrid natively (Elasticsearch, OpenSearch, Weaviate, Pinecone, and Postgres with full-text search beside pgvector), so you do not build the plumbing. Retrieve a generous candidate set from each side, say 50 to 100, before fusing.

Reranking

Fusion improves who is in the candidate list. A reranker improves the order within it. The reason it works is architectural.

A bi-encoder (the embedding model) encodes the query and each passage separately, so passage vectors are precomputed and searched fast. But the query and passage never see each other inside the model. A cross-encoder feeds query and passage into the transformer together, so every query token can attend to every passage token. That is more accurate, and also far more expensive: nothing can be precomputed, and each query-passage pair needs a full forward pass. The Sentence-Transformers documentation puts it as a pipeline: a fast bi-encoder narrows the corpus to a shortlist (their example is the top 100), and a cross-encoder reorders only that shortlist.

ColBERT (Khattab and Zaharia, SIGIR 2020) is the middle path. It keeps query and document encoding separate so document vectors are computed offline, then scores with a light token-level interaction step. The authors report it competitive with BERT-based rerankers and two orders of magnitude faster at query time. The price is storage: one vector per token instead of one per passage.

Latency and cost, as arithmetic you can do in the room. Reranking 100 candidates of about 400 tokens each pushes roughly 40,000 tokens through the model per query. A small cross-encoder on a GPU handles that in the range of tens to a few hundred milliseconds, depending on model and hardware; I would measure it rather than quote one, and say so. An LLM reranker on the same 100 passages costs 40,000 input tokens per query; at 40,000 chats a day that is 1.6 billion extra input tokens a day, which is why LLM reranking is a tool for narrow cases and not a default.

One real number for scale: Anthropic's contextual-retrieval post retrieved the top 150 chunks, reranked, and passed the top 20 to the model, and reported that adding reranking took failed retrievals from 2.9% to 1.9% on top of contextual embeddings and BM25. Gains depend on the baseline; where retrieval is already good, there is less to gain. See also RAG Applied, failure mode 1 for the short form of this argument.

A decision rule you can say aloud

Start from the failures, in the order of cost to fix:

1. Look at where the gold passage ranks for failing questions (the test in chapter 01). Absent from the top 100 means the candidate stage is at fault; present at rank 6 to 50 means ordering is at fault.
2. If it is absent and the misses involve identifiers, codes or rare terms, add BM25 and fuse with RRF. If the misses are paraphrases, check the chunking and embedding model first.
3. If it is present but ranked below k, add a cross-encoder over the top 50 to 100 and keep the top 5 to 10.
4. If recall@5 is already high (say 95% or more on the labelled set) and failures are in generation, add neither. Each added stage costs latency and a thing to operate.

Spoken: "I diagnose where the gold passage ranks. If retrieval never returns it, I add lexical search and fuse with RRF. If it returns it too low, I add a reranker. If retrieval is already good, I leave it alone."

When it flips: a very small corpus, where an exact scan is cheap, may not need any of this. A voice channel with a tight latency budget (Harbourline's phone line) may forgo the reranker and spend effort on better fusion and chunking. A regulated answer on coverage, where a missed passage is costly, justifies the reranker's latency even when the average looks fine.

Prove the gain

"It feels better" does not survive an interview or a launch review. Use the labelled set from earlier chapters: questions with their gold passages, including hard slices such as identifier lookups and paraphrases. Measure each configuration on the same set:

• recall@k at k = 5 and k = 20: is the gold passage there at all?
• nDCG@10 (normalised discounted cumulative gain): does the ranking put relevant passages high? It rewards early positions, so it is the metric a reranker should move.
• Answer quality from a grader, to check the gain survives to the final answer.
• p50 and p95 latency and cost per query, because a gain that blows the SLO is not a gain.

Report the grid: dense only; BM25 only; RRF; RRF plus reranker. Break it out by slice. A typical finding is that fusion lifts the identifier slice a lot and the paraphrase slice not at all, and the reranker lifts nDCG across the board at a latency cost. That table is the answer to "both, one or neither". Compare the two configurations question by question on the same items, not only as averages; the evals chapter covers eval design more broadly.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A customer pastes "claim ref HC-88213-B" and the assistant returns a different customer-facing FAQ. Why did a strong embedding model fail here, and what would have caught it?

   Show answer

   An identifier is an opaque token with no meaning for the embedding model, so near-identical strings embed close together. An exact-term retriever such as BM25 matches it directly. See Where each retriever misses.

2. In RRF a document is ranked 1st by BM25 and absent from the vector list. Another is ranked 3rd in both. With k = 60, which scores higher, and what does that say about the method?

   Show answer

   The document ranked 3rd in both: 2/63 is about 0.0317 against 1/61 about 0.0164. RRF rewards agreement across retrievers over a single strong vote, but a one-list document still enters the fused list. See Fusing two lists.

3. Your reranker is added, and recall@100 of the candidate stage is 78%. After reranking, recall@5 cannot exceed what? Where do you spend effort?

   Show answer

   It cannot exceed 78%, since the reranker only reorders what it receives. The effort belongs in the candidate stage: fusion, chunking, query rewriting. See Reranking.

4. The voice line needs retrieval inside a few hundred milliseconds. Chat has a looser budget. Do you use the same retrieval stack for both?

   Show answer

   Not necessarily. Voice may skip the reranker or rerank fewer candidates, trading some ordering quality for latency, while chat keeps it. Decide from p95 latency and the nDCG difference measured per channel. See A decision rule.

5. Your before-and-after table shows hybrid search improving average recall@5 by 1 point. Is "hybrid does not help" a safe conclusion?

   Show answer

   No. The average hides slices: if identifier queries are 5% of traffic and improve by 20 points, that moves the average by 1 point and may be the queries that matter most. Report per slice and weigh by cost of error. See Prove the gain.

Takeaway

• Dense retrieval misses exact identifiers, rare terms and negations; BM25 misses paraphrase; their failures barely overlap, which is why combining them works.
• Reciprocal rank fusion merges the two lists using ranks only, needs no tuning and is the right default; weighted score fusion can win once you have labelled data.
• A cross-encoder reads query and passage together, so it orders a shortlist better than embeddings can, at the cost of one forward pass per candidate.
• A reranker cannot recover a passage retrieval never returned, so candidate-stage recall bounds the whole system.
• LLM reranking suits low-volume, reasoning-heavy relevance; at chat volume the token cost rules it out of the main path.
• Add each stage only after a gold-passage rank check shows the problem it solves, and prove the gain with recall@k, nDCG, answer quality and latency by slice.

Retrieval is now debuggable, chunked and ranked; the next design question is who decides what to retrieve and when, which is where agents and workflows differ. → Agent or Workflow

Further reading

• Reciprocal rank fusion — Elasticsearch docs. The formula, the default rank constant of 60 and why it needs no weight tuning.
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT — Khattab and Zaharia, SIGIR 2020. The late-interaction design and its speed claims.
• Retrieve and re-rank — Sentence-Transformers docs. Bi-encoder versus cross-encoder and why the pipeline has two stages.
• Cohere Rerank — vendor docs. What a hosted reranker takes, its context limits and model options as of November 2026.
• Introducing Contextual Retrieval — Anthropic, 2024. A measured case for BM25 plus embeddings plus reranking, with failure-rate numbers.