Section B · Drills

Drill: Embedding & RAG Data Infrastructure

The 2024-onward classic: design the data plane that turns a corpus into embeddings, keeps them fresh, and serves retrieval for a product feature — with the freshness, cost and evaluation questions that separate a demo from a platform.

The prompt

As the interviewer would say it:

Prompt

"We have a knowledge base of about two million documents — support articles, product manuals, internal runbooks — that changes daily. Product wants an assistant that answers customer questions grounded in that content, and later wants the same retrieval to power search and recommendations. Design the data infrastructure: how the documents become embeddings, how they stay current, how retrieval is served at scale, and how you'd know it's working. Assume you can call any embedding model."

Notice what the prompt is not: it isn't "pick a vector database". A candidate who opens with "I'd use Pinecone" has skipped four of the five moves. The interviewer wants to see the pipeline as a pipeline — sources, chunking, embedding as a versioned transformation, an index that is a derived artifact of a model version, serving with a latency budget, and an evaluation loop. The deep-dive treatment is in 07 · Embeddings & Vector Data; this chapter is how you say it in the room.

Clarifying questions

Spend two minutes here. Each question below moves a real design decision, and saying why you're asking is part of the score.

Corpus shape. "Two million documents — what's the size distribution? A 400-word article chunks very differently from a 90-page manual." Drives chunking strategy and index size.
Change rate and freshness. "What fraction changes per day, and how quickly must an edit be reflected — minutes, or is next-day fine?" Drives batch vs incremental embedding and whether you need CDC on the source.
Query load and latency. "Peak QPS on the assistant, and the retrieval budget inside the end-to-end response — say 50 ms of a 2 s budget?" Drives index type, replication, caching.
Access control. "Are any documents restricted — internal runbooks visible only to staff, per-tenant content?" Drives metadata filtering and whether you need per-tenant indexes. This is the one most candidates forget.
Modalities and languages. "Text only? Tables, images, PDFs with layout? Multiple languages?" Drives the parsing layer and model choice.
Model ownership. "Hosted embedding API or self-hosted? Any constraint on sending content to a third party?" Drives cost, throughput and the compliance answer.

Pin assumptions if they say "you decide": 2M docs averaging 1,500 tokens → ~6M chunks at ~500 tokens with overlap; 2% daily churn (~40k docs); 200 QPS peak; 50 ms retrieval budget; some content is tenant-restricted. Say the numbers back so everything after is anchored.

The five moves, applied

Run the method from 02 · The Method out loud. The order matters: requirements and signal first, then data, then the offline path, then the online path, then how you operate it.

1 · Requirements & signal

The signal you're optimising is retrieval quality for a downstream generator, not vector similarity. Name the functional requirements (grounded answers with citations; freshness within an hour for edited articles; per-tenant isolation; three consumers — assistant, search, recs) and the non-functional ones (p95 retrieval ≤ 50 ms; 200 QPS peak with 5× headroom; cost ceiling; audit trail of which version of which document answered a question). Then estimate: 6M chunks × 1,536-dim float32 ≈ 6M × 6 KB ≈ 37 GB of raw vectors; with int8 quantisation ~9 GB; an HNSW index adds ~1.5–2× overhead. That's one large node or a small cluster — not a "big data" problem, which changes the conversation about tooling.

2 · Sources & labels

Sources: a CMS with an API and change events, a document store of PDFs, a wiki. Each needs a parser (HTML → text, PDF with layout awareness, tables preserved as tables) and a stable document identity plus a content hash — the hash is what makes incremental re-embedding cheap. The "labels" in a RAG system are the evaluation set: a few hundred real questions with known good passages and answers, built from support tickets and curated by the people who own the content. Say early that you'd build this set in week one, because without it every later tuning decision is a guess. Deep dive: 03 · Labels & Feedback Loops.

3 · The offline path

Ingest → parse → chunk → embed → index, as a versioned pipeline:

Land raw documents in object storage with metadata (source, tenant, ACL, updated_at, content hash). This is the system of record for "what did the assistant know at time T".
Chunk deterministically — a chunker version is part of the lineage. Semantic or heading-aware chunking beats fixed windows for manuals; fixed windows with overlap are fine for articles. Store chunks as rows in a lakehouse table (Lance or Iceberg) keyed by (doc_id, chunk_idx, chunker_version).
Embed in batch with the embedding-model version in the key. Throughput math: 6M chunks on a hosted API at ~1,000 chunks/s aggregate is ~100 minutes for a full rebuild; a self-hosted model on one GPU does ~200–500 chunks/s. Incremental daily load is ~120k chunks — trivial either way. The full rebuild is the scenario you size for, because a model upgrade forces it.
Build the index as a derived artifact — the vector index is to the chunk table what a warehouse mart is to bronze data. It has a version, it can be rebuilt from the table, and you swap it atomically (blue/green) rather than mutating it in place during a model migration.

4 · The online path

Query → embed the query (same model version as the index, non-negotiable) → ANN search with metadata filters (tenant, ACL, doc type) → optional rerank with a cross-encoder on the top 50 → return top-k with citations. Latency budget: query embedding 10–15 ms, ANN 5–15 ms, rerank 20–30 ms — it fits in 50 ms only if the reranker is small or skipped for search. Filters are the design trap: post-filtering after ANN destroys recall for small tenants, so you either pre-filter (partitioned index per tenant when tenants are large and few) or use an engine that filters during traversal. Serving detail lives in 08 · Serving Architectures.

5 · Operate

Freshness: CDC or webhooks from the CMS feed a small streaming job that re-parses, re-chunks and re-embeds changed documents within minutes, and tombstones deleted ones — a stale-but-present chunk is worse than a missing one when the answer is a citation. Evaluation: retrieval recall@k and answer faithfulness on the golden set, run on every model/chunker change and nightly against a sampled live traffic replay. Monitoring: index size, embed lag, filter-hit ratios, share of queries with low top-1 similarity (the "we don't know" signal). Cost: embedding calls and reranking dominate the API bill; vectors are cheap to store.

The whiteboard

Draw the offline path left to right on the top half, the online path on the bottom, and the index in the middle where they meet. Label the versions — that's what makes it look like you've run one.

rag-data-plane.txt

OFFLINE (versioned pipeline)
 CMS ─┐   ┌─────────┐  ┌──────────┐  ┌───────────────┐  ┌──────────────────────┐
 PDFs ├──▶│ parse   │─▶│ chunk    │─▶│ embed         │─▶│ chunks table (Lance)  │
 wiki ┘   │ (v3)    │  │ (v2,     │  │ model=e5-v2   │  │ doc_id, chunk_idx,    │
 CDC/     └─────────┘  │  512±64) │  │ batch: 6M     │  │ text, vec, tenant,    │
 webhooks              └──────────┘  │ incr: 120k/d  │  │ acl, hash, versions   │
   │ deletes → tombstones             └───────────────┘  └──────────┬───────────┘
   │                                                                │ build (blue/green)
   ▼                                                                ▼
 raw docs (object store, system of record)              ┌───────────────────────┐
                                                        │ ANN index (HNSW)      │
ONLINE                                                  │ + metadata filters     │
 query ─▶ embed (same model) ─▶ pre-filter tenant/acl ─▶│ top-50                │─▶ rerank ─▶ top-5 + citations
            ~12 ms                                       └───────────────────────┘   ~25 ms
                                                                    ▲
 eval loop: golden set (300 Q/A) → recall@k, faithfulness ── nightly + on every version change

Say the seam out loud

"The index is a derived artifact of the chunk table and a model version. When we upgrade the embedding model, we rebuild into a second index, run the golden set against both, and cut over — we never query an index with a mismatched model." That sentence answers three follow-ups before they're asked.

Trade-offs table

Fork	Option A	Option B	Pick A when	Flip to B when
Embedding model	Hosted API	Self-hosted open model	Small team, corpus < ~10M chunks, no data-residency constraint	Content can't leave the boundary, rebuilds are frequent, or API cost passes a GPU's monthly cost
Chunking	Fixed window + overlap	Structure-aware (headings, sections)	Homogeneous short articles	Manuals, contracts, anything where a heading carries meaning
Vector store	Extension in an existing DB (pgvector)	Dedicated engine (Qdrant, Weaviate, Milvus, managed)	< ~5M vectors, modest QPS, team already runs Postgres	Tens of millions of vectors, filtered search at high QPS, multi-tenant isolation needs
Freshness	Nightly batch re-embed of changed docs	Streaming incremental via CDC/webhooks	Next-day staleness is acceptable	Edits must show within minutes (support content during an incident)
Filtering	Single index + filter during search	Index per tenant/segment	Many small tenants, engine supports filtered HNSW	Few large tenants or strict isolation/audit requirements
Reranking	None (ANN top-k straight through)	Cross-encoder rerank of top-50	Latency-critical search, good enough recall	Answer quality matters more than 20 ms; assistant use case

What the interviewer is listening for

Versioning discipline. Chunker version, model version, index version — and that a model change means a rebuild and a cut-over, not an in-place mutation.
Deletes and ACLs. Tombstoning removed content and enforcing tenant/ACL filters at retrieval. Leaking a restricted runbook into a customer answer is the failure that ends a project.
An evaluation loop that exists before tuning starts. A golden set, recall@k and faithfulness, run automatically.
Sizing that shows the problem is small. 37 GB of vectors is a node, not a cluster — a candidate who reaches for a distributed system reflexively has not done the arithmetic.
The reuse story. Search and recommendations reuse the chunk table and index, not a second pipeline.

Follow-up probes

Expect three or four of these. Each strong answer is a sentence or two, then an offer to go deeper.

"The embedding model vendor releases a better model. Walk me through the migration." — Full re-embed into a new index version (about 100 minutes for 6M chunks at 1k/s), evaluate both on the golden set plus a shadow read on live traffic, cut over behind a flag, keep the old index for a week for rollback. Query embedding and index version are pinned together in config so they can't drift.
"A customer asks a question and gets an answer from a document that was deleted yesterday." — Deletion path is broken or too slow: the source emits deletes as tombstones, the incremental job removes the chunks and the index entries, and there's an alert on "deleted docs still present in index" reconciled nightly against the source. I'd also add the document's updated_at to the citation so staleness is visible.
"How do you know retrieval is good enough — not the LLM, the retrieval?" — Separate metrics: recall@k and MRR on the golden set for retrieval; faithfulness and answer correctness for generation. Track them separately so a regression is attributable.
"Multi-tenant: one index or many?" — Depends on tenant size distribution. Filtered search in one index for many small tenants; partitioned indexes for a handful of large ones or where isolation must be provable. Never post-filter after ANN for small tenants — recall collapses.
"Costs are 3× projection. Where's the money?" — Almost always embedding and rerank API calls, not storage: cache query embeddings for repeated questions, batch offline work, quantise vectors, skip reranking for the search surface, and check for a re-embed loop caused by hash instability.
"Tables and images inside the manuals?" — Parse tables into a structured text form or embed them separately with a table-aware model; images get captions or a multimodal embedding, stored as their own chunk type with a modality field so retrieval can route.

The answer at four levels

How the same design reads at L1 → L4

L1 · Foundational. Chunk documents, embed them, put them in a vector DB, do similarity search. Names a tool; no versions, no freshness story, no ACLs.
L2 · Practitioner. Adds a real pipeline: parsing, overlap chunking, batch embedding, a nightly refresh, metadata filters, and top-k with a reranker. Knows recall@k exists.
L3 · Senior. Sizes the corpus and shows it's one node; treats the index as a derived, versioned artifact with blue/green cut-over; incremental freshness via CDC with tombstones; pre-filtering for tenants; a golden set built before tuning; separates retrieval and generation metrics.
L4 · Staff. Frames it as a platform serving three consumers from one chunk table; reasons about cost drivers and when to self-host; designs the migration and rollback path; anticipates the ACL leak and the stale-citation failure as the two incidents that would kill trust, and puts monitors on both; knows when a plain Postgres + pgvector is the right answer.

Next drill: the operating side of every system you've designed so far — 07 · Drill: Data Quality & Drift Monitoring.

✓ Check yourself

Answer from memory before opening each one.

The embedding vendor releases a better model, and an engineer proposes re-embedding the corpus in place into the existing index. What is wrong, and what do you do?

Show answer
Vectors from two models share no geometry, so a query embedded with one model would be compared against a mix of both and return meaningless neighbours. Build a second index version with the new model, run the golden set against both, cut over behind a flag, and keep the old index briefly for rollback. See the five moves and the whiteboard.
A corpus yields 10M chunks embedded at 768 dimensions in float32. How big are the raw vectors, how big with the index, and does it need a cluster?

Show answer
10M x 768 x 4 B is about 31 GB raw. An HNSW index adds roughly 1.5 to 2 times, so about 46 to 61 GB, and int8 quantisation brings the raw vectors to about 8 GB. That fits one large-memory node, so a cluster is not forced by size. See the five moves.
A small tenant owns 0.1% of the vectors. Retrieval runs ANN for the top 50 across everything and then filters by tenant. What does that tenant get?

Show answer
Almost nothing, because the top 50 across all tenants contains few or none of that tenant's chunks, and filtering afterwards leaves a handful at best. Recall collapses for small tenants. Pre-filter, use an engine that filters during graph traversal, or partition the index. See the online path in the five moves.
A document is deleted in the CMS. Which two stores must the deletion reach, and what check catches a miss?

Show answer
The chunk table and the ANN index, both as tombstones driven by the source's delete events. Run a nightly reconciliation of documents deleted at the source against chunks still present in the index. A stale chunk is worse than a missing one because it yields a confident citation of removed content. See the follow-up probes.
After a chunker change, answer quality drops. How do you tell whether retrieval or generation is at fault, and what had to exist before the change?

Show answer
Score retrieval with recall@k and MRR on a golden set of questions with known good passages, and score generation separately with faithfulness and answer correctness. The golden set had to exist before tuning began, or every change is a guess. See the five moves.
The corpus is 6M chunks with tenant filtering at 200 QPS. When is pgvector the right store, and what flips it?

Show answer
pgvector fits below roughly 5M vectors with modest QPS, especially when the team already runs Postgres and joins matter. At this scale and with filtering, its filtered recall can degrade, so a dedicated engine with filtered search wins. Keep the chunk table in the lakehouse either way so the index can always be rebuilt. See the trade-offs table.

Takeaway

The vector index is a derived artifact of a chunk table and a model version. A model upgrade means a rebuild and a cut-over, never an in-place change.
Document identity plus a content hash make incremental re-embedding cheap, and deletes must reach both the chunk table and the index as tombstones.
Size before choosing a tool: millions of chunks are tens of gigabytes, which is a node, not a cluster.
Embed the query with the same model as the index, and apply tenant and ACL filters before or during the search, never after it for small tenants.
Build the golden set before tuning, and measure retrieval with recall@k separately from generation with faithfulness.
pgvector is right for small corpora and modest load. A dedicated engine is right for filtered search at higher scale. The chunk table stays in the lakehouse.

Building the system is half the job. The last drill is how you would know it had gone wrong. → Drill: Data Quality & Drift Monitoring