Embeddings, Vector Data & RAG Pipelines

An embedding is a feature with a model inside it

A tabular feature like acct_rentals_30d has a definition you can read: count rentals, trailing 30 days. An embedding's definition is the weights of the model that produced it. Two vectors are only comparable if the same model, at the same version, with the same preprocessing, produced both. Change the model and every stored vector is silently obsolete — not wrong in a way a test catches, just no longer in the same space as the new queries.

That is the whole architecture in one sentence. An embedding pipeline is a feature pipeline where the "feature version" includes a model version; a vector store is an online store whose index must be rebuilt when that version changes; and retrieval quality is a data-quality problem, monitored like drift. At GridDP the honest starting point is that most models don't need this at all — ranking, fraud, forecasting and pricing are tabular problems where gradient-boosted trees win. Embeddings earn their place for three jobs: semantic search over free-text workload descriptions ("fine-tune a 70B model" → suitable GPU configurations), similarity over behavior (hosts whose telemetry signature looks alike, for reliability and fraud rings), and retrieval for the AI interfaces that answer questions over docs, runbooks and metric definitions.

The embedding pipeline: batch and incremental

The pipeline has four stages, and the interesting decisions are in how it runs, not what it does: select the rows or documents that need vectors → preprocess (clean, chunk, normalize) → embed (a hosted model, an API, or a GPU batch job) → write vectors plus metadata to the store and the lakehouse.

• Batch (backfill and re-embed). Everything, once. This is a GPU job — a 7B-parameter embedding model over ten million support tickets is hours on a handful of GPUs, not minutes. Shard the input, checkpoint progress by shard, write to a new index version, and cut over atomically. At a neocloud this is the one ML data pipeline that competes with customers for the same hardware; schedule it on the marketplace's own idle capacity.
• Incremental (steady state). New and changed rows only, driven by CDC or an event stream (chapter 06's machinery). Latency is usually minutes; a new listing should be findable soon, not instantly. Idempotent upsert keyed by (entity_id, chunk_id).
• Deletes. The forgotten stage. A deleted listing, an erased account (GDPR), a retracted document must leave the index. CDC delete events → tombstone in the metadata table → delete from the vector store. Vector stores vary wildly in how cheaply they delete; check before choosing one.

# Incremental embedding of changed listings. Driven by the CDC topic; runs every 5 minutes.
MODEL_ID  = "bge-m3"                # embedding model
MODEL_VER = "2025-11"               # weights version — part of the vector schema
PREP_HASH = preprocess_hash()       # hash of chunking + cleaning code

changed = read_cdc("marketplace.listings", since=last_checkpoint())      # inserts, updates, deletes
docs    = [prepare(l) for l in changed if l.op != "delete"]              # title + description + specs
vecs    = embed_batch(docs, model=MODEL_ID, version=MODEL_VER)           # GPU or API, batched

rows = [dict(entity_id=d.listing_id, chunk_id=0, vector=v,
             model=MODEL_ID, model_version=MODEL_VER, prep_hash=PREP_HASH,
             gpu_model=d.gpu_model, region=d.region, price_hr=d.price_hr,  # filterable metadata
             embedded_at=now())
        for d, v in zip(docs, vecs)]

upsert("listings_v3", rows, key=("entity_id", "chunk_id"))               # vector store, index version v3
append_lakehouse("features.listing_embeddings", rows)                    # offline history for training/eval
tombstone([l.listing_id for l in changed if l.op == "delete"], "listings_v3")
commit_checkpoint()

Chunking & the document model

For RAG the unit of retrieval is a chunk, not a document, and chunking is where most retrieval quality is won or lost — before any vector store is involved. The decisions: chunk size (a few hundred tokens for Q&A over runbooks; larger for narrative docs), overlap (so a fact split across a boundary survives), structure-awareness (split on headings and tables, not on character counts), and what metadata rides along (source, section title, last-updated, access class). Each chunk is a row: (doc_id, chunk_id, text, vector, metadata). Store the text alongside the vector — a store that returns only IDs forces a second lookup in the latency path.

The document model also has to answer "what is the source of truth?" For GridDP's metric definitions the answer is the semantic layer, not a copied paragraph; for runbooks it is the wiki page at a version. Record it, because the AI interface will be asked "where did this come from?" and a chunk without provenance cannot answer.

Where vectors live

The choice is about scale, filters and how many systems you want to run. Nearly every option now offers HNSW or IVF-style approximate-nearest-neighbour indexes; the differences are in what surrounds the index.

Whatever serves queries, the lakehouse keeps the history. Vectors written only to a vector database are unreproducible the day the index is rebuilt with a new model; the append to features.listing_embeddings in the pipeline above is what lets you evaluate old versus new retrieval on the same corpus.

Hybrid search & filters

Pure vector similarity is a poor search engine. It has no notion of exact match — a query for "H100 SXM 80GB" should not rank an "H100 PCIe 80GB" listing as its nearest neighbour just because the words are similar — and it cannot enforce hard constraints like region or price ceiling. Production retrieval is hybrid: a lexical component (BM25) for exact terms, a vector component for semantics, fused by rank (reciprocal rank fusion is the workhorse) or by a learned reranker; plus metadata pre-filters applied inside the index so the top-k is computed over eligible rows only. The GridDP semantic search pre-filters on region and GPU memory, fuses BM25 over specs with the vector score over the free-text description, then reranks the top 50 with a cross-encoder. The vector store contributed one of four stages.

Refresh, re-embedding & drift

Three things change under an embedding index, and each needs a plan.

• The data changes. Handled by the incremental pipeline; measure the lag from source change to index visibility as a freshness SLA, like any feature.
• The model changes. A new embedding model is a migration, not an update: re-embed the full corpus into a new index version, evaluate retrieval quality on a held-out query set (recall@k against labeled relevant chunks, plus an LLM-judged answer quality if it feeds RAG), then cut queries over and retire the old index. Never embed new rows with the new model into the old index.
• The world changes. Query distribution drifts — a new GPU generation ships and nobody searched for "B200" during evaluation. Monitor the distribution of query embeddings and of top-k similarity scores over time; a falling median top-1 score means queries are landing in sparse regions of the index. Feed sampled queries back into the evaluation set monthly.

  eval set: 400 labeled queries → relevant chunk ids   (refreshed monthly from sampled production queries)

  index      model          recall@5   recall@20   p50 latency   notes
  ---------  -------------  ---------  ----------  ------------  -------------------------------
  listings_v2 bge-large 2025-03  0.71      0.88        18 ms       current
  listings_v3 bge-m3    2025-11  0.79      0.93        21 ms       candidate — cut over ✔
  listings_v3 + hybrid BM25       0.86      0.95        26 ms       ship with hybrid on

  drift monitor (weekly): median top-1 cosine 0.62 → 0.55 after B200 launch  ⚠ add queries to eval set

Worked example: GridDP

GridDP runs two embedding pipelines and, deliberately, no dedicated vector database. Listing embeddings (a few hundred thousand vectors, refreshed incrementally from the listings CDC topic) live in pgvector beside the listings table that marketplace search already queries — the pre-filters on region and GPU memory are ordinary SQL predicates, and hybrid search fuses pgvector similarity with Postgres full-text search. Internal-knowledge chunks for the AI interface (docs, runbooks, the semantic layer's metric definitions) live in LanceDB on object storage: low QPS, cheap, versioned per re-embedding, and readable directly by the evaluation notebooks. Both pipelines append to the lakehouse. The host-behavior embeddings for fraud-ring detection are computed weekly in batch and consumed as ordinary tabular features by the fraud model — they never see a vector index at all, because the consumer is a model, not a search box.

When to choose what

Next: the path from a feature value to a prediction, and the skew that opens up along it — 08 · Serving Architectures & Training/Serving Skew.

✓ Check yourself

Answer from memory before opening each one.

1. Your team swaps the embedding model for a better one and embeds only new listings with it, leaving the old vectors in the same index. Nothing errors. What is wrong?

   Show answer

   Vectors from two models live in different spaces, so distances between them mean nothing and queries return poor matches with high confidence. Treat the model, its version and the preprocessing hash as part of the vector schema, store them on every row, and re-embed the whole corpus into a new index version. See an embedding is a feature with a model inside it.

2. An account is erased under GDPR. Which parts of the embedding pipeline must act, and what is the usual gap?

   Show answer

   The change feed must carry the delete, a tombstone goes into the metadata table, and the vector store must remove the vectors. The usual gap is the deletes stage: stores differ widely in how cheaply they delete, and the lakehouse copy of the embeddings also needs the erasure. See the embedding pipeline.

3. A finance runbook and public docs share one vector index, and a filter hides finance chunks from unprivileged users after the top 10 are retrieved. Why is that not enough?

   Show answer

   Post-filtering lets the sensitive chunk leave the store, and it leaves the user with fewer results than asked for. Carry the access class as filterable metadata and apply it as a pre-filter on every query. See chunking and the document model.

4. A search for H100 SXM 80GB returns H100 PCIe 80GB listings as nearest neighbours. Why, and what retrieval design fixes it?

   Show answer

   Vector similarity captures meaning, not exact tokens, so near-identical names sit close together, and it cannot enforce hard constraints. Combine lexical match for exact terms with vector search, fuse by rank, and apply region or price constraints as filters. See hybrid search and filters.

5. You must replace the embedding model. Describe the migration and the gate that stops a bad cut-over.

   Show answer

   Re-embed the full corpus into a new index version, evaluate recall at k on a held-out labelled query set against the current index, then cut queries over and retire the old index. The gate is the evaluation, and the lakehouse history is what lets you compare both models on the same corpus. See refresh, re-embedding and drift.

6. The fraud team asks for a vector database to hold account embeddings that feed their gradient-boosted model. What do you say?

   Show answer

   Ask whether any consumer does similarity search. The model reads embeddings as input columns, so store them as ordinary features keyed by entity and skip the index. Choose a vector store only for a retrieval problem, and then by scale: pgvector under a few million vectors on Postgres, a dedicated store at tens of millions and high query rates. See where vectors live.

Takeaway

• An embedding is a feature whose definition is a model, so the model version and preprocessing hash belong in the vector's schema on every row.
• Changing the embedding model is a migration to a new index version with an evaluation gate, never an in-place update.
• Deletes are the forgotten stage: erasures and retractions must reach the index and the lakehouse copy.
• Chunking decides most of retrieval quality, and access control must live on the chunk as a pre-filter because an index flattens source permissions.
• Production retrieval is hybrid, combining lexical match, vector similarity and hard filters, with a labelled evaluation set to judge changes.
• Pick the store by scale and by what you already run, keep the vector history in the lakehouse, and skip the vector database when the consumer is a tabular model.

Features and vectors both end in a prediction request, and the path between them is where skew opens. → Serving Architectures & Training/Serving Skew

Further reading

• Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Malkov and Yashunin, 2016. The HNSW index behind most vector stores, including its recall and memory trade-offs.
• pgvector — pgvector project, accessed 2026. Vector type, HNSW and IVFFlat indexes, filtering and limits for Postgres.
• Lance — LanceDB, accessed 2026. The columnar format with random access and in-format versioning for vector datasets.
• Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods — Cormack, Clarke and Buettcher, SIGIR 2009. The rank-fusion method used to merge lexical and vector results.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., NeurIPS 2020. The paper that names RAG and sets out why retrieval quality bounds answer quality.