Freshness, Tenancy and Access Control

The index is derived data

Every chunk, vector and posting list in the index is computed from a source document by a pipeline: parser, chunker, embedding model, metadata rules. The index is therefore a cache of that pipeline's output, and like any cache it is wrong the moment the source or the pipeline changes. The freshness question is how long it stays wrong, and for what.

Harbourline's corpus makes the stakes concrete. About 12k documents at roughly 100 chunks each is on the order of a million chunks (an estimate; count yours). A new edition of a home policy wording replaces a 200-page document. A claims procedure changes on a Tuesday. A document is withdrawn after a regulatory review. Each is a different kind of change, and the pipeline needs a different response to each: re-embed, delete, supersede. Two further events change the whole index at once: a new embedding model and a new chunking scheme. The rest of the chapter takes these in turn, then the question of who is allowed to see what.

Treat the source system as the truth and give each chunk a lineage: doc_id, doc_version, content_hash, the pipeline version that produced it (parser, chunker and embedding model identifiers) and the time it was indexed. Everything below depends on those fields existing at ingestion. Adding them later means re-ingesting everything.

Change detection and incremental re-embedding

Re-embedding the whole corpus on every sync is simple and wasteful, and embedding is usually the largest ingestion cost. Change detection decides which documents need work. Three signals, in order of cost:

1. Source metadata: a modified time, ETag or version number from the document store. Cheap, but wrong when an upstream system touches files without changing them, or changes them without bumping the time.
2. Document content hash: hash the raw bytes (or the parsed text) and compare to the stored hash. This catches the metadata lies and ignores no-op touches.
3. Chunk content hash: after parsing and chunking, hash each chunk. Only chunks whose hash is new need embedding; chunks with an unchanged hash keep their vectors.

The third level matters for long documents with local edits. Updating one clause in a 200-page wording should re-embed a handful of chunks, not 600. It only works if the chunker is stable, meaning an edit at page 40 does not shift every boundary after it. Fixed-size windows by token count fail this test, since one inserted sentence moves every later boundary and every hash changes. Structure-aware chunking that splits on headings and clauses (chapter 02) keeps boundaries local. Include the heading path in the hashed text, or a chunk whose section was renamed will look unchanged.

import hashlib

def chunk_hash(chunk, pipeline_version):
    text = chunk["heading_path"] + "\n" + chunk["text"]
    return hashlib.sha256((pipeline_version + text).encode()).hexdigest()

def sync(doc, new_chunks, store, embed, pipeline_version):
    old = {c["hash"]: c for c in store.chunks_for(doc["id"], doc["version"])}
    new = {chunk_hash(c, pipeline_version): c for c in new_chunks}

    to_embed = [h for h in new if h not in old]
    vectors = embed([new[h]["text"] for h in to_embed])
    store.upsert([{**new[h], "hash": h, "doc_id": doc["id"],
                   "doc_version": doc["version"], "vector": v}
                  for h, v in zip(to_embed, vectors)])

    store.tombstone([old[h]["id"] for h in old if h not in new])
    return len(to_embed), len(old) - len(new) + len(to_embed)  # embedded, removed

Put the pipeline version inside the hash, as above. When you change the chunker or the cleaning rules, every hash changes and the sync re-processes the corpus without a special mode. When you change only the embedding model, though, you do not want a quiet mixing of vector spaces, which the re-embedding section handles.

How fresh is fresh enough? A nightly batch is fine for policy wordings that change a few times a year. It is not fine for a procedure change that agents must follow today. Harbourline can run an event-driven path for the internal procedures (document store webhook, queue, sync) and a nightly reconciliation job for everything, since webhooks get lost. The reconciliation, which lists source and index and diffs them, is the part that guarantees convergence.

Deletes, tombstones and orphans

An upsert pipeline is naturally good at adding and bad at removing. Documents withdrawn at the source leave their chunks in the index, still retrievable, still quoted. These are orphaned chunks, and they are one of the most common long-run RAG defects because nothing errors when they appear.

Two mechanisms are needed. First, delete by key: every chunk carries its doc_id, and a withdrawn document deletes all chunks with that id, in the vector index and in any lexical index, cache or side table that holds copies. Second, delete by absence: the reconciliation job lists the ids present in the index and absent from the source and removes them. The second catches the events that were lost.

Many approximate-nearest-neighbour engines do not remove a deleted vector from the graph at once. They mark it and reclaim the space later, through compaction or vacuum. Check that your engine filters marked deletes at query time and that the lag between delete and effective removal is measured. For a withdrawn document the requirement may be minutes, and for a regulatory withdrawal it may be "before the next question". Also check the derived stores nobody remembers: a semantic response cache, a reranker feature store, a log of "popular passages". A deleted policy that survives in a response cache is still being served.

Versions and effective dates

Deleting is the wrong response to a new edition. A claim filed in March 2026 for a loss in November 2025 is governed by the wording in force in November 2025. Harbourline holds several versions of each product per year, and "which wording applied on the claim date" is a question with exactly one right answer. If the index contains only the latest edition, the assistant is confidently wrong for every claim that straddles a change. If it contains all editions with no discipline, it returns whichever scored best, which is how last year's deductible reached a customer (chapter 07).

The model for this is bitemporal in spirit. Each document version carries effective_from and effective_to (when the wording applied), and the index records when it learned of it. Retrieval takes an as_of date as a filter, not a hint: effective_from <= as_of AND (effective_to IS NULL OR as_of < effective_to). The date comes from the caller. The claims-intake agent has a loss date. The customer chat has "today" unless the question names a date or the session has a policy number whose term implies one. Never let the model guess the date.

Retrieval then returns one version per document for a given date. Where the question does not pin a date and the user is a customer asking about their current policy, filter on the policy they hold, which is an identifier lookup and not a similarity search.

The cost is index size and filter selectivity. Keeping every edition multiplies chunks by the number of live versions, which for a handful of editions is acceptable and for a document revised weekly is not. For fast-changing procedures that carry no legal weight in the past, keep only the current version. State the retention rule per document class, rather than one rule for the corpus.

Changing the embedding model

Vectors from different models, or even different versions of one model, live in different spaces. A query vector from model B compared with document vectors from model A returns numbers with no meaning, and nothing raises an error. So an embedding change is a migration of the entire index, not a setting (chapter 03 covers why to change). The standard path is a parallel index and a cutover.

Backfill from the source documents, not by copying text out of the old index, so that the new index also picks up parser and chunker changes deliberately. During dual-write, every incremental change goes to both indexes, so the new one does not fall behind while the backfill runs. Shadow queries run the labelled set (and a sample of live questions) against both and compare retrieval metrics per slice (chapter 06). A model that wins on average can lose on identifiers or on limits tables, which are the slices Harbourline cannot afford to lose. Cut over behind a flag, per tenant if you have tenants, and keep the old index until the rollback window passes.

The cost is mostly embedding compute for the whole corpus plus a second index in storage for the overlap period. Estimate it before you start: chunks times average tokens times the model's per-token price from its current price list. Where documents number in the millions this is a budget line, and at Harbourline's size it is small next to the engineering time. The path flips to an in-place re-embed only when the index is small enough to tolerate a maintenance window.

Multi-tenancy: separate or shared

If the assistant serves more than one tenant, whether business customers, brands or the US and UK arms of Harbourline, you need a boundary in the index. There are two shapes.

The last row is a mechanism worth knowing. With approximate indexes, a filter applied after the graph scan can return fewer results than asked for: pgvector's documentation notes that if a condition matches 10% of rows, an HNSW scan with the default hnsw.ef_search of 40 returns about four matching rows on average, and since version 0.8.0 iterative index scans keep scanning until enough rows match. Other engines filter during traversal or use partitions per tenant. A highly selective tenant filter on a shared index is exactly the case that exposes the difference, so test the filter at your smallest tenant's share of the corpus and check that recall at k holds.

For a few large tenants, with strict contractual isolation or per-tenant keys, use separate namespaces or indexes. For thousands of small tenants, a shared index with filters, or the engine's partition key feature if it has one, is the only economical shape. Harbourline's US and UK corpora differ in regulation and wording, share a few procedures, and number two, so separate namespaces with a small shared one are the clean fit.

Access control belongs in the retrieval filter

Within one tenant, documents still have audiences. Adjuster guidance is not for customers. A claims procedure marked internal must never reach the public chat. The rule is simple and often broken: the access decision is made by the retrieval layer, as a filter on the query, before any text reaches the model. It is never made by telling the model to withhold what it was given.

The reason is mechanical. A model that has seen a passage can be made to repeat it. Instructions like "do not reveal internal procedures" are a request, not a control, and a paraphrase, a translation request or an injected instruction in another retrieved passage (chapter 10) defeats them. The control that works is absence: text the caller may not read is never in the context.

The implementation has four parts.

• Stamp every chunk at ingestion with its access attributes (audience, tenant, group ids or an ACL reference), inherited from the source document.
• Derive the caller's attributes from the authenticated session on the server, never from the request body or the model's output.
• Apply the filter inside the search call (and in the lexical leg and any reranking candidate fetch), so filtered-out chunks cannot occupy top-k slots.
• Handle permission changes. If ACLs are copied into chunk metadata, revoking access means updating metadata, which is another change the sync job must detect. Where permissions change often, store a group id on the chunk and resolve group membership at query time.

Test it as a security property rather than a quality metric. Build a set of probe questions for each audience pair whose answer exists only in documents the caller must not see, run them as the lower-privilege identity, and assert that no chunk from a forbidden document appears in the retrieved set, the assembled prompt or the cited sources. Run this in CI against a fixture index, and on a schedule against production. Add the adversarial cases: a caller whose group was just revoked, a query phrased to ask for the internal document by name, and a request that differs only in tenant header. The compliance guide treats the wider picture of data handling in RAG applied.

Operational signals

Freshness fails silently, so give it instruments. Four are enough to start.

• Index lag: time from a source change to the new chunks being retrievable, as a distribution (median and 99th percentile), per document class. Alert on the percentile against the target you set for that class.
• Orphaned chunks: count of chunks whose doc_id or version no longer exists at the source, from the reconciliation diff. The target is zero, and any nonzero count is a ticket.
• Pipeline version spread: chunks per parser, chunker and embedding version. More than one embedding version in a live index means mixed vector spaces.
• Stale-version serves: answers whose cited chunk has a newer effective version than the one applied, found by joining the citation log to the version table. This is the metric that would have caught the wrong deductible.

When an answer is wrong and the passage looks right, check these before touching the model; the order for doing so is in chapter 09, and the interview-style version is in Debugging a wrong RAG answer.

✓ Check yourself

Answer from memory before opening each one.

1. A 200-page wording gets one clause amended. Your fixed 512-token windows re-embed 600 chunks instead of two. What is wrong, and what would you change?

   Show answer

   Inserted or removed text shifts every later window boundary, so every chunk hash changes. Chunking on headings and clauses keeps boundaries local so only the edited chunks differ. See change detection.

2. Legal withdraws a wording on Monday. By Wednesday it is still being quoted. List three places it could still live even though you deleted it from the vector index.

   Show answer

   The lexical index, a semantic response cache, and any side stores (rerank features, popular-passage logs); also marked-but-uncompacted deletes in the vector engine. Delete by key across all stores, and let reconciliation catch the misses. See deletes.

3. A claim with a loss date in November 2025 is filed in March 2026. The index holds both editions and the assistant quotes the 2026 deductible. Why does ranking not solve this, and what does?

   Show answer

   Editions differ in a few numbers, so their embeddings and lexical scores are nearly equal and either can rank first. Filter on effective dates using the loss date supplied by the caller, before ranking. See versions.

4. You swap the embedding model by re-embedding documents in place, a batch per night, while queries use the new model from day one. What do users see in the first week?

   Show answer

   Queries compare new-space vectors with a mix of old and new document vectors. Not-yet-migrated documents score near-randomly and drop out of results, with no error raised, so recall falls unevenly. Build a parallel index, shadow-test per slice, then cut over. See changing the embedding model.

5. A shared index with a tenant filter works in tests, but your smallest tenant, 0.5% of the corpus, gets thin or empty results in production. What mechanism explains it, and what are two ways out?

   Show answer

   If the filter applies after the approximate scan, most of the scanned candidates belong to other tenants and few survive. Use iterative or in-traversal filtering, or give small and large tenants separate partitions or namespaces. See tenancy.

6. A teammate proposes keeping internal documents in the shared index and adding "never disclose internal material to customers" to the system prompt. Argue for or against, and say what you would build instead.

   Show answer

   Against. A prompt is a request, and anything in the context can be extracted by paraphrase or injection. Stamp chunks with audience attributes, derive the caller's attributes server-side, filter inside the query, and test with probe questions run as the lower-privilege identity. See access control.

Takeaway

• The index is derived data: store lineage on every chunk (document, version, content hash, pipeline version) at ingestion, because adding it later means re-ingesting.
• Hash at the chunk level so edits re-embed only what changed, which needs a chunker whose boundaries stay local.
• Deletes need both delete-by-key and reconciliation by absence, across every store that holds a copy.
• Which wording applied is a metadata filter on effective dates supplied by the caller; similarity cannot decide it.
• A new embedding model means a parallel index, dual-write, per-slice shadow comparison and a flagged cutover.
• Access control is a filter inside the retrieval query, derived from the authenticated session; a prompt instruction is not a control.
• Track index lag, orphaned chunks, pipeline-version spread and stale-version serves, since freshness fails without errors.

With the index kept true and the boundary enforced, the next question is what to do when an answer is still wrong. → Debugging a RAG System

Further reading

• pgvector README — how filtering interacts with approximate indexes, and the iterative scan settings added in 0.8.0.
• Pinecone indexing overview — namespaces and metadata filtering as the vendor describes them; follow its multitenancy page for the per-tenant pattern.
• Efficient and robust approximate nearest neighbor search using HNSW graphs — Malkov and Yashunin, 2016. The graph structure that explains why deletes and filters behave as they do.
• Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Greshake et al., 2023. Why access control must not rely on the model withholding retrieved text.