An embedding is derived data: it is computed from some source text by a specific model at a specific time, and it silently becomes wrong when either the text or the model changes. Interviewers probe this because the hard part of a RAG system is rarely the first insert; it is the second year, when documents are edited and deleted, the embedding model is upgraded, and a customer asks for their data to be erased. For a Backend / AI engineer the design question is what to store next to each vector so that staleness is detectable, cleanup is a query and a model migration is a controlled rollout instead of an outage.
documents chunks chunk_embeddings
------------------------- -------------------------------- -----------------------------------------
id PK id PK chunk_id FK -> chunks (CASCADE)
tenant_id who owns it document_id FK -> documents model 'embed-small'
source_uri where it came from chunk_index position in doc model_version 'v1'
doc_version bumps on edit content the exact text dims 1536
acl / labels who may see it content_hash hash of that text embedded_hash hash of text that was embedded
updated_at token_count for prompt budgeting embedding vector(1536)
deleted_at? (soft delete) metadata jsonb: page, section created_at
PK (chunk_id, model, model_version)
Vector row answers "which chunk is near this query?".
Everything else answers "may this user see it, where did it come from, is it still current?"The vector on its own is useless for a product: retrieval returns a chunk id, and you need the text to put in the prompt, the source to cite, and the tenant and access labels to make sure the caller is allowed to see it. Beyond that, three fields make the system maintainable. content_hash on the chunk and embedded_hash on the embedding let you find stale vectors with a plain SQL comparison. model, model_version and dims record which vector space each row lives in, which matters because vectors from different models cannot be compared and often do not even have the same length. doc_version and updated_at support incremental sync and let you drop retrieval results that point at an older version of a document. Keeping the text next to the vector, rather than only in an external object store, also means a re-embed job can run from the database alone.
-- Portable SQL run in sqlite3 3.51 (vector column replaced by a text stand-in; hashes shortened for readability)
PRAGMA foreign_keys = ON;
CREATE TABLE documents (id INTEGER PRIMARY KEY, tenant_id INTEGER NOT NULL, source_uri TEXT NOT NULL,
doc_version INTEGER NOT NULL, UNIQUE (tenant_id, source_uri));
CREATE TABLE chunks (id INTEGER PRIMARY KEY,
document_id INTEGER NOT NULL REFERENCES documents(id) ON DELETE CASCADE,
chunk_index INTEGER NOT NULL, content TEXT NOT NULL, content_hash TEXT NOT NULL,
UNIQUE (document_id, chunk_index));
CREATE TABLE chunk_embeddings (
chunk_id INTEGER NOT NULL REFERENCES chunks(id) ON DELETE CASCADE,
model TEXT NOT NULL, model_version TEXT NOT NULL, dims INTEGER NOT NULL,
embedded_hash TEXT NOT NULL, embedding TEXT NOT NULL,
PRIMARY KEY (chunk_id, model, model_version));
INSERT INTO documents VALUES (1, 42, 'wiki/refunds', 1), (2, 42, 'wiki/shipping', 1), (3, 7, 'wiki/hr', 1);
INSERT INTO chunks VALUES (10,1,0,'Refunds within 30 days','h_a'), (11,1,1,'Refunds need a receipt','h_b'),
(20,2,0,'Ships in 3 days','h_c'), (30,3,0,'Leave policy','h_d');
INSERT INTO chunk_embeddings VALUES
(10,'embed-small','v1',1536,'h_a','[..]'), (11,'embed-small','v1',1536,'h_b','[..]'),
(20,'embed-small','v1',1536,'h_c','[..]'), (30,'embed-small','v1',1536,'h_d','[..]');
-- 1) A source edit changes content_hash but the stored vector still describes the OLD text
UPDATE chunks SET content = 'Refunds need a receipt or order id', content_hash = 'h_b2' WHERE id = 11;
SELECT c.id AS chunk_id, c.content_hash AS now, e.embedded_hash AS embedded
FROM chunks c JOIN chunk_embeddings e ON e.chunk_id = c.id
WHERE e.model = 'embed-small' AND e.model_version = 'v1' AND e.embedded_hash <> c.content_hash;
-- 2) Which chunks still lack an embedding from the model we are migrating to?
SELECT c.id AS chunk_id FROM chunks c
LEFT JOIN chunk_embeddings e ON e.chunk_id = c.id AND e.model = 'embed-large' AND e.model_version = 'v2'
WHERE e.chunk_id IS NULL ORDER BY c.id;
-- 3) Backfill two chunks with the new model; the old rows stay so a rollback is possible
INSERT INTO chunk_embeddings VALUES (10,'embed-large','v2',3072,'h_a','[..]'), (20,'embed-large','v2',3072,'h_c','[..]');
SELECT model, model_version, dims, COUNT(*) AS n FROM chunk_embeddings GROUP BY model, model_version, dims ORDER BY model;
-- 4) Deleting a document removes its chunks and every embedding of every model, via ON DELETE CASCADE
DELETE FROM documents WHERE id = 2;
SELECT (SELECT COUNT(*) FROM chunks) AS chunks_left, (SELECT COUNT(*) FROM chunk_embeddings) AS embeddings_left;
-- Real output (sqlite3 3.51, run locally):
-- 1) chunk_id now embedded -> 11 h_b2 h_b
-- 2) chunk_id -> 10, 11, 20, 30 (none has a v2 embedding yet)
-- 3) model model_version dims n
-- embed-large v2 3072 2
-- embed-small v1 1536 4
-- 4) chunks_left embeddings_left -> 3 4Step 1 shows the stale-vector problem in one row: the text changed, so the old vector no longer describes it, and nothing in the database noticed until the hash comparison. Step 2 is the migration to-do list expressed as an anti-join. Step 3 shows why the key includes model and model_version: old and new embeddings coexist during a migration, and the row count of each tells you how far the backfill has got. Step 4 shows deletion as a cascade: 4 embeddings remain because document 2's chunk lost both its v1 and v2 vectors. That cascade is the difference between "delete a document" being one statement and a leaky cleanup job. In PostgreSQL the same schema works with vector(1536); see the next section for the dimension constraint.
Why you cannot just "swap the model":
model A (1536 dims) model B (3072 dims) a query embedded with B
stored vectors [....] stored vectors [........] cannot be compared with A's vectors:
index built on A different space + length different dimension, and even at equal
dimension the axes mean different things
Blue/green re-embedding:
1. add the new embeddings (new table or new column + its own index), model = B
2. backfill in batches, rate-limited; live writes embed with BOTH models
3. evaluate: run the same test queries on A and B, compare retrieval quality
4. flip the read path (feature flag / config: which model_version to query with)
5. keep A for a rollback window, then drop it and its index-- PostgreSQL + pgvector (not run here)
-- A vector column with a declared dimension can be indexed; mixing dimensions in one indexed column is not possible.
-- Simplest robust layout: one embeddings table per model version.
CREATE TABLE chunk_emb_v1 (chunk_id bigint PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
embedded_hash text NOT NULL, embedding vector(1536) NOT NULL);
CREATE TABLE chunk_emb_v2 (chunk_id bigint PRIMARY KEY REFERENCES chunks(id) ON DELETE CASCADE,
embedded_hash text NOT NULL, embedding vector(3072) NOT NULL);
-- Build the ANN index on v2 after (or while) backfilling; queries pick the table from config:
CREATE INDEX ON chunk_emb_v2 USING hnsw (embedding vector_cosine_ops);
-- Lower precision or dimension to cut storage (check the pgvector version and your model's documentation):
-- halfvec(n) stores 2 bytes per dimension instead of 4
-- some models support returning shorter vectors on request; that changes the space, so it is a new model_versionA vector index needs a fixed dimension, and an embedding model defines both the dimension and the meaning of every coordinate. So a model upgrade is not an in-place update; it creates a second, incompatible copy of your vector data that you must build, validate and switch to. That is why the model and its version belong in the data model. The blue/green pattern keeps search available: readers keep using version A while a throttled job fills version B, new and edited documents are embedded with both, and only when retrieval on B is validated do you flip the config. The costs are honest ones: temporary double storage and double index memory, embedding API cost or GPU time for the whole corpus, and a rollback window you must pay for.
Document v1 chunks Document v2 chunks
0 "Refunds within 30 days." 0 "Refunds within 30 days." -> same hash: KEEP vector
1 "Refunds need a receipt." 1 "Refunds need a receipt or order id." -> new hash: RE-EMBED
2 "Digital goods are excluded." 2 "Digital goods are excluded." -> same hash: KEEP vector
3 "Contact support for exceptions." -> new chunk: EMBED
(a chunk that exists in v1 but not in v2 -> DELETE its vectors)
Do it in ONE transaction: update chunk rows + delete removed + insert new placeholders,
then embed asynchronously (a queue) and write embedded_hash when the vector is stored.# plain Python, run locally with python3: compare chunk hashes between two versions of a document
# Real output (python3 3.14):
# old: {0: '7294ffb1', 1: '952ebfd0', 2: 'f04c8302'}
# new: {0: '7294ffb1', 1: 'c68a6f9c', 2: 'f04c8302', 3: 'e54d07ea'}
# keep embeddings for chunks: [0, 2]
# re-embed chunks: [1, 3]
# delete embeddings for chunks: []
# embedding calls saved: 2 of 4
# insert one chunk at top -> index-based diff re-embeds: [0, 1, 2, 3] of 4
# hash-based matching would keep: [1, 2, 3]Re-embedding a whole document on every save is correct but wasteful; hashing each chunk lets you re-embed only what changed, which saves embedding cost and index churn. The last two lines show the catch: if you match chunks by position, inserting one paragraph at the top makes every chunk look changed, while matching by content hash keeps the unchanged ones. Deletion has a second dimension beyond cascades: for privacy or compliance you must delete the vectors too, since embeddings are derived from the personal or confidential text and can leak information about it. In PostgreSQL a deleted row's index entries are cleaned up by VACUUM, and heavy churn on an HNSW index may call for a periodic reindex (REINDEX CONCURRENTLY). Backups and replicas still contain the old data until they expire, which is something you must state in a compliance answer.
content_hash / embedded_hash so staleness is a SQL query.model, model_version and dims with every vector; vectors from different models are not comparable and may differ in length.ON DELETE CASCADE from documents to chunks to embeddings so a delete cannot leave orphan vectors that still surface in search.halfvec) or fewer dimensions trades a little quality for a lot of memory.embedded_hash differs from content_hash, and repairs them; this catches lost queue messages.-- Run in any SQLite shell: sqlite3 :memory: then paste the schema and inserts from
-- "Detecting Stale Embeddings, Model Changes and Deletes" above. Exercises:
-- 1. Re-embed chunk 11: UPDATE chunk_embeddings SET embedded_hash = 'h_b2' WHERE chunk_id = 11;
-- then re-run query 1 and confirm it returns no rows.
-- 2. Finish the migration by inserting v2 rows for chunks 11 and 30, re-run query 2, expect no rows.
-- 3. Prove the cascade: DELETE FROM documents WHERE tenant_id = 7; then
-- SELECT COUNT(*) FROM chunk_embeddings WHERE chunk_id = 30; -- expect 0 (or omit PRAGMA foreign_keys = ON and see it stay 1)
-- 4. Write a query that lists documents with at least one stale embedding, using GROUP BY document_id.
--
-- Python: save the chunk-diff snippet (hashlib.sha256(text.encode()).hexdigest()[:8]) and try
-- reordering two chunks; observe which chunks a position-based diff would re-embed versus a hash-based one.
--
-- PostgreSQL + pgvector (needs the extension; write your own results): create chunk_emb_v1 and chunk_emb_v2 as above,
-- then check with EXPLAIN that queries against each table use their own HNSW index.Q: What metadata should be stored with embeddings?
A: At minimum: the chunk id and text, the document id and source, a document version or timestamp, the tenant and access labels, and the position of the chunk in the document. Then the fields that make the vector auditable: the model name, model version, dimension, and a hash of the text that was embedded. Each field costs storage and write work, so store what a filter, a citation, an authorisation check or a staleness query actually needs, and keep the rest in the source table.
Q: How would you update or delete embeddings when documents change?
A: Treat embeddings as derived data with foreign keys to the chunk and document. On a change, diff chunk hashes: keep unchanged vectors, re-embed changed and new chunks, delete removed ones, and do the DB changes in one transaction while the embedding call runs asynchronously so no transaction waits on an external API. Deletes cascade from the document, and a reconciliation job repairs anything a lost message left stale; the trade-off is eventual consistency between the document and its vector, so retrieval may briefly return old content unless you filter by document version.
Q: The team wants to switch to a newer embedding model. What is your plan?
A: Build the new vectors alongside the old, never in place: new table or column with its own index, a rate-limited backfill, dual embedding of live changes, then an offline comparison of retrieval quality on a saved set of queries. Switch readers with a flag and keep the old vectors for a rollback window. The trade-off is temporary double storage and embedding cost, in return for search staying up and a reversible cutover.
Q: Why must you record the model version, and not just the model name?
A: Providers can release new versions or change defaults under a similar name, and vectors from different versions live in different spaces even when the dimension matches. Without a recorded version you cannot tell which rows need re-embedding or which vectors are safe to compare with a new query. The cost is one more column and a slightly more complex query path, which is trivial next to a silent quality regression.
Q: How do dimensions affect your schema and your costs?
A: The dimension fixes the vector column type and the index, and storage grows linearly with it: 1536 float32 dimensions is 6144 bytes per vector, 3072 doubles that. Distance computation time and index memory grow too, so a smaller model or lower precision (for example halfvec in pgvector, where available) can cut cost at some quality loss that you should measure. Because the dimension is part of the column type, changing it is a migration, not an update.
Follow-up: Your retrieval starts returning slightly wrong answers a week after a quiet model upgrade by the provider. How do you diagnose it?
A: Query the embeddings table grouped by model and model version to see whether two versions are now mixed, for example new documents embedded with a different version than older ones while queries use a third. If they are, re-embed the minority version, pin the model version in configuration, and add an alert that fails ingestion when the response's model does not match the pinned one. Then re-run your saved evaluation queries to confirm quality recovered.
Follow-up: A user invokes their right to erasure. Is deleting the source row enough?
A: No. Delete the source document, its chunks and every embedding row for every model version (a cascade does this), and remove derived caches and any search results stored elsewhere. Replicas follow, but backups, logs and third-party stores keep copies until they expire or are purged, so state the retention window rather than claiming immediate total erasure.
Follow-up: Would you re-embed all documents every night to stay safe?
A: Rarely. It wastes embedding cost and churns the ANN index, and it hides bugs instead of fixing them. Change-driven updates keyed on content hashes, plus a cheap reconciliation query for missing or stale rows, give the same guarantee at a fraction of the cost. Full re-embedding is justified only for a model change or a chunking-strategy change.
In a multi-tenant RAG platform, the ingestion path is the part that decides whether the system stays correct over time. A document change lands in the system of record, an outbox row or CDC event triggers the sync worker, the worker diffs chunk hashes and calls the embedding service, and the vector table is updated idempotently keyed by chunk id and model version. The query path reads the active model version from configuration, embeds the question with that model, searches the matching vector table with a tenant filter, and only returns chunks whose document version is current. Monitoring covers the number of chunks missing an embedding, stale embeddings, embedding queue lag and the count of vectors per model version. This is the same read-model pattern as CQRS: the relational tables are the source of truth, and the vector store is a rebuildable projection of them.
Storing rich metadata, hashes and model versions makes every insert a little wider and forces you to write and operate ingestion and reconciliation jobs, in return for detectable staleness, safe migrations and simple deletion. Keeping old and new embeddings side by side during a migration doubles storage and index memory for a while and adds a config flag to the read path, in return for zero downtime and a way back. Embedding asynchronously means readers can briefly see stale content or miss a new document, which is fine for a knowledge base and not fine for anything that requires strict freshness, such as recently revoked access, where the authorisation check must run against the live source of truth, not the derived vectors. This design is over-engineered for a one-off prototype with a static corpus and a single model, where a single table and a nightly rebuild is the right level of effort.