A vector database stores embeddings (fixed-length arrays of floats produced by an embedding model) and answers one question quickly: "which stored vectors are closest to this query vector?", usually combined with a metadata filter such as tenant, language or access level. Interviewers probe it because it is the storage layer under RAG, semantic search and recommendations, and because the honest answers are all capacity and consistency arithmetic: how much RAM do 100M vectors need, what happens to recall when a filter removes 99% of them, what a shard is when the index is a graph, and what happens when you change embedding models. For a Backend or AI engineer the useful skill is telling "vector database", "vector extension" and "search engine with vectors" apart by their operational profile rather than by marketing. The relational side (operators, index syntax, SQL) is covered in Vector Databases & pgvector and Embeddings Storage; this page is the system-design angle: sizing, sharding, filtering, freshness and cost. The algorithms themselves are in Approximate Nearest Neighbor Search.
A vector database is a specialized storage system built to answer one particular question fast: "out of everything I've stored, what's most similar in meaning to this new thing?" It works by converting text, images or other data into long lists of numbers (called vectors) that capture their meaning, then finding the stored items whose number-lists are closest to a new one — which is how it finds things that mean the same thing even if the words are totally different.
Imagine a giant library where books aren't sorted alphabetically by title, but by how similar their subject matter is — so a book about French bread ends up on the shelf right next to books about Italian focaccia and German pretzels, even though none of their titles share a single word. A vector database is like a librarian who can instantly point you to the shelf that's "about the same kind of thing" as the book you're holding, based on what it's actually about rather than what it's called.
Dedicated vector DB Vector extension Search engine + vectors
(Milvus, Qdrant, (pgvector on PostgreSQL) (Elasticsearch/OpenSearch,
Weaviate, Pinecone) Vespa, Solr)
---------------------------------------------------------------------------------------------------
Primary job ANN search over relational rows that full-text/inverted-index
vectors + payload ALSO have a vector col search that ALSO has vectors
Index types HNSW, IVF, PQ, disk HNSW, IVFFlat HNSW (per segment) + inverted
variants index
Filtering built into the any SQL predicate; the filters + BM25 + vector in
index/segment layer planner decides how one query (hybrid natively)
Transactions usually none/limited full ACID, joins, none across docs; near-real-
same txn as your data time refresh
Scaling axis shard + replicate the vertical, read replicas; shard + replicate indexes
index natively sharding is your job natively
Sweet spot 10M .. billions, up to roughly the size you already run it for search
heavy filtered search one node's RAM can hold and want hybrid rankingThe distinction that matters in an interview is what the system was built around. A dedicated vector database owns the index and treats the vector as the primary key of the workload; it can shard, replicate, quantize and compact vector indexes because that is all it does. An extension such as pgvector adds a vector type and ANN index types to a general engine, so you gain transactions, joins, backups and one system to run, and you inherit that engine's limits: an HNSW index that must fit in memory to be fast, index builds that compete with OLTP traffic, and horizontal scaling that you assemble yourself with partitioning or Citus-style sharding. A search engine adds vectors to an inverted index and is the natural home when lexical and semantic ranking must be fused in one request (see Hybrid Search). None of these is universally faster; the honest comparison is measured recall at a latency target on your data, on your filters.
What one logical record becomes inside the system:
id | vector (d floats) | payload / metadata (JSON, filterable) | index entry
42 | [0.12, -0.80, ... d values] | {tenant:"acme", lang:"en", ts:...} | graph links (HNSW) or list id (IVF)
Memory sizing (ILLUSTRATIVE arithmetic; real engines add overhead, check current docs):
raw vectors = N x d x bytes_per_component
10M x 768 x 4 B (float32) = 28.6 GiB
10M x 768 x 1 B (int8) = 7.2 GiB (scalar quantization, ~4x smaller)
1B x 1536 x 4 B (float32) = 5722 GiB (~5.6 TiB: does not fit one machine)
HNSW links, M=16: ~2 x M x 4 B per vector ~ 128 B, so ~1.2 GiB per 10M vectors
PQ code, e.g. 96 bytes per vector instead of 3072: 32x smaller (recall drops; tune it)
Replication multiplies everything: 3 replicas of 28.6 GiB = ~86 GiB of RAM cluster-wide.The first number to compute in any vector-search design is vectors x dimensions x bytes, because latency-optimised ANN indexes are fast mostly when the working set is in RAM. Note how the graph links are small next to the vectors themselves: at typical dimensions the vectors dominate memory, so the three real levers are fewer dimensions (some embedding models support truncated or lower-dimension outputs; check the model's docs and re-measure recall), smaller components (float32 to int8, float16 or binary), and compression such as product quantization with a re-scoring step against full-precision vectors kept on disk. Each lever trades recall for memory, which is why the cost and recall targets have to be set together.
Storage engines usually organise data into immutable segments (a pattern shared with LSM trees and Lucene): new vectors land in a small mutable buffer that is searched by brute force, background jobs seal it into an ANN-indexed segment, and compaction merges segments. Queries fan out to all segments and merge the top-k. That structure explains the operational behaviour: freshly written vectors are searchable almost immediately but on a slower path, compaction consumes CPU, and disk-backed variants (memory-mapped or SSD-resident graph indexes such as DiskANN-style designs) keep only compressed vectors in RAM at the price of higher and more variable tail latency. See Storage Internals & WAL for the write-ahead and segment ideas this builds on.
Query: "refund policy", filter tenant = "acme" (acme owns 5% of all vectors), k = 10
POST-FILTER (search first, filter the result)
ANN returns global top-10 -> only ~0.5 of them belong to acme -> 0-2 results, under-filled
Fix attempt: over-fetch top-500, then filter (cost grows with 1 / selectivity)
PRE-FILTER (restrict first, search the subset)
build the allowed-id set / bitmap, search ONLY those -> k results, but:
- a graph index walks through NON-matching nodes to stay connected; if most are masked out,
the walk can stall or lose recall (the "filtered HNSW" problem)
- when the filter is very selective, brute force over the small matching set is cheaper AND exact
IN-INDEX / FILTER-AWARE search (what dedicated engines aim for)
choose a plan by filter selectivity: very selective -> scan the matching rows exactly
mild -> ANN with the filter applied while traversing
none -> plain ANNFiltering is where "just add a vector index" designs quietly fail, because the nearest neighbours in embedding space have no relationship to your metadata. Post-filtering returns fewer than k results, or none, whenever the filter is selective, and it does so silently (the demo below measures this). Pre-filtering fixes the count but interacts with the index structure: a navigable graph is built over all vectors, so masking most of them can leave the search with few valid entry points and paths. Practical systems therefore choose a strategy per query from the estimated selectivity, often falling back to exact scan when the filter matches only thousands of rows, and support payload indexes (equality, range, keyword) so the allowed set is cheap to compute. Two rules of thumb worth stating with hedges: check that your chosen engine documents how it filters (during traversal, before, or after), and always test recall with your real filters, not only unfiltered queries.
Hard access-control filters (tenant, permissions) deserve special care because correctness matters more than recall: a post-filter design that fetches top-k then drops rows the user may not see is secure but empty; a design that skips the filter to "improve recall" is a data leak. See Authentication & Authorization.
Scatter-gather over shards (random/hash partitioning of vectors):
client -> router -> shard 1 (own HNSW over 1/S of data) -- top-k --+
-> shard 2 ... -- top-k --+--> merge, take global top-k
-> shard S -- top-k --+
Every query touches EVERY shard (nearest neighbours can be anywhere), so
latency = slowest shard (tail latency), throughput = replicas x shards / per-shard cost.
Tenancy models:
A) shared index + tenant_id filter : cheapest, filtering recall risk, noisy neighbours
B) index/collection per tenant : strong isolation, easy delete-tenant, cost per tiny index
C) partition/namespace per tenant : middle ground (engine-specific), route the query by tenant
D) big tenants isolated, small tenants pooled : common in practiceSharding a vector index differs from sharding a key-value table in one way that changes the arithmetic: there is no key that tells you which shard holds the answer, so each shard builds its own graph over its slice and a query is scatter-gathered to all shards, each returning its local top-k that a coordinator merges (the same pattern as distributed text search). Adding shards therefore cuts memory per node and per-shard search time, but every query still pays fan-out and waits for the slowest shard; adding replicas raises query throughput and availability without reducing per-query work. Partitioning by a business key (tenant, region, time) is the exception: if queries always carry that key, the router can search one shard, which is why per-tenant partitioning is attractive, and why hot tenants and skew are its failure mode. General sharding trade-offs are in Database Replication & Sharding and Partitioning & Sharding.
Multi-tenancy is a cost-versus-isolation decision. One shared index with a tenant_id filter is cheapest but inherits the filtering problem above and lets one tenant's bulk load slow everyone; an index per tenant isolates performance and makes deleting a tenant (a compliance requirement) trivial, but thousands of tiny indexes carry fixed memory and management overhead. Most large deployments mix them: pool the long tail, isolate the whales, and route by tenant so queries never scan tenants they cannot access.
Write path (typical segment-based engine):
upsert -> WAL -> mutable buffer (brute-force searchable) -> seal -> build ANN segment -> compact
DELETE: usually a tombstone (id marked dead, filtered out at query time)
space and graph nodes are reclaimed at compaction/rebuild
UPDATE: delete + insert (the vector changed, so its graph neighbours are wrong)
an update that changes only payload may avoid touching the vector index
Freshness = time from "document written" to "returned by search"
buffer/brute-force path : seconds sealed+indexed path : depends on flush/refresh policy
Changing the embedding model:
old vectors and new vectors live in DIFFERENT spaces -> cannot be compared or mixed
=> re-embed everything into a NEW index, dual-run, cut over, delete the old (blue/green)
cost = (corpus tokens x embedding price) + index build time + 2x storage during migrationANN graphs are cheap to grow and awkward to shrink: inserting a node is a local operation, but removing one leaves a hole other nodes still point at, so engines mark deletions and clean them up later. A workload with heavy churn (a constantly edited knowledge base, expiring sessions) will show growing dead-space, degrading recall or latency, and periodic rebuilds; plan a compaction window and monitor the fraction of tombstones. Freshness has a tail as well: RAG over a wiki that changes hourly needs the write path (chunk, embed, upsert) to keep up, and an embedding-model outage stalls it, which is why ingestion is usually a queued pipeline (see Chunking & Embedding Strategies and Message Queues & Event Streaming).
The most expensive "update" is changing the embedding model or its dimension. Vectors from different models are not comparable, so the entire corpus must be re-embedded and re-indexed, typically into a parallel index that is validated and then swapped in. Store the model name and version with each vector so you can tell which space it belongs to, and keep the source text so you can re-embed at all.
N x dims x bytes; 10M x 768 x float32 is about 28.6 GiB before graph links, replicas and headroom (ILLUSTRATIVE arithmetic).Brute-force exact k-NN over 20,000 seeded random unit vectors (32 dimensions), the memory arithmetic from above, and two ways of combining a metadata filter with similarity search. Metadata is independent of vector position, which is the worst case for post-filtering (real data is often correlated, and the effect may be smaller or larger). Standard library only; it takes a few seconds.
import random, math
random.seed(7)
N, DIM, K = 20000, 32, 10
N_TENANTS, N_CATS = 20, 10
def unit(v):
n = math.sqrt(sum(x * x for x in v))
return [x / n for x in v]
def rand_vec():
return unit([random.gauss(0, 1) for _ in range(DIM)])
# Each row = (id, vector, metadata). Metadata is unrelated to vector position (worst case for post-filtering).
corpus = [(i, rand_vec(), {"tenant": random.randrange(N_TENANTS), "cat": random.randrange(N_CATS)})
for i in range(N)]
# ---- 1. Memory arithmetic (ILLUSTRATIVE; raw vectors only, no index, no metadata) ----
def gib(n, d, b):
return n * d * b / 2**30
print("Raw vector memory = vectors x dims x bytes, in GiB:")
for n, d, b, label in [(N, DIM, 4, "this demo (20k x 32 float32)"),
(10_000_000, 768, 4, "10M x 768 float32"),
(10_000_000, 768, 1, "10M x 768 int8"),
(1_000_000_000, 1536, 4, "1B x 1536 float32")]:
print(f" {label:30s} {gib(n, d, b):10.4f} GiB")
print(f" HNSW links, M=16, ~2*M*4 B/vector (ILLUSTRATIVE): {10_000_000*2*16*4/2**30:.2f} GiB for 10M vectors")
# ---- 2. Exact k-NN by brute force (dot product == cosine on unit vectors) ----
def dot(a, b):
return sum(x * y for x, y in zip(a, b))
def ranking(query):
"""All rows ordered best-first: the exact answer, used as ground truth."""
return sorted(corpus, key=lambda r: -dot(query, r[1]))
queries = [rand_vec() for _ in range(100)]
ranked = [ranking(q) for q in queries] # computed once, reused below
def search_then_filter(rk, pred, k, overfetch=1):
"""Take the top k*overfetch by similarity, THEN drop rows failing the filter."""
return [r for r in rk[:k * overfetch] if pred(r[2])][:k]
def filter_then_search(rk, pred, k):
"""Restrict to matching rows first, then take the top k among them."""
return [r for r in rk if pred(r[2])][:k]
def report(title, pred):
sel = sum(1 for r in corpus if pred(r[2])) / N
print(f"\n{title}: filter matches {sel:.1%} of rows (~{int(sel*N)})")
for name, fn in [("search-then-filter x1", lambda rk: search_then_filter(rk, pred, K, 1)),
("search-then-filter x10", lambda rk: search_then_filter(rk, pred, K, 10)),
("search-then-filter x50", lambda rk: search_then_filter(rk, pred, K, 50)),
("filter-then-search", lambda rk: filter_then_search(rk, pred, K))]:
counts = [len(fn(rk)) for rk in ranked]
under = sum(c < K for c in counts)
recall = sum(len({r[0] for r in fn(rk)} & {r[0] for r in filter_then_search(rk, pred, K)}) / K
for rk in ranked) / len(ranked)
print(f" {name:24s} avg {sum(counts)/len(counts):5.2f}/{K} results, "
f"under-filled {under:3d}/100, recall vs true filtered top-{K} {recall:.2f}")
report("Broad filter (tenant==3)", lambda m: m["tenant"] == 3)
report("Narrow filter (tenant==3 AND cat==4)", lambda m: m["tenant"] == 3 and m["cat"] == 4)
# Output:
# Raw vector memory = vectors x dims x bytes, in GiB:
# this demo (20k x 32 float32) 0.0024 GiB
# 10M x 768 float32 28.6102 GiB
# 10M x 768 int8 7.1526 GiB
# 1B x 1536 float32 5722.0459 GiB
# HNSW links, M=16, ~2*M*4 B/vector (ILLUSTRATIVE): 1.19 GiB for 10M vectors
#
# Broad filter (tenant==3): filter matches 5.0% of rows (~1008)
# search-then-filter x1 avg 0.54/10 results, under-filled 100/100, recall vs true filtered top-10 0.05
# search-then-filter x10 avg 4.99/10 results, under-filled 99/100, recall vs true filtered top-10 0.50
# search-then-filter x50 avg 10.00/10 results, under-filled 0/100, recall vs true filtered top-10 1.00
# filter-then-search avg 10.00/10 results, under-filled 0/100, recall vs true filtered top-10 1.00
#
# Narrow filter (tenant==3 AND cat==4): filter matches 0.6% of rows (~111)
# search-then-filter x1 avg 0.10/10 results, under-filled 100/100, recall vs true filtered top-10 0.01
# search-then-filter x10 avg 0.61/10 results, under-filled 100/100, recall vs true filtered top-10 0.06
# search-then-filter x50 avg 3.02/10 results, under-filled 100/100, recall vs true filtered top-10 0.30
# filter-then-search avg 10.00/10 results, under-filled 0/100, recall vs true filtered top-10 1.00Read the result as arithmetic: a filter matching fraction s of the corpus leaves about k x s survivors of a top-k post-filter (0.5 of 10 at 5%, 0.06 at 0.6%), so to get k results you must over-fetch roughly k / s candidates. At 0.6% selectivity even x50 fails, and in a real ANN engine over-fetching also raises latency. Here the pre-filter is exact because brute force ignores graph structure; with an HNSW index, pre-filtering is where traversal quality becomes the question. Real engine usage (labelled pseudocode, not run here):
-- PostgreSQL + pgvector (needs the extension; no output is invented here)
CREATE TABLE chunks (id bigint PRIMARY KEY, tenant text, embedding vector(768));
CREATE INDEX ON chunks USING hnsw (embedding vector_cosine_ops);
SET hnsw.ef_search = 100;
SELECT id FROM chunks WHERE tenant = 'acme' ORDER BY embedding <=> $1 LIMIT 10;
-- Check with EXPLAIN whether the planner uses the vector index or a filter first, and
-- whether fewer than 10 rows come back for selective tenants (a known iterative-scan/recall concern;
-- see the current pgvector docs for its filtering options).Q: When would you choose pgvector over a dedicated vector database?
A: When the vectors sit next to relational data you already keep in PostgreSQL, the corpus fits comfortably in one node's RAM (roughly single-digit to low tens of millions of vectors at typical dimensions, depending on hardware), and you value transactions, joins and one system to operate. Move to a dedicated engine when you need native sharding, billions of vectors, heavy filtered search, or an index whose build and compaction must not compete with OLTP. The trade-off is operational simplicity against scale ceiling; measure recall and p99 on your data rather than trusting either camp.
Q: How much memory does an index of 50M 1024-dimension float32 embeddings need?
A: Raw vectors are 50M x 1024 x 4 B, about 190 GiB (ILLUSTRATIVE, plus graph links, payload and headroom), and three replicas triple the cluster total. That does not fit one commodity node, so you shard, and/or quantize: int8 cuts vectors to about 48 GiB, product quantization further, with recall cost. The follow-through is to state the recall target and re-measure after each compression step.
Q: Why does filtered vector search return fewer than k results?
A: Post-filtering takes the top-k by similarity and then removes rows that fail the filter, and similarity is unrelated to metadata, so about k times the selectivity survive. With a 5% filter that is roughly 0.5 results out of 10. The fix is filter-aware search (pre-filtering or in-traversal filtering), an over-fetch factor of about k divided by selectivity, or an exact scan when the filter is highly selective.
Q: Why can pre-filtering hurt an HNSW index?
A: The graph was built over all vectors and search relies on hopping between neighbours; if most nodes are masked out, the walk still needs to traverse non-matching nodes to reach matching regions, and it may terminate early with few or low-quality matches. Engines mitigate this by choosing a plan from filter selectivity, building extra links for filtered categories, or falling back to brute force on the small allowed set. The trade-off is engine complexity and unpredictable latency between plans.
Q: How do you shard a vector index, and what does a query cost?
A: Typically hash or random partitioning, with each shard holding its own ANN index over its slice. A query is scatter-gathered to every shard because neighbours can be anywhere, and a coordinator merges the local top-k lists. Capacity scales with shards, per-query latency is bounded by the slowest shard, and throughput scales with replicas; partitioning by tenant lets you route to one shard when every query carries that key.
Q: A tenant asks you to delete all their data. What does that involve in a vector store?
A: With an index or partition per tenant you drop it and the job is done. With a shared index you delete by filter, which usually creates tombstones that are physically removed only at compaction, so you must also handle backups, replicas, caches and any derived stores. Model it as a compliance requirement up front, since it is the strongest argument for physical tenant isolation.
Follow-up: You swap to a better embedding model. What is the migration plan?
A: Vectors from different models are not comparable, so build a new index with the new model, re-embedding all source text, while the old index keeps serving. Backfill, then dual-write new content to both, evaluate retrieval quality offline (recall@k, MRR against a labelled set), shift a small share of traffic, then cut over and retire the old index. Cost is embedding the whole corpus plus temporarily double storage, so keep the raw text and a model-version field.
Follow-up: Latency was fine at 1M vectors and degraded at 30M with the same settings. Why?
A: Possible causes: the index no longer fits in RAM and spills to disk, so each graph hop becomes an I/O; larger graphs need a higher search-effort parameter for the same recall, which costs more distance computations; or compaction and merges contend with queries. Diagnose by checking memory versus index size, cache hit rates and recall at the current settings before adding hardware, and consider quantization or sharding.
Follow-up: How do you guarantee users never see documents they lack permission for?
A: Enforce the permission filter inside the retrieval query itself (tenant partition or an ACL filter applied before or during search), never after generation and never only in the UI. Post-filtering is safe but can return too little, so combine it with filter-aware search and monitor under-filled results. Treat the vector store as another data store with its own access control, and log which chunks were retrieved for audit.
In a RAG support-bot or semantic-search design (see Retrieval-Augmented Generation and Design a RAG-Based Support Chatbot), the vector store is the component that sets your memory and cost floor. State assumptions and do the arithmetic aloud: assume 5M documents chunked into 20M chunks at 768 dimensions float32 is about 57 GiB of raw vectors (20M x 768 x 4 B, ILLUSTRATIVE), so with graph links, payload and replicas you provision a few hundred GiB of RAM across a small cluster, or halve it with int8 quantization. Then walk the write path (queued ingestion, embedding rate limits, idempotent upserts by chunk id), the read path (embed the query, filter by tenant and permissions, top-50 candidates, re-rank to 5 to 10), and freshness targets (seconds for new tickets, hours for archived docs). Mention that embedding calls and re-embedding after a model change are a real cost line (see Cost Optimization for AI Systems), that the query embedding can be cached (see Caching Strategies), and that the vector DB is usually the fastest hop in the pipeline compared with the LLM call, so optimising it matters mainly for cost and recall rather than for end-to-end latency.
The decision is where the vectors live and how much of the machinery you operate. A dedicated vector database buys native sharding, quantization, filter-aware planning and headroom to billions of vectors, and costs another stateful system to run, secure, back up and keep consistent with the source of truth. A pgvector-style extension buys ACID, joins and a single system, and costs scale ceiling, index builds competing with transactional load, and DIY sharding. A search engine with vectors wins when hybrid lexical and semantic ranking is the requirement. It is the wrong choice to adopt a dedicated store for a corpus of a few hundred thousand vectors that brute force or one Postgres node handles easily, and equally wrong to force hundreds of millions of filtered vectors into a database that must hold the index in RAM next to your OLTP workload. Whatever you pick, the recurring costs are RAM for the index (the dominant line), embedding compute for ingestion and migrations, and the engineering to keep recall measured: approximate search silently degrades under filters, deletes and growth, so treat recall as a monitored metric, not a one-time tuning result.