Chunking is the step that decides what a retrieval system can possibly find. A RAG pipeline never searches your documents; it searches the chunks you cut from them, represented as embedding vectors. If the sentence that answers a question was sliced in half, buried in a 2,000-token block of unrelated text, or stripped of the heading that gave it meaning, no vector index, re-ranker or prompt can recover it. Interviewers probe this topic because it separates people who have wired up a tutorial from people who have debugged a RAG system: the usual first question is "answers are wrong, where do you look?", and the honest answer starts with chunk boundaries and the embedding model, not the LLM. For a Backend or AI engineer it is also an operations topic, because chunker and embedding-model choices become stored data: changing either one later means rebuilding the index.
Chunking is a trade-off, not a solved problem. There is no universally best size or strategy; the right answer depends on how self-contained the passages in your corpus are, what your queries look like, and what your embedding model handles well. What you can do is measure, and this page shows both the design space and a small experiment. Related pages: Retrieval-Augmented Generation for the whole pipeline, Vector Databases and Approximate Nearest Neighbor Search for the index, and Storing Embeddings — Schema, Dimensions & Versioning for how the vectors and their metadata live in a database.
Chunking is the process of cutting a long document into smaller pieces before feeding it to an AI system. A single book or manual is too big to search or hand to a chatbot all at once, so it gets sliced into bite-sized sections, like paragraphs or short passages, that can each be looked up and understood on their own.
An embedding is a way of turning a piece of text into a list of numbers that captures its meaning, so a computer can compare how similar two pieces of text are just by comparing their number lists. Two sentences that mean roughly the same thing end up with number lists that are close together, even if they don't share many of the same words.
Imagine a huge library with no card catalog, just one giant book containing every other book glued end to end. Chunking is like going through and cutting that giant book into individual chapters and putting each one on its own shelf. Embedding is like giving each chapter a scent that librarians (and readers) can sniff to tell which chapters are about similar topics, so that when you ask "where do I find information about roses," you're guided straight to the gardening chapters instead of having to read the entire library cover to cover.
Source text: "... within 30 days of delivery for a full refund. Items must be unused and in
original packaging. Refunds are issued to the original payment method
within 5 business days after the warehouse receives the parcel. ..."
FIXED-SIZE (every N characters or tokens, optional overlap)
[ ...within 30 days of delivery for a full refund. Items must be unused and in original pa ]
[ ckaging. Refunds are issued to the original payment method within 5 business days after ]
^ cut through the middle of a word / sentence: two half-facts, neither retrievable
SENTENCE / PARAGRAPH-AWARE (pack whole sentences until the size budget is reached)
[ Orders can be returned within 30 days ... Items must be unused and in original packaging. ]
[ Items must be unused and in original packaging. Refunds are issued to ... parcel. ] <- 1 sentence of overlap
STRUCTURE-AWARE (split on headings, sections, list items, code blocks, table rows)
# Returns policy chunk = "Returns policy > Refunds" + the section body
## Refunds keeps the heading path with the text so the vector "knows" what it is about
SEMANTIC (embed sentences, start a new chunk where similarity to the previous sentences drops)
s1 s2 s3 | s4 s5 | s6 s7 s8 s9 | <- boundaries where topic shifts; costs one embedding per sentence at ingestFixed-size splitting is the baseline because it is trivial, deterministic and fast, and it is also the strategy most likely to cut a fact in half. Sentence-aware splitting keeps sentences intact, which fixes the worst failure at almost no cost. Structure-aware splitting uses the document's own markup (Markdown or HTML headings, PDF sections, code functions, table rows) and is usually the biggest single upgrade for manuals, wikis and API docs, because the heading path is real information. Semantic chunking chooses boundaries by topic shift using embeddings; it can produce more coherent chunks but costs extra embedding calls at ingestion, adds tuning knobs (the similarity threshold), and its benefit over structure-aware splitting is not guaranteed, so evaluate before adopting it.
Chunk size is usually measured in tokens, not characters, because the embedding model and the LLM both consume tokens, and every embedding model has a maximum input length (check the model's documentation; some providers truncate silently). Common starting points are a few hundred tokens per chunk, but treat any such number as a hypothesis to test rather than a rule. Real corpora are messier than tutorials: text extracted from PDFs has broken line wraps and headers repeated on every page, tables lose their structure when flattened, and code needs to be split on function boundaries. Cleaning and structure extraction often matter more than the choice of splitter.
chunk too SMALL chunk too LARGE
"It is valid for 15 minutes." [ refunds + shipping + passwords + support hours ]
what is "It"? which token/link? one vector must "average" four topics
+ precise match to a narrow question - vector is a blur: specific questions match weakly
- context (subject, conditions) is lost - top-k returns pages of text: cost, latency, distraction
- many more vectors to store and search - fewer, cheaper vectors
overlap (chunk i and i+1 share some text):
no overlap [ AAAAAAAA ][ BBBBBBBB ][ CCCCCCCC ] a fact straddling a cut lives in neither chunk whole
with overlap [ AAAAAAAA ] a fact near the cut appears whole in at least one
[ AAB BBBBB ] cost: N% more vectors and near-duplicate results
[ BBC CCCCC ]
SMALL-TO-BIG (parent-child)
index / search : child chunks (single sentences or ~100 tokens) -> precise vector match
return to LLM : the child's PARENT (surrounding section or a window of neighbours) -> enough context
storage : child.parent_id -> parent text (a normal key-value or row lookup, no vector needed)Small chunks give precise matching but strip context; large chunks keep context but blur the embedding and burn prompt tokens. Overlap is insurance against boundary cuts: it costs storage (10-20% overlap means roughly 10-25% more vectors, depending on chunk size) and creates near-duplicate hits that waste top-k slots unless you de-duplicate. Overlap mainly rescues fixed-size splitting; a sentence-aware chunker already avoids the worst cuts, and there overlap mostly carries a little neighbouring context forward. Small-to-big (also called parent-child or "sentence-window" retrieval) separates the two jobs: search on small units for precision, then return the surrounding parent for context. It costs an extra lookup and a larger prompt, and it can pull in irrelevant text if parents are huge.
Here is small-to-big on the same toy corpus used in "Try It Yourself" below (append this to that file to run it). Both configurations retrieve the right answer on this tiny sample; the visible difference is how much context each hands to the LLM, which is the point of the pattern: the child decides what matched, the parent decides what the model reads.
# Appended to the same file as the chunkers above (reuses sentence_chunks, cosine, toks, QUERIES, evaluate)
parents = sentence_chunks(TEXT, 400, 0) # "big" chunks handed to the LLM
children = [(s, i) for i, p in enumerate(parents) for s in split_sentences(p)] # "small" units that are searched
def small_to_big(query):
best = max(children, key=lambda ch: cosine(toks(query), toks(ch[0]))) # nearest single sentence
return parents[best[1]] # ...but return its parent
hits = sum(gold in small_to_big(q) for q, gold in QUERIES)
ctx = sum(len(small_to_big(q)) for q, _ in QUERIES) / len(QUERIES)
print(f"small-to-big: hit@1 {hits}/7, avg context returned {ctx:.0f} chars")
flat = sentence_chunks(TEXT, 200, 1)
ctx2 = sum(len(max(flat, key=lambda c: cosine(toks(q), toks(c)))) for q, _ in QUERIES) / 7
print(f"flat sentence 200/1: hit@1 {evaluate(flat,1)}/7, avg context returned {ctx2:.0f} chars")
# Output:
# small-to-big: hit@1 7/7, avg context returned 348 chars
# flat sentence 200/1: hit@1 7/7, avg context returned 151 charsOne stored chunk (a row, or a vector-DB record with payload):
chunk_id stable id (e.g. hash of doc_id + offset + content)
doc_id, source where it came from (URL, file path, page number, section heading path)
text the chunk text (what the LLM will read)
parent_id for small-to-big
content_hash detect unchanged chunks so re-ingestion skips re-embedding
updated_at, version, language, tenant_id / acl freshness filters and access control
embedding_model e.g. "provider/model-name@version" <- REQUIRED to manage upgrades
chunker_version e.g. "sentence-400-o1@v3" <- lets you find chunks made by an old strategy
embedding float[d]
Raw vector storage, float32, before index overhead and replicas (10M chunks):import hashlib, math, random, re
# 1) raw vector storage, float32, no index overhead, no replicas
for dims in (384, 768, 1536, 3072):
n = 10_000_000
print(f"{n:,} chunks x {dims:4d} dims x 4 B = {n*dims*4/1e9:6.1f} GB")
print("1536 dims, int8 quantised =", f"{10_000_000*1536/1e9:.1f} GB")
# 2) why two embedding models cannot share one index
def embedder(seed, dims=64):
def embed(text):
v = [0.0]*dims
for w in re.findall(r"[a-z0-9]+", text.lower()):
rng = random.Random(hashlib.md5(f"{seed}:{w}".encode()).hexdigest())
for i in range(dims): v[i] += rng.gauss(0, 1)
n = math.sqrt(sum(x*x for x in v)) or 1
return [x/n for x in v]
return embed
def cos(a, b): return sum(x*y for x, y in zip(a, b))
docs = ["refund within 30 days of delivery", "express shipping is unavailable on weekends",
"password reset link valid 15 minutes", "premium phone line open 24 hours",
"customs fees paid by the buyer", "chat support weekdays 9 am to 6 pm"]
queries = ["refund 30 days", "express weekends", "password reset link", "premium phone", "customs buyer", "chat support weekdays"]
A, B = embedder("model-A"), embedder("model-B") # same dimension, different "models"
index = [A(d) for d in docs]
def recall(qe):
ok = 0
for i, q in enumerate(queries):
qv = qe(q); best = max(range(len(docs)), key=lambda j: cos(qv, index[j])); ok += best == i
return ok
print("index built with A, query embedded with A:", recall(A), "/ 6 correct top-1")
print("index built with A, query embedded with B:", recall(B), "/ 6 correct top-1 (chance would be about 1 of 6)")
# Output:
# 10,000,000 chunks x 384 dims x 4 B = 15.4 GB
# 10,000,000 chunks x 768 dims x 4 B = 30.7 GB
# 10,000,000 chunks x 1536 dims x 4 B = 61.4 GB
# 10,000,000 chunks x 3072 dims x 4 B = 122.9 GB
# 1536 dims, int8 quantised = 15.4 GB
# index built with A, query embedded with A: 6 / 6 correct top-1
# index built with A, query embedded with B: 2 / 6 correct top-1 (chance would be about 1 of 6)The dimension count is a direct multiplier on memory. The printed table is only raw float32 storage: real vector indexes (HNSW graphs, for instance) add overhead on top, and replication multiplies it again, so treat these as lower bounds. Higher-dimensional embeddings can capture more nuance but cost more RAM, more disk, more bandwidth and slower distance computations; the quality gain from more dimensions has diminishing returns and depends on the model. Two common mitigations are quantization (storing int8 or binary codes, with some accuracy loss) and, for models specifically trained for it (often called Matryoshka-style), truncating vectors to fewer dimensions. Truncation is only valid for models documented to support it; cutting dimensions off an ordinary embedding can damage quality badly.
Choosing a model is an evaluation problem: look at public retrieval benchmarks such as MTEB for a shortlist, then test the shortlist on your documents and queries, because domain vocabulary (legal, medical, code) changes rankings. Also weigh context length, language coverage, price or self-hosting cost, latency, and whether the model needs a query/document prefix (some do, and forgetting it quietly hurts recall). For the storage schema side (dimension columns, model-version columns, index rebuilds) see Storing Embeddings.
Vector space of model A Vector space of model B
"refund" -> [0.12, -0.40, ...] "refund" -> [-0.71, 0.05, ...] different axes, different meaning
Each model learns its own coordinate system. The numbers only mean something relative
to other vectors from the SAME model. Same dimension count does not make them compatible.
Migration to a new embedding model (blue/green index):
1. create index_v2 (new dimension, model id in the collection name or metadata)
2. backfill: re-embed every chunk with the new model (batch, rate-limited, resumable)
3. dual-write new/changed documents to both indexes while the backfill runs
4. shadow-evaluate v2 on the labelled query set (recall@k, MRR) and compare with v1
5. switch query traffic (flag / canary), keep v1 for rollback
6. delete index_v1 after the rollback windowA vector is meaningless in isolation: it only has geometry relative to other vectors produced by the same model. If you embed the query with a different model than the index, similarity scores become close to noise even when the dimensions happen to match (otherwise you would at least get an error). The second half of the earlier script demonstrates this with two toy "models": the same six documents retrieved 6 of 6 correctly with matched models and only 2 of 6 with mismatched ones, roughly chance level for such a small sample. The same applies to a silent provider-side model update behind an unchanged name, which is why you store an explicit model identifier and, where the provider offers it, pin a specific model version.
Re-embedding is therefore a data migration, and it is not free. As an ILLUSTRATIVE estimate: 10M chunks of about 300 tokens each is 3 billion tokens. At an assumed price of $0.10 per million tokens (an invented round number; check the provider's current price) that is about $300 of embedding calls; at $1 per million it is $3,000; self-hosting swaps that for GPU-hours. Rate limits may stretch the wall-clock time to hours or days, and you need the old index running throughout. Store the raw chunk text, so you never have to re-parse documents, and a content_hash, so unchanged chunks are not re-embedded during routine updates.
Build a labelled set: (query, answer_span_or_source) ~50-200 real or realistic questions
Label at the level of the ANSWER TEXT or the source document, not chunk ids,
because a chunk id changes every time you change the chunking strategy.
For each candidate config (chunker + size + overlap + embedding model + k):
ingest -> retrieve top-k for each query -> hit if the answer span is fully inside a retrieved chunk
recall@k = hits / queries MRR = mean(1 / rank of first hit) (nDCG if graded relevance)
Judge retrieval first (recall@k, MRR, nDCG). Only then look at generation
(faithfulness / groundedness to the retrieved context). A wrong final answer with the right chunk
retrieved is a generation problem; with the wrong chunks it is a retrieval problem.Retrieval quality bounds answer quality, so measure retrieval on its own. The "hit only if the whole answer sentence sits inside one retrieved chunk" rule in the demo below is deliberate: it is what penalises fixed-size cuts, where half of the fact is in each of two chunks and neither matches well or answers alone. Keep evaluation honest with a few practices: hold out queries you did not use when tuning, include hard cases (multi-sentence answers, tables, questions using different words than the document), and log which queries fail so you can see whether the cause is chunking, embeddings or a missing keyword match. If exact tokens (product codes, error numbers) matter, look at hybrid search, which combines keyword and vector scores, because no chunking strategy fixes an embedding model's weakness on rare identifiers.
Standard-library Python only. It builds two chunkers (fixed-size with character overlap, and sentence-aware with sentence overlap) over a small sample text, prints chunk counts and boundaries, and runs a tiny retrieval comparison. Retrieval here is a bag-of-words cosine score, not a real embedding model, and the corpus is 864 characters with 7 queries, so read the numbers as an illustration of the mechanism (facts cut by boundaries become unretrievable), not as a benchmark of either strategy.
import math, re
from collections import Counter
TEXT = (
"Orders can be returned within 30 days of delivery for a full refund. "
"Items must be unused and in original packaging. "
"Refunds are issued to the original payment method within 5 business days after the warehouse receives the parcel. "
"Gift cards are non-refundable. "
"Standard shipping takes 3 to 5 business days inside the country. "
"Express shipping delivers within 24 hours but is unavailable on weekends. "
"International orders ship through a partner carrier and customs fees are paid by the buyer. "
"To reset your password, open Settings, choose Security and press Reset password. "
"A reset link is emailed to you and stays valid for 15 minutes. "
"If you enable two-factor authentication, you must keep your recovery codes in a safe place. "
"Support is available by chat from 9 am to 6 pm on weekdays. "
"Premium subscribers get a dedicated phone line that is open 24 hours a day."
)
def split_sentences(text):
return [s.strip() for s in re.split(r"(?<=[.!?])\s+", text) if s.strip()]
def fixed_chunks(text, size, overlap):
step = size - overlap
return [text[i:i + size] for i in range(0, len(text), step) if text[i:i + size].strip()]
def sentence_chunks(text, size, overlap_sents):
sents, chunks, cur = split_sentences(text), [], []
for s in sents:
if cur and len(" ".join(cur + [s])) > size:
chunks.append(" ".join(cur))
cur = cur[-overlap_sents:] if overlap_sents else [] # carry the last sentence(s) forward
cur.append(s)
if cur:
chunks.append(" ".join(cur))
return chunks
STOP = set("the a an to of in on for is are be by and or you your it at from after with that within".split())
def toks(t):
return [w for w in re.findall(r"[a-z0-9]+", t.lower()) if w not in STOP]
def cosine(a, b):
ca, cb = Counter(a), Counter(b)
dot = sum(ca[w] * cb[w] for w in ca)
na = math.sqrt(sum(v * v for v in ca.values())); nb = math.sqrt(sum(v * v for v in cb.values()))
return dot / (na * nb) if na and nb else 0.0
# (query, the ONE sentence that contains the answer)
QUERIES = [
("how long do I have to return an order", "Orders can be returned within 30 days of delivery for a full refund."),
("when do refunds reach my payment method", "Refunds are issued to the original payment method within 5 business days after the warehouse receives the parcel."),
("is express shipping available on weekends", "Express shipping delivers within 24 hours but is unavailable on weekends."),
("who pays customs for international orders", "International orders ship through a partner carrier and customs fees are paid by the buyer."),
("how long is the password reset link valid", "A reset link is emailed to you and stays valid for 15 minutes."),
("what hours is chat support open", "Support is available by chat from 9 am to 6 pm on weekdays."),
("is there a phone line for premium users", "Premium subscribers get a dedicated phone line that is open 24 hours a day."),
]
def evaluate(chunks, k):
hits = 0
for q, gold in QUERIES:
ranked = sorted(chunks, key=lambda c: cosine(toks(q), toks(c)), reverse=True)[:k]
hits += any(gold in c for c in ranked) # hit only if the WHOLE answer sentence is inside a retrieved chunk
return hits
def cut_sentences(chunks):
"""How many original sentences are not fully contained in any single chunk?"""
return sum(not any(s in c for c in chunks) for s in split_sentences(TEXT))
sents = split_sentences(TEXT)
print(f"text: {len(TEXT)} chars, {len(sents)} sentences\n")
print(f"{'strategy':26}{'chunks':>7}{'avg len':>9}{'cut sents':>10}{'hit@1':>7}{'hit@2':>7}")
configs = [
("fixed 200, overlap 0", fixed_chunks(TEXT, 200, 0)),
("fixed 200, overlap 40", fixed_chunks(TEXT, 200, 40)),
("fixed 400, overlap 0", fixed_chunks(TEXT, 400, 0)),
("sentence 200, overlap 0", sentence_chunks(TEXT, 200, 0)),
("sentence 200, overlap 1", sentence_chunks(TEXT, 200, 1)),
("sentence 400, overlap 0", sentence_chunks(TEXT, 400, 0)),
]
for name, ch in configs:
avg = sum(map(len, ch)) / len(ch)
print(f"{name:26}{len(ch):>7}{avg:>9.0f}{cut_sentences(ch):>10}{evaluate(ch,1):>5}/7{evaluate(ch,2):>5}/7")
print("\nFixed 200/0 boundaries (| marks each cut):")
for c in fixed_chunks(TEXT, 200, 0)[:3]:
print(" ...", repr(c[-28:]), "|")
print("\nSentence 200/1 chunks:")
for i, c in enumerate(sentence_chunks(TEXT, 200, 1)[:3]):
print(f" [{i}] {c[:60]}... ({len(c)} chars)")
# Output:
# text: 864 chars, 12 sentences
#
# strategy chunks avg len cut sents hit@1 hit@2
# fixed 200, overlap 0 5 173 3 3/7 4/7
# fixed 200, overlap 40 6 177 3 4/7 5/7
# fixed 400, overlap 0 3 288 1 5/7 6/7
# sentence 200, overlap 0 6 143 0 6/7 7/7
# sentence 200, overlap 1 9 155 0 7/7 7/7
# sentence 400, overlap 0 3 287 0 6/7 7/7
#
# Fixed 200/0 boundaries (| marks each cut):
# ... 'n 5 business days after the ' |
# ... ' is unavailable on weekends.' |
# ... '. A reset link is emailed to' |
#
# Sentence 200/1 chunks:
# [0] Orders can be returned within 30 days of delivery for a full... (116 chars)
# [1] Items must be unused and in original packaging. Refunds are ... (192 chars)
# [2] Gift cards are non-refundable. Standard shipping takes 3 to ... (169 chars)Things to notice: overlap of 40 characters improved fixed-size results (3/7 to 4/7 at hit@1) but still left 3 sentences cut, because overlap only helps when the fact fits inside the shared region. Sentence-aware chunks cut nothing. Enlarging fixed chunks to 400 helped too, but a larger size costs precision in a real corpus with many topics, which this tiny text cannot show. Try changing the queries, the size and the overlap, and add a long document to see the trade-offs move.
Q: Why does chunking matter so much in RAG?
A: The retriever can only return the units you indexed, so chunk boundaries decide what is findable. A fact split across two chunks may be in neither in a form that matches the query, and a chunk that mixes several topics produces a blurred embedding. That is why retrieval quality, not the LLM, is often the ceiling on answer quality.
Q: How would you choose a chunk size?
A: Start from the document structure and the embedding model's input limit, pick a candidate (a few hundred tokens is a common starting point), then measure recall@k and MRR on a labelled query set for several sizes. Smaller chunks raise precision but lose context; larger ones keep context but blur vectors and consume prompt budget. The answer is corpus-specific, so I would not quote a universal number.
Q: What does overlap buy you, and what does it cost?
A: Overlap makes a fact near a boundary appear whole in at least one chunk, which mostly rescues fixed-size splitting. It costs extra vectors (roughly overlap fraction more storage, and a proportional bump in embedding cost) and produces near-duplicate hits that need de-duplication. With sentence- or structure-aware chunking it matters much less.
Q: Explain small-to-big retrieval.
A: You embed and search small units, such as sentences, for precise matches, then return their larger parent section to the LLM for context. It resolves the tension between matching precision and context size. The costs are an extra parent lookup, larger prompts, and possibly repeated parents when several children of the same parent match.
Q: Why can't you mix two embedding models in one index?
A: Each model defines its own vector space, so distances only mean something between vectors from the same model. If the dimensions match nothing errors out, but similarity becomes essentially noise; if they differ, the database rejects the insert. So an index is tied to one model and version, and I store the model id with every vector.
Q: What metadata would you store with each chunk and why?
A: Source and section path for citations, position for reconstructing neighbours, timestamps and version for freshness filters, tenant/ACL for authorization at query time, a content hash to skip unchanged chunks, a parent id for small-to-big, and the embedding-model and chunker versions so a migration can find and replace stale chunks. Forgetting the ACL is the expensive mistake, because filtering after generation is too late.
Q: How do you evaluate a chunking change without running the whole chatbot?
A: Build a labelled set of queries with answer spans or source documents, ingest with each configuration, retrieve top-k and compute recall@k and MRR, labelling by answer text so it survives changes in chunk ids. Only after retrieval looks good do I evaluate faithfulness of generated answers. Separating the two tells me which stage to fix.
Q: Higher-dimensional embeddings are better, so always take the biggest model?
A: Not automatically. Dimension scales memory and search cost linearly (10M chunks at 1,536 dims is about 61 GB of raw float32 before index overhead and replicas), and quality gains are model-dependent with diminishing returns. I would test a shortlist on my own data and consider quantization or a model that supports dimension truncation if cost matters.
Follow-up: You must upgrade the embedding model for a live 10M-chunk index without downtime. How?
A: Build a second index with the new model, backfill it with a rate-limited resumable job while dual-writing new and changed documents, evaluate it against the labelled set, then shift traffic gradually with a feature flag and keep the old index for rollback. The constraints are embedding throughput and cost, double storage during the migration, and never querying an index with the wrong model. Stored raw text and content hashes make the backfill cheap to restart.
Follow-up: Recall is fine in offline tests but users complain about missing answers for product codes. What would you check?
A: Dense embeddings handle rare identifiers poorly, so I would check whether those codes are even in the chunk text (PDF extraction may have dropped them) and then add lexical retrieval such as BM25 in a hybrid search fused with the vector results. I would also verify that the offline query set actually contains identifier-style queries, since offline recall only measures what you put in the test set.
Follow-up: Your corpus has long tables and code samples. How does that change chunking?
A: Character or sentence splitting destroys them, so I would split tables by row groups with the header repeated in each chunk, and code by function or class, keeping the file path and signature as metadata. Sometimes it is better to embed a natural-language summary of a table or snippet while returning the original for the LLM to read; that adds an LLM call at ingest and the summary can be wrong, so I would check it against the source.
In a RAG support chatbot (see Design a RAG-Based Support Chatbot) chunking and embedding are the offline ingestion path, and their choices set the online cost. As an ILLUSTRATIVE estimate: 50,000 documents averaging 4,000 tokens is 200M tokens; chunked at 400 tokens with 10% overlap that is roughly 550,000 chunks. At 1,536 float32 dimensions the raw vectors are about 3.4 GB before index overhead, which fits in memory on one node, while a 100M-chunk corpus at the same dimension is over 600 GB and forces sharding, quantization or a disk-based index. Query-side cost is small (one embedding call plus one ANN search, typically tens of milliseconds, depending on provider and index), but top-k times chunk size drives prompt tokens: k=5 chunks of 400 tokens adds about 2,000 input tokens to every request, which multiplies across traffic. The design also needs an incremental-ingestion pipeline (change detection via content hashes, deletes propagated to the index so removed documents stop being retrievable), per-tenant filtering, an evaluation harness in CI, and a versioned re-index path for model changes.
Every chunking decision buys one thing with another. Smaller chunks buy matching precision at the cost of context and vector count; larger chunks buy context at the cost of blurred embeddings and prompt tokens; overlap buys boundary safety at the cost of storage and duplicates; semantic chunking and LLM-generated context buy coherence at the cost of ingestion compute and new failure modes; higher-dimensional embeddings buy nuance at the cost of memory and latency. The choice is wrong when it was made by convention instead of measurement, or when it is treated as permanent: because chunker and embedding-model versions are baked into the stored data, changing them is a migration with real cost, so record versions in metadata from day one and keep an evaluation set that lets you prove an upgrade is worth it. A pragmatic default is structure-aware or sentence-aware chunks of a few hundred tokens with light overlap, a pinned embedding model, and hybrid retrieval, followed by measurement to see where the corpus needs something different.