A semantic cache stores LLM responses and returns a stored response when a new prompt is close enough in meaning to an earlier one, rather than only when the strings are identical. "How do I reset my password?" and "how can I reset my password" should cost one model call, not two. Interviewers probe it because it looks like an easy cost win and hides a correctness problem: unlike a normal cache, a semantic cache can return a confidently wrong answer to a question that merely looks similar. For a Backend or AI engineer the questions are where the similarity threshold comes from, what goes into the cache key so tenants and prompt versions never leak into each other, what must never be cached, and whether the hit rate actually pays for the extra lookup on every request. Classic cache mechanics (TTL, invalidation, stampedes, eviction) are covered in Redis & Caching and Caching Strategies; this page is about what changes when the key is a meaning instead of a string.
A semantic cache is a shortcut that lets an AI system skip asking the expensive model a question it has essentially answered before, even if this time the question is worded differently. Instead of matching the exact same text, it checks whether a new question means roughly the same thing as one already answered, and if so, hands back the old answer instead of generating a brand new one.
Think of a librarian who keeps a small stack of "questions I've already answered today" next to the reference desk. If someone asks "where's your cookbook section?" and a little later another visitor asks "do you have any recipe books?", the librarian doesn't walk the shelves again — they recognize the two questions mean the same thing and give the same answer from memory. The risk is if a slightly different question just sounds similar: mistaking "where's your cookbook section?" for "where's your comic book section?" would send the wrong visitor to the wrong shelf.
request
|
v
[1] normalise (trim, collapse whitespace, optional lowercase) --> exact key = hash(namespace + normalised prompt)
|
+--> exact cache (Redis GET) HIT --> return cost ~ microseconds to sub-ms
| MISS
v
[2] embed the prompt (small embedding model) cost ~ one embedding call, paid on every miss of [1]
|
v
[3] nearest-neighbour search WITHIN the same namespace cost ~ vector index lookup
| best similarity s
+--> s >= threshold HIT --> return cached answer (risk: FALSE HIT)
| s < threshold
v
[4] call the LLM (seconds, dollars) --> store (embedding, answer) under the namespace with a TTL --> returnAn exact-match cache is cheap, predictable and never wrong: the same key means the same input, so, if the model is deterministic enough for your purpose, the same output. It hits rarely for free-text prompts because humans rephrase, but often for programmatic ones such as fixed templates, repeated system-generated queries or retried requests. A semantic cache raises the hit rate by generalising over phrasing, and pays for it with an embedding call and a vector search on every request that reaches it, plus the risk of returning an answer meant for a different question. That is why the two are layered with the exact cache first: it absorbs the free hits, the semantic layer catches paraphrases, and the model handles the rest. The vector lookup itself is an ordinary similarity-search problem (see Vector Databases and Approximate Nearest Neighbor Search), and at small cache sizes a brute-force scan is fine.
Toy demo (bag-of-words cosine; see Try It Yourself). Two cached questions:
Q1 "how do I reset my password" Q2 "what is the refund policy for annual plans"
incoming query cosine t=0.5 t=0.7 t=0.9
how can I reset my password 1.00 HIT HIT HIT
reset my account password 0.82 HIT HIT miss (lost saving)
how do I reset my password manager 0.82 FALSE-HIT FALSE-HIT miss
what is the refund policy for monthly plans 0.75 FALSE-HIT FALSE-HIT miss
I forgot my login credentials 0.00 miss (lost) miss (lost) miss (lost)
Lower threshold -> more hits, more FALSE hits (wrong answer served)
Higher threshold -> fewer false hits, more misses (money and latency not saved)Every threshold is a point on a precision/recall curve. A false hit (a different intent above the threshold) is a correctness failure the user may never notice; a miss (a same-intent query below the threshold) only costs one model call. Those costs are asymmetric, so you generally bias the threshold high and accept a lower hit rate, and you choose it empirically: label a sample of real query pairs as same or different intent, compute the similarity for each pair, and pick the threshold that meets a target false-hit rate. Two things make this harder than the toy suggests. Similarity scores depend on the embedding model, so the threshold is not portable across models, and changing the embedder invalidates the whole cache. And embeddings often score near-identical sentences that differ in one decisive token (monthly vs annual, cannot vs can, a number, a name) as very similar, so a global threshold cannot fully separate them. Mitigations are per-category thresholds, a stricter threshold for high-stakes intents, verifying the hit with a cheap check (matching entities or numbers extracted from both prompts), or restricting the semantic layer to intents where wrong answers are cheap, such as FAQ-style content.
namespace = ( tenant_id, # data isolation: tenant A must never receive tenant B's answer
model_id + version, # a different model gives a different answer
prompt_template_version, # system prompt / instructions changed -> old answers stale
generation params, # temperature, top_p, max_tokens, response_format, tools enabled
retrieval corpus version / index version, # for RAG: answers depend on the documents
locale / language ) # if the answer text depends on it
exact key = hash(namespace, normalised prompt)
semantic key = the namespace PLUS the vector; similarity search is done only INSIDE one namespace.
What is NOT safe to cache (or must be scoped to a user):
- answers using personal data (account balance, order status, "my" anything) -> key by user, or skip
- prompts containing PII or secrets -> do not store the prompt/answer at all
- anything time-sensitive ("today", "latest", stock price, weather) -> skip or very short TTL
- answers from tool calls / agent runs with side effects -> cache reads, never the action
- high temperature / creative requests where variety is the point -> skip
- safety-sensitive outputs whose policy may change -> short TTL, invalidate on policy updateThe namespace rule is the one that prevents the worst incidents. If the key omits the tenant, one customer's cached answer (possibly containing their data) can be served to another; if it omits the prompt version, a fix to the system prompt appears not to work because old answers keep being returned; if it omits the model, a model upgrade silently changes nothing until entries expire. Put the version fields in the key so a change invalidates naturally instead of requiring a purge, and treat the semantic search as filtered by namespace before ranking, not after. The question "is this response the same for everyone who asks it?" decides cacheability: a public FAQ answer is, a personalised one is not. Anything that includes user-specific context must either include the user in the key (which usually destroys the hit rate, so it is often better not to cache it) or be excluded. Storing prompts and answers in a shared cache also creates a data-retention and privacy surface, so apply the same PII handling and access controls you would to a database; see Guardrails, Content Moderation & Safety.
# ILLUSTRATIVE economics of a semantic cache. Every number below is an ASSUMPTION, not a measurement.
req_per_day = 1_000_000
llm_cost = 0.0020 # assumed $ per uncached LLM call
llm_latency = 2.0 # assumed seconds per uncached LLM call
lookup_cost = 0.00002 # assumed $ per request for embedding + vector lookup (paid on EVERY request)
lookup_latency= 0.05 # assumed seconds for embed + lookup (paid on EVERY request)
print("break-even hit rate (cost only) =", round(lookup_cost / llm_cost, 3))
print(f"{'hit rate':>8s} {'$/day no cache':>15s} {'$/day cached':>13s} {'saved $/day':>12s} {'avg latency s':>14s}")
for h in (0.05, 0.20, 0.40):
base = req_per_day * llm_cost
cached = req_per_day * lookup_cost + req_per_day * (1 - h) * llm_cost
lat = lookup_latency + (1 - h) * llm_latency
print(f"{h:8.2f} {base:15,.0f} {cached:13,.0f} {base-cached:12,.0f} {lat:14.2f}")
f = 0.01
print(f"\nif 1% of the 20% hits are false hits: {req_per_day*0.20*f:,.0f} wrong answers/day served confidently")
# break-even hit rate (cost only) = 0.01
# hit rate $/day no cache $/day cached saved $/day avg latency s
# 0.05 2,000 1,920 80 1.95
# 0.20 2,000 1,620 380 1.65
# 0.40 2,000 1,220 780 1.25
#
# if 1% of the 20% hits are false hits: 2,000 wrong answers/day served confidentlyThe economics are simple, which is why they are worth stating as arithmetic. The lookup (embedding plus search) is paid on every request that reaches the semantic layer, while the saving is only earned on hits, so the break-even hit rate is roughly lookup cost divided by model cost. With the assumed numbers that is about 1%, so a semantic cache usually pays for itself on cost when the embedding step is much cheaper than the generation it replaces; the bigger benefit is latency, since a hit replaces a multi-second generation with a sub-second lookup. The figures are assumptions: real token prices, model latencies and achievable hit rates vary widely by workload and provider, so measure your own hit rate on production-like traffic before promising savings. The line at the bottom is the part that is easy to forget: a 1% false-hit rate on 200,000 daily hits is 2,000 confidently wrong answers a day, and that cost does not appear on the cloud bill.
Freshness is handled with TTLs and versioned keys, and the same trade-off as any cache applies. A short TTL bounds staleness but lowers the hit rate; a long TTL raises the hit rate but serves out-of-date answers after the underlying documents, policies or prices change. For RAG systems, include the index or corpus version in the key so a re-index invalidates related answers, or attach a tag to each entry (the source document ids) so you can delete every entry that depended on a changed document. Eviction is usually LRU or LFU with a size cap. Cache the negative case sparingly: caching a refusal or an error response can lock in a transient failure, so do not cache errors, timeouts or truncated outputs.
# Toy semantic cache: bag-of-words cosine similarity, namespaced keys, threshold sweep.
import math, re
from collections import Counter
STOP = {"how", "do", "i", "can", "my", "the", "a", "an", "is", "for", "what", "to", "of"}
def embed(text): # "embedding" = term-frequency vector (no learned semantics)
return Counter(w for w in re.findall(r"[a-z]+", text.lower()) if w not in STOP)
def cosine(a, b):
dot = sum(a[t] * b[t] for t in a)
na = math.sqrt(sum(v * v for v in a.values())); nb = math.sqrt(sum(v * v for v in b.values()))
return dot / (na * nb) if na and nb else 0.0
class SemanticCache:
def __init__(self, threshold):
self.threshold, self.entries = threshold, [] # (ns, vector, query, answer)
def key_ns(self, tenant, model, prompt_v, temperature):
return (tenant, model, prompt_v, temperature) # anything that changes the answer
def put(self, ns, query, answer):
self.entries.append((ns, embed(query), query, answer))
def get(self, ns, query):
q, best, best_sim = embed(query), None, 0.0
for ens, vec, cq, ans in self.entries:
if ens != ns: continue # never cross namespaces
s = cosine(q, vec)
if s > best_sim: best, best_sim = (cq, ans), s
return (best, best_sim) if best and best_sim >= self.threshold else (None, best_sim)
NS = ("acme", "model-x", "prompt-v3", 0.0)
SEED = [("how do I reset my password", "ANSWER: use Settings > Security > Reset"),
("what is the refund policy for annual plans", "ANSWER: 30-day refund on annual plans")]
# (incoming query, index of seed with the SAME intent, or None if it needs a fresh answer)
TESTS = [("how do I reset my password", 0),
("how can I reset my password", 0),
("reset my account password", 0),
("how do I reset my password manager", None), # different intent
("what is the refund policy for monthly plans", None), # different answer!
("I forgot my login credentials", 0)] # same intent, no shared words
print(f"{'query':46s} {'sim':>5s} " + " ".join(f"t={t}" for t in (0.5, 0.7, 0.9)))
sims = {}
for q, want in TESTS:
row = []
for t in (0.5, 0.7, 0.9):
c = SemanticCache(t)
for sq, sa in SEED: c.put(NS, sq, sa)
hit, sim = c.get(NS, q)
sims[q] = sim
if hit is None:
row.append("MISS" if want is None else "MISS(lost saving)")
else:
correct = want is not None and hit[0] == SEED[want][0]
row.append("HIT" if correct else "FALSE-HIT")
print(f"{q:46s} {sims[q]:5.2f} " + " ".join(f"{r:>17s}" for r in row))
print()
c = SemanticCache(0.7)
for sq, sa in SEED: c.put(NS, sq, sa)
other_tenant = ("globex", "model-x", "prompt-v3", 0.0)
new_prompt = ("acme", "model-x", "prompt-v4", 0.0)
print("same query, same namespace :", c.get(NS, "how do I reset my password")[0] is not None)
print("same query, other tenant :", c.get(other_tenant, "how do I reset my password")[0] is not None)
print("same query, new prompt version :", c.get(new_prompt, "how do I reset my password")[0] is not None)
# Real output from python3:
# query sim t=0.5 t=0.7 t=0.9
# how do I reset my password 1.00 HIT HIT HIT
# how can I reset my password 1.00 HIT HIT HIT
# reset my account password 0.82 HIT HIT MISS(lost saving)
# how do I reset my password manager 0.82 FALSE-HIT FALSE-HIT MISS
# what is the refund policy for monthly plans 0.75 FALSE-HIT FALSE-HIT MISS
# I forgot my login credentials 0.00 MISS(lost saving) MISS(lost saving) MISS(lost saving)
#
# same query, same namespace : True
# same query, other tenant : False
# same query, new prompt version : False
# What to notice:
# - "how do I reset my password manager" and "...refund policy for monthly plans" score as similar
# to a cached question but need a DIFFERENT answer: a low threshold turns them into FALSE-HITs.
# - at t=0.9 the false hits disappear, but the true paraphrase "reset my account password" is also lost.
# - "I forgot my login credentials" has the same intent but shares no words, so word overlap
# scores 0.00. A real embedding model would score it high; bag-of-words cannot. This toy shows
# the THRESHOLD mechanics, not the quality of a learned embedding.
# - the namespace check means another tenant, or a new prompt version, never reads the entry.Q: What is a semantic cache and when does it beat an exact-match cache?
A: It stores prompt embeddings with their responses and returns a stored response when a new prompt's embedding is similar enough. It wins when many users ask the same thing in different words, such as FAQ and support traffic, where exact-match hit rates are low. It loses when prompts are unique, personalised or contain long variable context, because the hit rate is small while the embedding and search cost is paid on every request.
Q: How do you choose the similarity threshold?
A: Empirically, per embedding model. Sample real queries, label pairs as same or different intent, compute similarity, and pick the threshold that meets a target false-hit rate, then look at the hit rate you can afford at that point. Because a false hit is a wrong answer and a miss is only one extra model call, I bias high, and revisit it whenever the embedder, domain or traffic mix changes.
Q: What must be in the cache key besides the prompt?
A: Tenant, model and version, prompt-template version, generation parameters (temperature, max tokens, response format, tools), and for RAG the corpus or index version. Each of these changes the correct answer or the audience of the answer, so omitting one causes cross-tenant leaks or stale behaviour after a deploy. Similarity search is then restricted to that namespace.
Q: What should you never cache?
A: Responses that depend on a specific user's data, prompts with PII or secrets, time-sensitive answers, outputs of tool calls with side effects, errors and truncated generations, and high-temperature creative output. For personal data you either key by user, which usually kills the hit rate, or skip caching. The general test is whether the answer is the same for everyone who asks and stays correct for the TTL.
Q: Is a semantic cache always worth it? How would you decide?
A: Compare the break-even hit rate (lookup cost over model cost) with the measured hit rate, and also weigh the latency gain and the false-hit risk. If lookup is about 1% of a model call and you get a 20% hit rate, the saving is real; if hits are 2% and the intent is high-stakes, it is not worth the risk. Run it in shadow mode first: compute would-be hits, sample them and have reviewers judge whether the cached answer was acceptable.
Q: How do you invalidate a semantic cache when the source documents change?
A: Version the corpus or index in the namespace so a re-index invalidates everything at once, and for finer control tag each entry with the ids of the documents used so a document update can delete exactly those entries. A TTL is the backstop for anything missed. The trade-off is that broad invalidation is simple but throws away good entries, while tagging needs the RAG pipeline to report which documents contributed to each answer.
Q: How do exact-match and semantic layers interact with a cache stampede?
A: Semantic caches are exposed to the same problem: when a popular entry expires many concurrent requests miss and call the model. The same defences apply: single-flight per namespace and prompt so one request generates while others wait, TTL jitter, and serving a stale entry while refreshing. Model calls are slow and expensive, so a stampede here costs money and rate-limit budget as well as latency.
Follow-up: How would you detect that the cache is returning wrong answers in production?
A: Log every hit with the incoming prompt, matched prompt, similarity score and namespace, then sample hits for human or LLM-judge review and track a false-hit rate. Watch the similarity score distribution: a pile of hits just above the threshold is where errors live. Also expose user feedback (thumbs down) joined against whether the answer was cached, and keep a kill switch or per-intent disable so a bad category can be turned off without redeploying.
Follow-up: A single tenant complains they saw another tenant's data in an answer. What do you check?
A: First whether the tenant id is in the namespace and whether the vector search filter is applied before ranking, since a post-filter on top-k can also lose valid results but a missing filter leaks data. Then check whether prompts or contexts containing tenant data were stored in a shared namespace, whether a normalisation step dropped the tenant field, and whether the exact-cache key hashes the same fields. Purge the affected entries, add a test that asserts cross-tenant reads miss, and treat it as a security incident.
Follow-up: Could you cache at the provider level instead? How is that different?
A: Some providers offer prompt (prefix) caching, which reuses the computation for a repeated prompt prefix to cut cost and latency of the input side, but the model still generates a fresh response each time. A semantic or response cache skips generation entirely and is your responsibility. They are complementary, and the details and pricing vary by provider, so check current documentation.
In an LLM gateway or a RAG support chatbot the semantic cache sits in the gateway, after authentication and rate limiting and before the provider call (see Design an LLM API Gateway and Cost Optimization for AI Systems). Stating assumptions, if the service handles 1M requests per day with an average of 12 requests per second and perhaps 10 times that at peak, and the FAQ-like share of traffic is what the cache can serve, a 20% hit rate removes 200,000 model calls a day and lowers the average latency roughly in proportion, as the arithmetic above shows, while the same 20% is a burst-absorbing buffer if the provider rate-limits you. The exact layer can live in Redis (Redis & Caching) with TTL and jitter, while the semantic layer needs a vector index scoped by namespace, either a vector database or Redis with a vector module, sized by entries times embedding dimensions times bytes (for example 1M entries of 768 float32 dimensions is about 3 GB before index overhead, which is an estimate to be checked against the store you choose). Design the namespace, invalidation on prompt or corpus change, per-tenant enable and disable, PII exclusion, and the metrics (hit rate, false-hit sample rate, lookup latency) explicitly, and remember that the cache is a derived optimisation: the system must remain correct, only slower and costlier, when the cache is empty or turned off. Related building blocks are Chunking & Embedding Strategies for the embedding side and Prompt Engineering & Prompt Management for versioning prompts that feed the key.
The decision is to spend an embedding call and a vector search on every request in exchange for skipping some model calls. You gain lower cost, lower average latency and protection against provider rate limits, and you pay in added infrastructure, a threshold that must be tuned and re-tuned, a new privacy and tenancy surface, and a failure mode where a wrong answer is returned instantly and confidently. It is the wrong choice when queries are mostly unique or personalised, when answers depend on fresh or per-user data, when the cost of one wrong answer exceeds the savings (medical, legal, financial guidance), when the traffic is too small for the hit rate to cover the lookup, or when an exact-match cache plus a prompt-prefix cache already captures most of the benefit. Start with exact-match and provider prefix caching, add semantic caching only for the intents where measurement shows a good hit rate and reviewed false-hit rate is acceptable, and keep the ability to disable it per tenant or per intent.