Caching means keeping the result of expensive work close to where it is needed so the next request for the same thing does not repeat the work. It exists because reads are usually far more frequent than writes and because access is usually skewed: a small set of keys receives most of the traffic. Interviewers probe it because "add a cache" is the reflex answer to almost every latency or load problem, and a strong candidate can say which layer, which write policy, how stale the data may get, what happens when the cache is cold, and how to know it is helping. For a Backend or AI engineer the stakes are higher than usual, because a miss can mean a GPU-second and real money, not just a database round trip.
This page is the AI-backend view. The mechanics of Redis, cache-aside code and stampede locks are on Redis & Caching; matching by meaning rather than exact text is in Semantic Caching for LLMs; the edge layer is in CDN. Here the focus is how the layers fit together, how eviction really behaves on a skewed workload, and what is different when the cached thing is an embedding, a model output or an attention cache.
Caching is saving the answer to something the first time it's worked out, so the next time it's needed, the system can just hand back the saved answer instead of redoing all the work. It trades a bit of extra storage for a lot of saved time, and it works well because the same handful of things tend to get asked for again and again.
Think of a librarian who keeps the ten most-requested books on a cart right by the front desk, instead of walking to the back shelves every time someone asks for one. Fetching from the cart is nearly instant; if a book isn't on the cart, the librarian still has to walk to the shelves to get it, but the popular ones almost never require that walk.
client / browser -> CDN / edge -> app-process cache -> shared cache (Redis) -> database
(HTTP cache, (static + (in-memory map, (network hop, shared (buffer pool,
app memory) cacheable no network, per across instances, query cache
API GETs) instance) holds derived data) varies by engine)
closer to user = faster + cheaper to serve, but harder to invalidate and less shared
closer to source = fresher + easier to invalidate, but every miss goes furtherEach layer answers a different question. The browser or mobile client cache (HTTP Cache-Control, an in-app store) saves the whole network trip. The CDN serves cacheable responses from a nearby point of presence and absorbs traffic before it reaches your origin. An in-process cache (a plain map or a library such as Caffeine or functools.lru_cache) costs no network hop but is per instance, so N instances hold N copies and invalidating one does not invalidate the others. A shared cache such as Redis is one consistent copy for the fleet at the cost of a network round trip and an extra system to operate. The database has its own caches (for example the buffer pool), which you benefit from without asking.
The rule of thumb is to cache as close to the consumer as staleness allows, and to be honest that every layer you add multiplies the invalidation problem: a value can be stale in the browser, the CDN, the process and Redis at once. That is why the layers nearest the user are used for data that changes rarely or tolerates a version in the URL, and why writes usually invalidate only the shared layer plus a short TTL elsewhere. An Android developer will recognise the client layer immediately: an ETag plus If-None-Match turns a full download into a tiny 304.
cache-aside app owns the logic read : cache miss -> app reads DB -> app fills cache
write: app writes DB -> app deletes (invalidates) key
read-through cache owns the load read : cache miss -> cache itself loads from DB -> returns
(app only ever talks to the cache; same idea, different owner)
write-through writes hit both write: app -> cache -> DB, ack only after both succeed
write-back cache buffers writes write: app -> cache -> ack; cache flushes to DB later (batched)
(fast, but unflushed data is lost if the cache node dies)Read-through and cache-aside are the same behaviour with a different owner: in cache-aside the application code loads on a miss, in read-through the cache component does it, which centralises the loading logic and makes single-flight easier to add. Write-through pays double write latency to keep the cache fresh for keys about to be read; write-back trades durability for write speed and batching, which suits counters and metrics you can afford to lose a little of, and is wrong for orders or balances. Most production reads use cache-aside because failure of the cache degrades to slower reads, never wrong ones. The detailed race between a delayed reader and a writer, and why you delete rather than update, is worked through on the Redis & Caching page.
Real traffic is skewed (roughly Zipf-like): a few keys get most reads.
hit rate of an LRU cache, 10,000 keys, 100,000 requests, seeded run (see Try It Yourself):
cache holds 1% of keys -> ~39% hits (skewed) vs ~1% hits (uniform)
cache holds 10% of keys -> ~67% hits (skewed) vs ~10% hits (uniform)
LRU: evict the least RECENTLY used (recency signal; cheap: hash map + linked list)
LFU: evict the least FREQUENTLY used (frequency signal; resists scans, but old popular
keys can linger after they go cold unless counts decay)Eviction only matters when the cache is smaller than the working set, which is the normal case. The measurement above shows why caching works at all: on a skewed workload a cache holding a tenth of the keys serves about two thirds of the requests, but on a uniform workload the hit rate is roughly the fraction of keys you can hold, so a cache is nearly useless. Before adding one, look at the access distribution. LRU is the sensible default because recency predicts reuse for most traffic. Its weakness is scan pollution: a burst of one-off keys (a report, a crawler, a batch job) pushes the genuinely hot keys out. LFU resists that because one-off keys have a count of one, but plain LFU needs frequency aging so a key that was popular last week does not occupy space forever. Redis, for example, offers LRU and LFU approximations, and the exact sampled behaviour depends on version and configuration.
Average latency = h * t_hit + (1 - h) * (t_hit + t_miss) ILLUSTRATIVE numbers:
assume t_hit = 1 ms (Redis round trip), t_miss = 20 ms (database query)
h = 50% -> 0.5 * 1 + 0.5 * 21 = 11.0 ms
h = 90% -> 0.9 * 1 + 0.1 * 21 = 3.0 ms
h = 99% -> 0.99 * 1 + 0.01 * 21 = 1.2 ms
Hot key: one key gets a huge share of traffic (a celebrity profile, a trending prompt).
A single Redis node/shard serving it becomes the bottleneck even if the cluster is idle.
Fixes: in-process near-cache with a short TTL, replicate the key under N suffixes
(key#0 .. key#7) and read a random one, or push it to the CDN.Two problems dominate at scale. The first is the hot key: hashing distributes keys evenly, but it cannot split one key, so one shard takes all the traffic for it. The standard remedies are a small local cache in front of the shared one, or key replication with random suffixes at the cost of invalidating N copies. The second is invalidation: TTL is simple and self-healing but serves stale data up to its length; explicit delete-on-write is fresher but needs every write path to remember it; versioned keys (put a version or content hash in the key) avoid invalidation entirely by making stale entries unreachable and letting eviction clean them up. Stampedes, where a hot key expires and many callers rebuild it together, are covered with single-flight and jitter on the Redis page. Because a miss on an AI request is far more costly than a database read, the stampede defence matters more there, not less.
What you can cache Key Safe when
--------------------------------------------------------------------------------------------
embedding of a text chunk / query hash(embedding-model-version, text) deterministic model; text unchanged
final LLM answer (exact match) hash(model, prompt, params, tools) temperature 0 or acceptable variance
final LLM answer (semantic match) nearest-neighbour over query vectors threshold tuned; low-risk domain
retrieval results / rerank output hash(index-version, query, filters) index version in the key; short TTL
tool / API call results hash(tool, args) tool is read-only, staleness OK
Provider-level (inside the model server, not your Redis):
prompt / prefix caching : the provider reuses work already done for an identical prompt PREFIX
KV-cache reuse : the attention key/value tensors of a shared prefix are kept, so the
prefill phase for that prefix is skipped (e.g. vLLM automatic prefix caching)Two families of cache exist here and they behave differently. Application caches sit in your Redis and return a stored result without calling the model at all, so a hit is nearly free, but correctness is your problem: the key must include everything that changes the answer (model version, prompt template version, sampling parameters, tool list, and any user or tenant context that leaks information), and non-deterministic outputs at high temperature mean a cached answer is one sample, not the answer. Embedding caches are the easiest win because embeddings of unchanged text are deterministic for a fixed model; include the embedding model version in the key so an upgrade cannot mix incompatible vectors.
Provider and serving-layer caching works underneath your code. Many providers and serving engines can reuse the computation for an identical prompt prefix (system prompt, tool definitions, a long document placed first), which reduces prefill cost and time to first token; some providers price cached input tokens differently, and the rules (minimum prefix length, lifetime, whether you must mark cache points, discounts) differ by provider and change often, so check current docs rather than memorising numbers. The design consequence is prompt layout: put stable content first and variable content last, or the shared prefix is too short to hit. This is a KV cache reuse, distinct from your application cache; see LLM Inference Fundamentals for why prefill and decode differ, and Cost Optimization for where the savings land.
# Python 3 (stdlib only): LRU cache and its hit rate on a seeded skewed workload
import random
from collections import OrderedDict
class LRUCache:
def __init__(self, capacity):
self.capacity, self.data = capacity, OrderedDict()
self.hits = self.misses = 0
def get(self, key):
if key in self.data:
self.data.move_to_end(key) # mark most recently used
self.hits += 1
return self.data[key]
self.misses += 1
return None
def put(self, key, value):
self.data[key] = value
self.data.move_to_end(key)
if len(self.data) > self.capacity:
self.data.popitem(last=False) # evict least recently used
@property
def hit_rate(self):
n = self.hits + self.misses
return self.hits / n if n else 0.0
def zipf_workload(n_keys, n_requests, s, seed):
rng = random.Random(seed)
weights = [1 / (rank ** s) for rank in range(1, n_keys + 1)]
return rng.choices(range(n_keys), weights=weights, k=n_requests)
def uniform_workload(n_keys, n_requests, seed):
rng = random.Random(seed)
return [rng.randrange(n_keys) for _ in range(n_requests)]
def run(workload, capacity):
c = LRUCache(capacity)
for k in workload:
if c.get(k) is None:
c.put(k, "value-%d" % k) # cache-aside: fill on miss
return c.hit_rate
N_KEYS, N_REQ = 10_000, 100_000
skew = zipf_workload(N_KEYS, N_REQ, s=1.0, seed=42)
flat = uniform_workload(N_KEYS, N_REQ, seed=42)
print("keys=%d requests=%d" % (N_KEYS, N_REQ))
print("cache size (% of keys) | skewed hit rate | uniform hit rate")
for cap in (50, 100, 500, 1000, 2000):
print("%6d (%4.1f%%) | %14.1f%% | %15.1f%%" % (
cap, 100 * cap / N_KEYS, 100 * run(skew, cap), 100 * run(flat, cap)))
# Real output from python3:
# keys=10000 requests=100000
# cache size (% of keys) | skewed hit rate | uniform hit rate
# 50 ( 0.5%) | 30.9% | 0.5%
# 100 ( 1.0%) | 38.9% | 1.0%
# 500 ( 5.0%) | 58.2% | 4.9%
# 1000 (10.0%) | 67.1% | 9.9%
# 2000 (20.0%) | 76.1% | 19.9%
# Python 3: LRU vs LFU when one-off keys are interleaved with hot keys (scan pollution)
from collections import OrderedDict
def lru_run(seq, cap):
d, hits = OrderedDict(), 0
for k in seq:
if k in d: d.move_to_end(k); hits += 1
else:
d[k] = 1
if len(d) > cap: d.popitem(last=False)
return hits / len(seq)
def lfu_run(seq, cap):
freq, hits = {}, 0
for k in seq:
if k in freq: freq[k] += 1; hits += 1
else:
if len(freq) >= cap:
victim = min(freq, key=freq.get) # O(n) here; real LFUs use O(1) structures
del freq[victim]
freq[k] = 1
return hits / len(seq)
rng = random.Random(1)
seq, nxt = [], 1000
for _ in range(3000):
seq.append(rng.randrange(15)) # hot read among 15 hot keys
for _ in range(2): # interleaved one-off keys (never reused)
seq.append(nxt); nxt += 1
print("cache size 20 | 3000 hot reads (15 hot keys), each followed by 2 never-repeated keys")
print("LRU hit rate: %.1f%%" % (100 * lru_run(seq, 20)))
print("LFU hit rate: %.1f%%" % (100 * lfu_run(seq, 20)))
# Real output from python3 (the best possible hit rate here is 3000/9000 = 33.3%):
# cache size 20 | 3000 hot reads (15 hot keys), each followed by 2 never-repeated keys
# LRU hit rate: 11.9%
# LFU hit rate: 32.7%
# Python 3: a deterministic cache key for an LLM call or embedding
import hashlib, json
def cache_key(model, prompt, params):
norm = " ".join(prompt.split()) # collapse whitespace; do NOT lowercase blindly
payload = json.dumps({"m": model, "p": norm, "params": params}, sort_keys=True)
return "llm:v1:" + hashlib.sha256(payload.encode()).hexdigest()[:16]
a = cache_key("model-x-2025-01", "Summarize this text", {"temperature": 0})
b = cache_key("model-x-2025-01", "Summarize this text", {"temperature": 0})
c = cache_key("model-x-2025-01", "Summarize this text", {"temperature": 0.7})
d = cache_key("model-x-2025-06", "Summarize this text", {"temperature": 0})
print("a == b (whitespace only differs):", a == b)
print("a == c (temperature differs) :", a == c)
print("a == d (model version differs) :", a == d)
print(a)
# Real output from python3:
# a == b (whitespace only differs): True
# a == c (temperature differs) : False
# a == d (model version differs) : False
# llm:v1:dbce3a45e4d1d709Q: Where would you put caches in a read-heavy service, and why not just one big Redis?
A: Layer them: HTTP caching on the client and a CDN for cacheable GETs, a small in-process cache for the hottest keys, Redis as the shared layer, and the database's own buffer pool underneath. Each layer removes a different cost (network trip, origin load, Redis hop, disk read). A single Redis puts a network round trip on every read, becomes the hot-key bottleneck for viral keys, and is one more thing that can take the whole service down, whereas an in-process layer with a 1-5 second TTL absorbs the hottest keys with no hop at all, at the price of slightly staler data.
Q: Cache-aside versus read-through versus write-through: how do you choose?
A: Cache-aside is the default because the application controls the logic and a dead cache only means slower reads. Read-through is the same behaviour moved into the cache component, useful when you want one place to add single-flight, metrics and loading rules. Write-through fits data that is read soon after it is written and where you want few misses, but every write pays for two stores and it caches data that may never be read. I would avoid write-back for anything you cannot afford to lose, since the buffered writes vanish if the cache node dies before flushing.
Q: LRU or LFU?
A: LRU is the default because recency predicts reuse for most traffic and it is cheap to implement, but it is vulnerable to scans of one-off keys. LFU keeps the genuinely popular keys through such a scan, which in my simulation of hot keys mixed with never-repeated keys kept a hit rate near the ceiling where LRU fell to about a third of it. Its cost is memory for counters and the need to age frequencies, otherwise last week's popular key sits in the cache forever. If the workload is unknown, start with LRU and measure; if you have a known scan-heavy pattern, consider LFU or an admission policy.
Q: A cache is not improving latency. How do you diagnose it?
A: First measure hit rate per key class, not just overall, then compare the miss cost with the hit cost, because latency is the hit-rate-weighted average of the two. Common causes are an access pattern that is nearly uniform (nothing to cache), keys that include a per-request field such as a timestamp or request id so they never repeat, a TTL shorter than the reuse interval, a cache too small for the working set so entries are evicted before they are reused, or a key built without normalisation so equivalent inputs miss. The fix depends on which one it is, and sometimes the answer is that the real problem is a missing database index.
Q: What can you cache in an LLM application, and what must the key contain?
A: Embeddings of unchanged text, retrieval results, tool call results, and deterministic model outputs; a semantic cache can also answer paraphrases at some risk of a wrong match. The key must include everything that changes the answer: model and version, prompt-template version, sampling parameters, tool definitions, retrieval index version and any tenant or user scope, otherwise you serve another user's data or an answer from a retired model. At non-zero temperature a cached response is one sample, so caching changes the behaviour users see and you should say that explicitly.
Q: What is prompt caching at the provider level and how does it differ from your Redis cache?
A: Generally the provider or serving engine reuses the already computed attention key/value state for an identical prompt prefix, so the prefill work for that prefix is skipped; you still make the call and still get a fresh generation. Your Redis cache instead returns a stored response and skips the model entirely. The provider mechanism therefore lowers cost and time to first token but not decode cost, and it requires stable content at the front of the prompt. The exact rules, lifetimes and pricing differ by provider, so verify them in current documentation.
Follow-up: A single key receives 40% of all traffic and one Redis shard is saturated. What do you do?
A: Hashing cannot split one key, so I would add a short-TTL in-process cache on every app instance so most reads never reach Redis, and if that is not enough replicate the key under several suffixes and read a random one, accepting that invalidation must touch every copy. If the value is cacheable HTTP content I would push it to the CDN. I would also confirm it is really a hot key and not a slow command blocking the shard.
Follow-up: How do you invalidate cached LLM answers when you change the system prompt or upgrade the model?
A: Put the model version and a prompt-template version in the key, so the change makes old entries unreachable without a mass delete, and let TTL and eviction remove them. The cost is a temporary drop in hit rate and a burst of misses after the change, so warm the highest-traffic keys or roll out gradually and watch the cost. Deleting by pattern in a large cache is slow and risky, which is another reason to prefer versioned keys.
Follow-up: Would you cache a user's personal data in a shared cache?
A: Only with the user or tenant id in the key and an access check before the lookup, because a key that omits the scope can return one user's data to another, which is a security incident, not a cache bug. I would also keep TTLs short for sensitive data and never put personal data in a CDN response without private cache headers. This is a frequent real-world bug with semantic caches, where a similar query from a different tenant matches.
In a design interview, specify the whole caching policy rather than saying "add Redis": which layers, the key structure, TTL with jitter, the write and invalidation path, stampede and hot-key handling, the eviction policy and what happens when the cache is cold or down. Do the arithmetic aloud: for an ILLUSTRATIVE service at 10,000 reads per second with a 90% hit rate, the database sees 1,000 reads per second instead of 10,000, and a cache that dies takes that back to 10,000 at once, so the database has to survive the cold cache or you need load shedding and a circuit breaker. For a RAG or chatbot backend, layer an embedding cache (deterministic, near-free to reuse), a retrieval cache keyed by index version, an exact-match response cache for repeated deterministic prompts, optionally a semantic cache with a strict threshold for low-risk domains, and provider prompt caching by placing the long system prompt and documents at the front. Assume, say, that 30% of requests repeat an exact prompt and an average answer costs 1 cent of model spend: an exact cache then avoids about 0.3 cents per request on average, which is worth having but is smaller than what routing to a cheaper model might save, so compare options by measured hit rate and cost per miss. Mention the related LLM gateway design, where the cache typically lives.
A cache trades correctness and simplicity for latency, throughput and cost: you accept stale reads, one more system to operate, a new failure mode (cold or dead cache) and a hit rate you must keep measuring. Cache-aside is the safe default, write-through and write-back move cost and risk to the write path, TTLs bound staleness at the price of extra misses, and versioned keys avoid invalidation at the price of memory held by unreachable entries until they are evicted. For AI workloads the trade-off is sharper: exact caches are safe but hit only on repeats, semantic caches hit more but can return a wrong answer, and non-deterministic outputs make any cache change what users experience. Caching is the wrong choice when data changes about as often as it is read, when reads must reflect the latest committed state (balances, inventory decisions, permissions), when access is near uniform so hit rates stay low, when responses are personalised so nothing is shared, or when the real fix is a missing index or a cheaper model. Measure hit rate, miss cost and the load on the origin with and without the cache before claiming it helps.