A real-time recommendation system decides, within roughly a hundred milliseconds of a user opening a screen, which handful of items out of millions to show, using both what is known about the user's long-term taste and what they did a minute ago. Interviewers probe it because it is where ML and backend engineering meet: the problem is not "train a model", it is a multi-stage serving pipeline with a strict latency budget, two feature paths (offline and online) that must agree, cold-start cases where there is no data, and a feedback loop where the system's own outputs become its future training data. For a Backend or AI engineer, the useful skill is reasoning about where the time, the money and the silent failures (training-serving skew, feedback loops) are.
Related pages: approximate nearest neighbor search (candidate retrieval), feature stores, A/B testing and canary deployments for models, and model monitoring and drift detection. This page is the end-to-end case study that ties them together.
A real-time recommendation system is the part of an app that picks which items to show you right now, out of far more items than could ever fit on a screen. It combines what the app already knows about your long-term taste (what you've liked or bought before) with what you just did seconds ago (what you scrolled past, clicked, or skipped), and returns a short, ranked list fast enough that the screen doesn't feel like it's waiting on anything.
Picture a good waiter at a restaurant you visit often. They remember you usually order fish, but they also notice tonight you seem to be in a rush and glanced at the sandwich menu. Instead of reciting the entire menu, they quickly suggest two or three dishes that fit both what they know about you and what they're noticing right now. A recommendation system is doing the same thing, just across millions of "diners" at once and in a fraction of a second.
All numbers here are ILLUSTRATIVE assumptions to agree with the interviewer, not measurements of any real product.
FUNCTIONAL
F1 Given (user, context) return a ranked list of N items (say N = 20) for a surface:
home feed, "up next", or "similar items".
F2 Personalise from long-term history (weeks/months) AND recent activity (this session).
F3 Handle cold start: brand-new users and brand-new items must still get sensible results.
F4 Apply business rules: filter already-seen / blocked / unavailable / age-restricted items.
F5 Support experimentation: multiple model versions live at once, users assigned to arms.
F6 Log every served list (what was shown, in what order, with which features and model)
so it can be joined with outcomes for training and evaluation.
F7 Update models and indexes without downtime.
NON-FUNCTIONAL (assumed targets)
N1 End-to-end p50 ~ 80 ms, p99 ~ 200 ms (server-side, illustrative).
N2 Availability 99.9%+: if ML components fail, return a degraded but valid list, never an error page.
N3 Feature freshness: recent-activity features visible to ranking within seconds (streaming path);
long-term features may be hours to a day old (batch path).
N4 Training-serving consistency: a feature must mean the same thing offline and online.
N5 Cost: bounded per-request compute; the model with the best offline metric is not automatically affordable.
OUT OF SCOPE (say so): content ingestion/moderation, ad auctions, the training platform internals,
notification/email recommendations (same ideas, different latency).ASSUMPTIONS (ILLUSTRATIVE)
100M registered users, 20M daily active, 5M items in the catalog
each DAU opens the recommendation surface 10x/day, generates ~50 events/day (click, view, like, skip)
peak-to-average 3x, N = 20 items shown per request
TRAFFIC
requests/day = 20M x 10 = 200M avg = 200M / 86,400 ~ 2,300 rps peak ~ 7,000 rps
events/day = 20M x 50 = 1B avg ~ 11,600 events/s peak ~ 35,000 events/s
FUNNEL (where the compute goes)
catalog 5,000,000 -> candidate generation ~ 500-1,000 -> ranking model scores ~ 500 -> re-rank -> show 20
ranker cost per request = candidates x cost-per-candidate; 500 x (say) 0.05 ms = ~25 ms CPU-equivalent
(real cost depends entirely on model type and hardware: measure it)
FEATURE STORE
user features : 100M x ~2 KB = ~200 GB (sharded in-memory KV or SSD-backed KV)
item features : 5M x ~1 KB = ~5 GB (small enough to cache on every ranker node)
item embeddings: 5M x 128 dims x 4 B = ~2.6 GB float32 (ANN index adds overhead; quantisation can cut it)
online reads per request ~ 1 user key + ~500 item keys (batched multi-get, mostly local cache hits)
-> 7,000 rps x 1 user read = 7,000 user-store reads/s at peak; item reads mostly served locally
LOGGING
impressions/day = 200M x 20 = 4B rows; at ~300 B per row (ids, position, score, feature snapshot ref,
model + experiment ids) ~ 1.2 TB/day -> partition by day, compress, keep hot for weeks only
SERVING FLEET
if one node handles ~150 rps within the latency budget (assumption), peak 7,000 rps needs ~47 nodes;
provision N+30% and multiple zones ~ 60-70 nodes. Ranking dominates CPU; ANN and feature reads dominate memory.The conclusion to say aloud: the read path is a small number of milliseconds per stage but a large fan-out, so the design is dominated by latency budgeting, memory-resident indexes and features, and the daily log volume, not by request counts alone.
ONLINE SERVING PATH (target ~100-200 ms)
client -> API gateway -> RECOMMENDATION SERVICE (stateless)
|
1. fetch user profile + recent events --------------> ONLINE FEATURE STORE (KV, in-memory)
2. candidate generation, IN PARALLEL: ^ ^
a ANN on user embedding --------------------> vector index | |
b item-item co-occurrence / CF ------------> precomputed neighbours | |
c trending / popularity ------------------> cached lists | |
d "because you did X" (recent-item seeds) | |
3. merge + dedupe + filter (seen, blocked, unavailable) | |
4. fetch item features (local cache first) ------------------------------+ |
5. RANKING MODEL scores ~500 candidates (features from step 1 + 4) |
6. RE-RANK: diversity, freshness boost, exploration slots, business rules |
7. return top N, LOG the impression (ids, positions, scores, features/model/experiment) |
|
NEAR-REAL-TIME PATH |
clients emit events -> Kafka/PubSub -> stream processor (windowed counts, session state) -+
\-> raw event lake (source for training)
OFFLINE PATH (hours to a day)
event lake + impression log -> batch feature jobs -> OFFLINE STORE (warehouse) --sync--> ONLINE STORE
\-> training data (point-in-time joins) -> train retrieval + ranking models
\-> embeddings + ANN index build -> model registry -> canary/experiment -> servingThe two feature paths exist because freshness costs money and latency: long-term aggregates (favourite categories over 90 days) are cheap to compute in batch and change slowly, while "what did this user click in the last five minutes" needs a streaming job and a low-latency store. Both feed the same online store under the same feature definitions, which is the foundation of consistency covered below.
-- ONLINE feature store (key-value, in-memory or SSD-backed; read on the request path)
user:{user_id} -> { long_term: {top_categories, avg_session_len, ...}, -- batch, hours old
recent: {last_20_item_ids, clicks_10m, skips_10m}, -- streaming, seconds old
embedding: [128 floats], feature_version, updated_at }
item:{item_id} -> { category, age_hours, ctr_7d, popularity_rank, language, is_available,
embedding: [128 floats], feature_version }
trending:{region} -> [item_id, score] (refreshed every few minutes)
neighbours:{item_id}-> [(item_id, sim)] (top-K item-item co-occurrence, batch-built)
-- EVENT LOG (append-only, Kafka -> lake, partitioned by day)
events(event_id, ts, user_id, item_id, type, surface, request_id) -- click/view/like/skip/purchase
-- IMPRESSION LOG (the training-serving bridge; written at serve time)
impressions(request_id, ts, user_id, item_id, position, ranker_score, candidate_source,
features_snapshot_ref, feature_version, model_version, experiment_id, arm,
explore_flag, propensity)
-- outcomes are joined later by (request_id, item_id); a missing click after N minutes = negative
-- CONTROL
model_registry(model_id, kind, trained_on_range, metrics, artifact_uri, state)
experiments(experiment_id, arms[], allocation, salt, start, end, guardrail_metrics[])The important design choice is the impression log: it records the exact feature snapshot (or a reference to it), model version, position and whether the slot was exploratory. Training on features recomputed later would use information the model did not have when it served the item; training on the logged values reproduces what serving really saw.
cost per item items in
retrieve very cheap (vector 5,000,000 -> ~500-1,000 HIGH RECALL, many sources
lookup / table read)
rank moderate (feature ~500 -> ~50 HIGH PRECISION, richer model
join + model)
re-rank cheap (rules, MMR) ~50 -> 20 diversity, freshness, policy
candidate sources (union, tag each candidate with its source):
ANN retrieval : nearest items to the user embedding (two-tower model) -> personalised, semantic
item-item CF : items that co-occur with what the user just consumed -> strong, explainable
popularity : trending / top items in region -> robust fallback, cold start
content-based : items similar to seed items by metadata/embedding -> helps new itemsYou cannot run an expensive model over five million items in 100 ms, so the pipeline is a funnel: a cheap stage optimised for recall narrows the catalog to hundreds of candidates, and a costlier stage optimised for precision orders them. The consequence interviewers want you to state: the ranker can only reorder what retrieval supplies, so retrieval recall is a hard ceiling on quality, and the usual improvement is adding a source (for example item-item neighbours for a user's last few items) rather than making the ranker bigger. Multiple sources also give robustness, because one source can be stale or down without emptying the list.
Where it breaks at scale: ANN indexes are memory-hungry and must be rebuilt or incrementally updated as items are added, so new items are invisible to retrieval until indexed, which is one reason a separate "fresh items" source exists (see approximate nearest neighbor search for recall/latency trade-offs and vector databases). Candidate sources run in parallel, so the retrieval stage takes as long as its slowest source, and per-source timeouts with partial results are needed so one slow source does not hold the request. Item-item co-occurrence must be normalised (as in the demo below) or the most popular items dominate every neighbour list.
FEATURE FRESHNESS TIERS
static / slow item category, language, embedding recomputed daily batch -> online store
medium user 90-day category affinity, item CTR hourly / daily batch -> online store
fast clicks in last 10 min, session items seconds stream -> online store
request-time device, time of day, surface, position computed in the request
SKEW: how the same feature can differ between training and serving
training: SQL over the warehouse, "clicks in last 7 days" as of the training timestamp
serving : separate service code counting clicks_7d, window ends "now", null->0 default here, null->-1 there
=> model sees a different distribution online than it was trained on; offline metrics look great,
online metrics quietly disappoint. No exception is thrown.
DEFENCES
1. one feature definition used to produce BOTH offline and online values (feature store)
2. log the served feature values and train on the log (point-in-time correct by construction)
3. point-in-time joins for any recomputed history (never use data from after the label time)
4. monitor online feature distributions against training distributions (drift alarms)
5. version features; a model is pinned to the feature version it was trained withAn online feature store is a low-latency key-value system (in-memory or SSD-backed) holding the latest value per entity; the offline store is the warehouse holding history. They serve different access patterns: millisecond point lookups versus large scans with point-in-time correctness. The batch path materialises features into the online store on a schedule, and the streaming path updates fast-moving ones as events arrive; the price of streaming is operational complexity (out-of-order and late events, windowing, replays) and a second implementation that can drift from the batch definition, which is why unified definitions matter. See feature stores and data pipelines and ETL for ML.
Training-serving skew is the failure mode to name explicitly because it is silent: the service keeps returning plausible lists. Beyond definition differences it also comes from time (using future data in training), default values for missing features, model and feature versions that do not match during a rollout, and population skew when logged data only covers items the previous model chose to show. Detect it by comparing feature distributions and prediction score distributions between training data and live traffic (model monitoring and drift detection).
COLD START CASES
new user : no history -> popularity / trending for region + context (device, time, referrer),
onboarding picks, then switch to personalised after a few events
new item : no interactions -> content-based embedding, similar-item sources, reserved exploration slots
new context: unseen surface -> fall back to popularity for that surface, log heavily, learn
EXPLORATION (slots reserved so the model can learn about things it would not otherwise show)
epsilon-greedy : with prob eps, replace slot k with a random eligible item (simple, wasteful)
UCB / Thompson : favour items with high uncertainty (few impressions) or high optimistic estimate
ALWAYS log: explore_flag and the probability the item had of being shown (propensity),
otherwise later training cannot correct for the selection bias.
DIVERSITY (re-rank stage)
greedy MMR: next = argmax( lambda * relevance - (1 - lambda) * max_similarity_to_already_picked )
or simple caps: at most 2 items per category / creator in the top 5Real-time signals matter because intent changes within a session: a user who just watched three cooking videos is probably not looking for the sports content that dominates their 90-day profile. The design handles this by keeping the last few interacted items in the fast feature path and using them both as retrieval seeds (item-item neighbours of the last items) and as ranker features. The cost is the streaming pipeline and the freshness SLO you now own; a common compromise is to update session state within seconds and everything else in batch.
Cold start is a data problem, so the fallbacks are decided by what data exists: popularity for users, content and similarity for items, and deliberate exploration so new items get impressions at all. Without exploration a pure exploit system suffers a rich-get-richer feedback loop: items shown get clicks, get ranked higher, get shown more, while equally good unseen items never collect evidence. Exploration costs short-term engagement, so treat it as a budget (a small fixed share of slots) and log propensities so later models can use inverse-propensity weighting or other off-policy corrections. Diversity is similar: ranking purely by predicted click probability tends to fill the list with near-duplicates, and re-ranking with MMR or per-category caps trades a little predicted relevance for a list that covers more of the user's interests and reduces the filter-bubble effect.
BUDGET (ILLUSTRATIVE, p50-ish, server side; total target ~ 100-150 ms with headroom under 200 ms p99)
edge + auth + request parsing 5 ms
user profile / recent events read (KV) 10 ms
candidate generation (parallel, max of 4) 20 ms ANN 15 | co-occurrence 10 | trending 3 | seeds 8
merge, dedupe, filters 5 ms
item feature multi-get (mostly local cache) 10 ms
ranking model scoring (~500 candidates) 30 ms
re-rank, diversity, business rules 5 ms
serialise + log enqueue (async) 5 ms
network between services + slack 20 ms
------------------------------------------------------
sum on the critical path ~110 ms
CACHING
user's ranked list TTL ~1-5 min for repeated opens (invalidated on strong new signals)
item features + embeddings local in-process cache (5 GB fits on each node), refreshed by version
trending / popularity shared cache, refreshed every few minutes
precomputed batch recs per-user list computed offline: the FALLBACK when online path fails
DEGRADATION LADDER (each step has its own timeout)
full pipeline -> ranker timed out? return candidates ordered by retrieval score
-> retrieval down? return precomputed batch recs
-> feature store down? return trending for region (never an error page)Latency is the binding constraint, so the budget is enforced stage by stage with timeouts, not hoped for. Independent work is parallelised (candidate sources, feature fetches), which makes the stage latency the maximum of its branches; and fan-out amplifies tail latency, since a request touching ten backends at their own p99 will hit at least one slow call far more often than 1%. So each dependency gets a deadline, and the service returns the best partial result it has when a deadline passes. Item features are small enough to keep in each node's memory, turning most feature reads into local lookups and taking the feature store off the hot path for items.
Caching in a recommender is a freshness trade, not a free win: a cached list is fast and cheap but ignores the click a user made ten seconds ago, so short TTLs and invalidation on strong signals are typical, and cache hit rates are inherently lower than for static content because personalised keys are numerous (see caching strategies and Redis and caching for stampede and invalidation issues). The degradation ladder is what makes the 99.9% availability target realistic: precomputed batch recommendations and popularity lists are always available and are far better than an error. During load spikes, shed work by shrinking the candidate count or skipping the ranker for low-value surfaces before dropping requests.
EXPERIMENT PLUMBING
arm = hash(user_id + experiment_salt) mod 100 -> stable, independent between experiments
each impression logs: experiment_id, arm, model_version, feature_version
metrics: primary (e.g. click-through, watch time), long-term (return rate, retention),
guardrails (latency p99, error rate, diversity, complaint/hide rate)
THE LOOP
serve -> log impressions + outcomes -> build training data -> train -> evaluate offline
-> canary/experiment online -> ramp or roll back -> serve
OFFLINE vs ONLINE METRICS
offline: recall@k, NDCG, AUC on logged data (fast, cheap, but biased by what the OLD model showed)
online : A/B test on live traffic (slow, costly, the actual truth)Offline metrics gate which models deserve an experiment; only an online A/B test tells you whether users are actually better off, because logged data reflects what the previous model chose to show (selection bias) and proxies such as click-through can be gamed by sensational items while hurting retention. Assign users, not requests, to arms with a stable hash so a person sees a consistent experience, and watch guardrail metrics (latency, diversity, hides) as well as the target, since a model that lifts clicks by adding 60 ms of latency may not be a win. Ramp gradually (canary then wider) and keep the ability to roll back the model, the feature version and the index together. See A/B testing and canary deployments for models.
The feedback loop is the deep problem: the system's own outputs determine its future training data. This produces popularity bias, position bias (top slots get more clicks regardless of relevance), and filter bubbles, and it makes drift hard to attribute (did user taste change, or did the model change what users see?). Countermeasures are exploration with logged propensities, position as a training feature (set to a neutral value at serving), holdout groups that see an older or simpler policy, and regular monitoring of input features, prediction distributions and business metrics for drift. Retraining cadence is a judgement: daily for fast-moving catalogs, weekly or less otherwise, with feature drift alerts triggering earlier retrains. See model monitoring and drift detection.
FAILURE SYMPTOM / DETECTION BEHAVIOUR / MITIGATION
online feature store slow/down feature-read p99, errors timeout -> default features or degrade to trending; local cache for items
streaming pipeline lags "freshness" metric, event lag recent-activity features stale; fall back to batch profile; alert on lag
ANN index stale/corrupt recall probe, empty results keep previous index generation, atomic swap, other sources cover
ranker timeout / bad deploy p99, score distribution shift return retrieval-ordered list; canary + auto rollback
model/feature version mismatch drift + CTR drop with no errors pin model to feature version; deploy them together
training-serving skew offline good / online bad log-and-train, distribution monitors, shared definitions
cold cache after restart hit-rate dip, latency spike warm caches before taking traffic; single-flight; rate-limit
hot user/item key one shard saturated replicate hot keys, local cache, key salting
popular item goes unavailable user-visible dead links availability filter at re-rank, not only in features
feedback-loop collapse diversity and new-item share fall exploration budget, holdout, diversity guardrails
logging pipeline backs up impression lag, dropped rows buffer, sample debug fields; NEVER block the request on the log
experiment mis-assignment sample-ratio mismatch (SRM) alert on arm size vs allocation before trusting resultsSTAGE 1 small product, 1 region
popularity + simple item-item co-occurrence in one service, batch features in a DB,
a linear or GBDT ranker, impressions logged from day one. No streaming, no ANN.
STAGE 2 growing, tens of millions of users
online feature store (KV) fed by nightly batch + a streaming job for recent activity,
ANN retrieval for personalised candidates, candidate merging from several sources,
A/B testing framework, offline eval suite, drift monitors.
STAGE 3 large scale, strict latency
sharded feature store and index, local caching of item features/embeddings,
parallel retrieval with per-source deadlines, degradation ladder, precomputed fallbacks,
GPU or optimised CPU serving for the ranker, request batching where it does not hurt latency.
STAGE 4 multi-region, multi-surface
regional serving stacks (data stays near users), shared training with regional features,
multi-task ranking (click + watch time + hide), long-term-value objectives,
exploration and counterfactual evaluation infrastructure.Cost follows the funnel: the biggest lever is usually the number of candidates the ranker scores, then feature-store reads, then the model itself. Cutting candidates from 500 to 200 saves roughly 60% of ranking compute (by the linear cost assumption above) but lowers the recall ceiling, so it is tested in an experiment, not decided by arithmetic. Precomputing recommendations offline for inactive users and serving from cache moves work to cheap batch hardware, while online scoring is reserved for active sessions where freshness pays off.
Standard-library Python only, seeded. It builds a synthetic catalog of 60 items in 6 categories, generates 400 users with 10 interactions each, computes item-item co-occurrence and popularity from the first 8 interactions (the "offline batch"), trains a small logistic-regression ranker with SGD on candidate features to predict the 9th interaction, then evaluates hit@5 on the 10th. It also shows a diversity cap and the popularity fallback for a brand-new user. The data is synthetic, with a planted category structure, so the numbers only illustrate the mechanism; they are not a benchmark.
import random, math
from collections import defaultdict
random.seed(7)
N_ITEMS, N_USERS, CATS = 60, 400, 6
cat_of = {i: i % CATS for i in range(N_ITEMS)} # item -> category
pop_w = [1.0 / (1 + i % 10) for i in range(N_ITEMS)] # skewed baseline popularity
def make_user():
fav = random.sample(range(CATS), 2) # each user prefers 2 categories
seq = []
while len(seq) < 10:
pool = [i for i in range(N_ITEMS) if (cat_of[i] in fav) == (random.random() < 0.85)]
it = random.choices(pool, weights=[pop_w[i] for i in pool])[0]
if it not in seq: seq.append(it)
return seq
users = [make_user() for _ in range(N_USERS)] # 10 interactions each, ordered
# ---- offline "batch" job: statistics built from the first 8 interactions only (no label leakage)
count, co = defaultdict(int), defaultdict(int)
for seq in users:
b = seq[:8]
for i in b: count[i] += 1
for i in b:
for j in b:
if i != j: co[(i, j)] += 1
def sim(i, j): # cosine-style normalisation
return co[(i, j)] / math.sqrt(count[i] * count[j]) if count[i] and count[j] else 0.0
top_pop = sorted(range(N_ITEMS), key=lambda i: -count[i])
# ---- online path: candidate generation + features + ranker
def candidates(history, k=30):
if not history: return top_pop[:k], "popularity-fallback" # COLD START
score = defaultdict(float)
for h in history:
for j in range(N_ITEMS):
if j not in history: score[j] += sim(h, j)
ranked = sorted(score, key=lambda j: -score[j])[:k]
return ranked, "item-item"
def features(history, j):
cooc = sum(sim(h, j) for h in history) / max(1, len(history))
share = sum(cat_of[h] == cat_of[j] for h in history) / max(1, len(history))
return [cooc, math.log1p(count[j]) / 5.0, share]
# ---- train a linear (logistic) ranker with SGD on: history = first 8, label = item #9
w, b0 = [0.0, 0.0, 0.0], 0.0
def predict(x): return 1 / (1 + math.exp(-(sum(a * c for a, c in zip(w, x)) + b0)))
train = []
for seq in users:
hist, target = seq[:8], seq[8]
cands, _ = candidates(hist)
for j in cands: train.append((features(hist, j), 1 if j == target else 0))
for epoch in range(30):
random.shuffle(train)
for x, y in train:
g = predict(x) - y
for k in range(3): w[k] -= 0.5 * g * x[k]
b0 -= 0.5 * g
print("learned weights [cooc, log-popularity, category-share]:", [round(v, 2) for v in w])
# ---- evaluate on item #10 with history = first 9 (hit@5)
def hit_at5(ranker):
hits = 0
for seq in users:
hist, target = seq[:9], seq[9]
hits += target in ranker(hist)[:5]
return hits / N_USERS
pop_only = lambda h: [i for i in top_pop if i not in h]
cooc_only = lambda h: candidates(h)[0]
def full(h):
c, _ = candidates(h)
return sorted(c, key=lambda j: -predict(features(h, j)))
print("hit@5 popularity only :", round(hit_at5(pop_only), 3))
print("hit@5 candidates only :", round(hit_at5(cooc_only), 3))
print("hit@5 candidates+ranker:", round(hit_at5(full), 3))
# ---- diversity: at most 2 items per category in the final top-5
def diversify(ranked, n=5, cap=2):
out, per = [], defaultdict(int)
for j in ranked:
if per[cat_of[j]] < cap: out.append(j); per[cat_of[j]] += 1
if len(out) == n: break
return out
h = users[0][:9]
print("user0 history cats:", [cat_of[i] for i in h])
print("ranked top5 :", full(h)[:5], "cats", [cat_of[i] for i in full(h)[:5]])
print("diversified top5 :", diversify(full(h)), "cats", [cat_of[i] for i in diversify(full(h))])
# ---- cold start: brand-new user has no history
c, source = candidates([])
print("new user ->", source, c[:5])
# Real output from python3:
# learned weights [cooc, log-popularity, category-share]: [3.27, 5.25, 3.97]
# hit@5 popularity only : 0.235
# hit@5 candidates only : 0.29
# hit@5 candidates+ranker: 0.357
# user0 history cats: [2, 2, 2, 2, 1, 1, 0, 1, 4]
# ranked top5 : [50, 32, 2, 30, 0] cats [2, 2, 2, 0, 0]
# diversified top5 : [50, 32, 30, 0, 10] cats [2, 2, 0, 0, 4]
# new user -> popularity-fallback [50, 0, 30, 10, 20]
# Reading it: candidate generation alone beats popularity (0.29 vs 0.235 hit@5), and the ranker, which
# combines co-occurrence, popularity and category affinity, improves it further (0.357).
# The diversity cap replaced the third category-2 item with an item from category 4, at some cost in predicted relevance.
# A user with no history gets the popularity list (cold-start fallback).
# Not modelled: real embeddings/ANN, a feature store, streaming, latency, exploration, position bias.Q: Why is recommendation serving split into candidate generation and ranking?
A: Because the ranking model is too expensive to run over the whole catalog. With 5M items and a 100 ms budget, retrieval uses cheap lookups (ANN, co-occurrence tables, popularity lists) to shrink to a few hundred candidates with high recall, and the ranker spends its budget on those. The trade-off is that the ranker can never recover an item retrieval missed, so retrieval recall bounds quality.
Q: What is the difference between the online and offline feature store, and why do you need both?
A: The offline store (warehouse or lake) holds full history for training, with cheap large scans and point-in-time joins; the online store holds the latest value per key for millisecond lookups on the request path. You need both because training needs history while serving needs speed, and the design goal is that both are produced from the same feature definitions so a feature means the same thing in each. The cost is a sync process and a freshness gap between the two.
Q: What is training-serving skew and how do you catch it?
A: It is a mismatch between the feature values a model saw in training and the values it gets at serving, caused by different code paths, time windows, null handling or feature versions. It is dangerous because nothing errors; offline metrics stay good while online quality drops. Prevent it with shared feature definitions and by training on logged serving-time features, and detect it by comparing feature and score distributions between training data and live traffic.
Q: How do you serve a brand-new user?
A: With no history I use popularity or trending for the region and context (device, time, referrer), optionally onboarding choices, and switch to personalised retrieval after a few events, because session signals arrive quickly. It is a fallback with a known ceiling, so I log the cold-start traffic separately and measure how fast users graduate to personalised results. Popularity is safe but generic, and it reinforces already-popular items.
Q: How do new items ever get shown?
A: Content-based similarity (embeddings from metadata or content) gives them candidates without interaction history, and a reserved share of slots explores them, using epsilon-greedy or a bandit approach that favours uncertain items. Without exploration the system falls into a rich-get-richer loop. The cost is some short-term engagement, so the exploration share is a budget, and each exploratory impression is logged with its propensity.
Q: How do you stay within a 150 ms budget?
A: I allocate the budget per stage (for example 10 ms profile read, 20 ms parallel retrieval, 10 ms item features, 30 ms ranking), run independent calls in parallel, keep item features in a local cache, and give every dependency a timeout so a slow one returns partial results. Fan-out raises tail latency, so I watch p99 per stage, not just the total. If a stage times out, a degradation ladder returns a valid but less personalised list.
Q: How would you A/B test a new ranking model?
A: Offline metrics (NDCG, recall@k) select candidates for testing, then I assign users, not requests, to arms via a stable hash, log arm and model version on every impression, and compare a primary metric with guardrails (latency, diversity, hides, retention). I check sample-ratio mismatch first, run long enough to cover weekly cycles, and ramp gradually with rollback. Offline gains that do not appear online usually point to selection bias in the logs or to skew.
Q: What goes in the impression log and why?
A: Request id, user, item, position, ranker score, candidate source, the feature snapshot (or a reference), model and feature versions, experiment arm, and the exploration flag with its propensity. It lets you join clicks to what was actually shown, train on exactly what serving saw, debug regressions, and correct position and selection bias. It is written asynchronously so a logging problem cannot fail a recommendation request.
Follow-up: Your offline NDCG improved 3% but the A/B test shows no change. What do you check?
A: First skew: whether the offline features match what serving computes, and whether the new model was evaluated on data drawn from the old model's exposures (selection bias). Then the experiment itself: sample-ratio mismatch, too little power or duration, and whether latency or a fallback rate changed so the treatment actually served a different pipeline. Finally the metric: NDCG on logged clicks may not correlate with the online objective (watch time or retention), so I would look at the metric relationship and at segments (new versus heavy users).
Follow-up: How would you incorporate a click that happened five seconds ago?
A: Stream events through Kafka or a similar log into a stream processor that updates per-user session state (last items, short-window counts) in the online store within seconds; the service reads it on each request, using the last items as retrieval seeds and as ranker features. The trade-offs are the streaming infrastructure and its ordering and late-event issues, plus a freshness SLO to monitor. Caches of ranked lists then need short TTLs or invalidation on strong signals.
Follow-up: Popularity bias is making the feed repetitive. What levers do you have?
A: Diversity in re-ranking (MMR or category and creator caps), an exploration budget with logged propensities, debiasing in training (position as a feature, inverse-propensity weighting on exploration data), a holdout policy to measure the long-term effect, and diversity or new-item share as explicit guardrail metrics. Each costs some short-term engagement, so I would let an experiment measure retention and satisfaction before accepting the change.
This pattern appears in feeds, video and music "up next", e-commerce, ads relevance and search ranking, and the estimation an interviewer wants is the funnel arithmetic: with 20M daily users opening the surface ten times a day the read path is about 2,300 rps average and 7,000 rps at peak, each request touching one user record and a few hundred item records, so the design lives on memory-resident features (roughly 200 GB of user features sharded across the online store, a few GB of item features and embeddings cached on every node), parallel retrieval within a 100-200 ms envelope, and a ranker that scores hundreds rather than millions of items. The event stream at around a billion events a day, about 12,000 per second on average, feeds both the streaming feature job and the training lake, and the impression log at billions of rows a day (over a terabyte per day under the stated row-size assumption) forces partitioning, compression and short hot retention. All of these are ILLUSTRATIVE and should be re-derived from the interviewer's numbers. The same components show up in other designs: the vector index is the vector database problem, the event bus is message queues and event streaming, and the feature store and experimentation tooling are shared MLOps platform pieces described in feature stores and A/B testing and canary deployments.
Every stage trades quality for latency and cost: more candidates raise recall and cost, a bigger ranker improves ordering and eats the budget, and fresher features improve relevance but need streaming infrastructure and a freshness SLO. Exploration and diversity sacrifice short-term engagement for catalog coverage and long-term health, and caching reduces cost but serves slightly stale, less reactive lists. Splitting batch and streaming features is efficient but creates two code paths that can drift unless definitions are shared. The full real-time architecture is the wrong choice when the catalog is small (a few hundred items can be ranked directly with no retrieval stage), when interests change slowly (nightly precomputed lists served from a cache are far cheaper and easier to operate), when there is too little data for personalisation to beat a popularity baseline, or when the team cannot operate a streaming pipeline and the monitoring it needs. Start with popularity plus simple item-item recommendations and good logging, and add stages only when an experiment shows the extra complexity pays back. Be honest that no offline metric fully predicts online results, and that the feedback loop means the system needs continuous monitoring, not a one-time launch.