Shipping a new model, prompt, or retrieval setup is a deployment problem and a measurement problem at the same time. A model that wins on a held-out test set can still lose in production because real traffic has a different mix, users react to the output, latency and cost change under load, and the offline metric was only a proxy for what the business cares about. Interviewers probe this topic because it separates people who "deploy the model" from people who can answer "how do you know the new version is better, and how do you limit the damage if it is worse?". For a Backend or AI engineer the working vocabulary is a ladder of increasingly risky exposure (offline evaluation, shadow, canary, A/B test, full rollout), plus enough statistics to avoid the classic mistakes: peeking, underpowered tests, broken randomisation, and declaring victory on the wrong metric. It pairs directly with model monitoring and drift detection (the signals a canary watches) and the ML model lifecycle and CI/CD (the pipeline that promotes or rolls back a version).
A/B testing is a way of comparing two versions of something — a webpage, a recommendation model, a checkout flow — by showing each version to a different group of real users at the same time and measuring which one performs better on a specific goal, like clicks, purchases, or task completion. Because the two groups are chosen randomly and see everything else the same, any consistent difference in outcomes can be credited to the change itself rather than luck or timing.
A canary deployment is a cautious way of releasing a new version of software: instead of switching every user over at once, you first send it to a tiny slice of traffic and watch closely for errors, crashes, or slowdowns before gradually increasing that slice. The name comes from the old mining practice of bringing a caged canary into a coal mine — if the canary showed signs of trouble, miners knew to get out before anyone else was harmed.
Imagine a restaurant chain testing a new recipe. Instead of changing the menu at every location overnight, it first serves the new dish at one or two restaurants and asks staff to watch closely for complaints or send-backs (the canary). Once it is clearly not a disaster, it runs the new recipe alongside the old one in several markets, tracking which one customers reorder more often (the A/B test), before finally putting it on the menu everywhere.
Stage Users see new output? Answers the question Cannot tell you
--------------- --------------------- -------------------------------- ---------------------------
Offline eval no "Is it better on a fixed set?" real traffic, user reaction
Shadow no (response dropped) "Does it run, how fast, how does whether users prefer it
its output differ on live input?"
Canary (1-50%) yes, a small slice "Is it BROKEN? errors, latency, whether it is *better*
guardrail metrics" (sample too small/short)
A/B test yes, randomised split "Is it better on the metric we why (needs analysis), and
care about, with confidence?" long-term effects if short
request ---> router ---> PROD model ---> response returned to user
|
+--(copy)--> NEW model ---> logged and compared, NEVER returned (shadow)Each rung answers a different question at a different risk. Offline evaluation is cheap and repeatable but only as good as the dataset and metric; it cannot see feedback loops or a shifted input distribution. Shadow deployment sends a copy of live requests to the candidate and discards its answer, so users carry zero risk while you observe latency, resource use, error rate and how much its outputs differ from production. Its limits are practical: it doubles the compute for the mirrored fraction, it must not perform side effects (writes, emails, tool calls with real consequences), and because nobody sees the answer it produces no user-behaviour signal. A canary exposes a small real slice to the new version and is fundamentally a safety check; an A/B test is a decision experiment. Teams commonly run them back to back: shadow for plumbing, a short canary for safety, then a properly sized A/B test for the ship decision.
Ladder: 1% -> 5% -> 25% -> 50% -> 100% (each step held until checks pass)
Guardrail metrics (compared against the concurrent baseline, not last week):
reliability : 5xx / timeout rate, malformed-output rate (JSON/schema failures)
latency : p50, p95, p99 (and time-to-first-token for LLMs)
quality : proxy signals - thumbs-down rate, retry rate, escalation-to-human rate
cost : tokens or GPU-seconds per request
business : conversion / task completion (usually too slow to gate a canary alone)
Rollback rule (automatic): if (error z-score > threshold AND enough events)
OR (p95 ratio > limit) => route 100% back to baseline, page a human
Blast radius arithmetic (ILLUSTRATIVE): 200 req/s, canary at 5%, bad model detected after 3 min
= 200 * 0.05 * 180 = 1,800 affected requests vs 36,000 if it had shipped at 100%A canary works because the damage is proportional to the traffic share times the time to detect. Two design decisions matter most. First, compare the canary against the concurrent baseline over the same window, not a historical number, so a traffic spike or upstream outage does not look like a model regression. Second, route stickily (per user or per session) so one person does not bounce between versions, which also makes their experience interpretable. The statistics are the trap: at 1% traffic with a 1% baseline error rate, 600 canary requests contain only about six errors, so an error rate that is truly 2.5% cannot be distinguished from noise yet. The simulation below shows this: the buggy canary passes the 1% stage because it has too few error events, and is caught at 5%. That is why rollback rules require a minimum event count, why early stages exist to catch catastrophic failures (crashes, schema breakage, 10x latency) rather than subtle ones, and why stricter thresholds are used when you evaluate repeatedly, since every check is another chance for a false alarm. A rule that is too sensitive rolls back healthy releases and teaches the team to ignore or bypass it; too lax and it protects nobody.
Assignment: bucket = hash(experiment_salt + user_id) mod 10000 # deterministic, sticky
bucket < 5000 -> control (A) else -> treatment (B) # salt differs per experiment
Sample size per arm, two proportions (alpha = 0.05 two-sided, power = 0.80):
n ~= (z_alpha/2 + z_power)^2 * [ p1(1-p1) + p2(1-p2) ] / (p2 - p1)^2
z_alpha/2 = 1.96, z_power = 0.84 -> (2.80)^2 = 7.85
baseline 10% conversion: detect +1.00 pt -> ~14,749/arm
detect +0.50 pt -> ~57,760/arm (half the effect = ~4x the users)
detect +0.25 pt -> ~228,552/arm (numbers from the script below)
Duration (ILLUSTRATIVE): 10,000 eligible users/day, 50/50 split, need 57,760/arm
-> 115,520 users total -> ~11.6 days; run whole weeks so weekday/weekend mix is balancedThe randomisation unit decides what "independent" means. Randomise by user when the experience should be consistent across visits and outcomes are per user; randomise by request only when requests really are independent and the treatment has no memory (a stateless classifier), otherwise a user sees both versions and effects contaminate each other. Watch for interference: users who share a marketplace, a shared cache, a rate-limited GPU pool or a social graph can affect each other, so a "win" for B may partly come at A's expense. Power is the probability of detecting a real effect of a chosen size (the minimum detectable effect, MDE); the sample-size formula shows the cost of sensitivity, because required n scales with 1/MDE squared. Sizing before you start is what stops the "let it run until it is significant" habit. Two guards should run on every experiment. A sample-ratio mismatch (SRM) check compares observed arm sizes with the intended split using a chi-square test; a tiny p-value (often below 0.001) means assignment or logging is broken (bots, a crashing variant dropping events, a redirect losing users), and the metric result should not be trusted. And beware novelty and primacy effects: a new UI or response style can get a temporary bump from curiosity, or a temporary dip while users adapt, so run long enough to see the effect settle and consider looking at only new versus only returning users separately.
A/A tests (both arms identical, so any "significant" result is a false positive)
2,000 simulated experiments, 5,000 users per arm, alpha = 0.05:
look once at the end -> 4.9% declared significant (as designed)
peek 10 times, stop at first p<0.05 -> 19.2% declared significant (~4x inflated)
Multiple comparisons: test 20 independent metrics at alpha = 0.05 with no real effect
P(at least one false positive) = 1 - 0.95^20 = 64%
Fixes: fixed horizon (decide n up front) | alpha-spending / sequential tests (SPRT, group-sequential)
Bonferroni / Benjamini-Hochberg | one PRIMARY metric, the rest labelled secondary or guardrailA p-value of 0.05 means "if there were no difference, a result this extreme would occur about 5% of the time for one pre-planned look". Each additional peek is another draw from the noise, and stopping at the first significant one turns a 5% error rate into the 19% the simulation measured. The same logic applies across metrics and segments: slice a flat experiment by ten countries and five devices and something will look like a win. The honest options are to fix the sample size and look once, to use a method designed for continuous monitoring (group-sequential or always-valid sequential tests), or to correct for the number of comparisons. Guardrail metrics are treated differently on purpose: for a guardrail such as latency you want sensitivity to harm, so you accept more false alarms, whereas the primary success metric is held to a stricter standard. Pre-registering the primary metric, the MDE, the duration and the decision rule before launch removes most of the temptation to rationalise afterwards.
Team-draft interleaving (ranking): one result list, items from both rankers alternate
ranker A: [a1 a2 a3 a4] ranker B: [b1 b2 b3 b4]
shown : [a1 b1 a2 b2 ...] (duplicates removed; each item tagged with its source)
click on a tagged item = a vote for that ranker -> every user is a paired comparison
Bandit (Thompson sampling): each variant has a Beta(success+1, fail+1) belief;
sample from each, send the request to the highest draw => traffic drifts to the winner
LLM change (prompt, model, retrieval, temperature) Metric type
task success / thumbs-up / escalation rate primary (user or human-labelled)
LLM-judge win rate on a fixed set offline gate (calibrate against humans)
p95 latency, time-to-first-token guardrail
cost per successful task, tokens per request guardrail
refusal rate, schema-valid rate, safety filter hits guardrailInterleaving suits ranking and recommendation: each user receives a merged list and the clicks are compared within the same user, which removes between-user variance, so it generally detects preferences with far fewer users than a split test. Its limits: it measures relative preference, not an absolute metric like revenue or latency, and it only applies where results can be merged. Bandits trade the clean answer for lower regret: they shift traffic toward the better arm as evidence accumulates, which is attractive for short-lived choices with many variants (headlines, prompt variants, which of several models to route to). But adaptive allocation complicates unbiased effect estimates, copes poorly with drifting rewards unless designed for it, and is a poor fit when you need a defensible effect size for a decision. For LLM changes the same discipline applies with extra hazards: outputs are stochastic (higher variance, so larger samples), the quality signal is often indirect, an LLM judge has known biases (position, length, preference for its own family's style) and should be calibrated against a human-labelled sample, and cost and latency belong in the decision as guardrails, not afterthoughts. A prompt edit is a model change and should be versioned and rolled out the same way; a slightly better answer that costs 40% more per successful task, or doubles p95 latency, may be a loss.
# Part 1 - z-test, sample size, sample-ratio mismatch, and peeking inflation
# Python 3 stdlib only: z-test, sample size, SRM check, peeking inflation
import math, random
from statistics import NormalDist
N = NormalDist()
def two_prop_z(x1, n1, x2, n2):
"""Two-sided two-proportion z-test (pooled variance). Returns (diff, z, p)."""
p1, p2 = x1 / n1, x2 / n2
pool = (x1 + x2) / (n1 + n2)
se = math.sqrt(pool * (1 - pool) * (1 / n1 + 1 / n2))
z = (p2 - p1) / se
return p2 - p1, z, 2 * (1 - N.cdf(abs(z)))
def sample_size_per_arm(p_base, mde_abs, alpha=0.05, power=0.80):
"""Approximate n per arm to detect an absolute lift `mde_abs` (two-sided)."""
p2 = p_base + mde_abs
za, zb = N.inv_cdf(1 - alpha / 2), N.inv_cdf(power)
var = p_base * (1 - p_base) + p2 * (1 - p2)
return math.ceil((za + zb) ** 2 * var / mde_abs ** 2)
# 1) A real difference: 10.0% vs 10.6% conversion, 20,000 users per arm
diff, z, p = two_prop_z(2000, 20000, 2120, 20000)
print(f"z-test: diff={diff:+.4f} z={z:.2f} p={p:.4f}")
# 2) Sample size: baseline 10%, detect +0.5 points absolute
for mde in (0.010, 0.005, 0.0025):
print(f"baseline 10%, MDE +{mde*100:.2f} pts -> n/arm = {sample_size_per_arm(0.10, mde):,}")
# 3) Sample-ratio mismatch: intended 50/50, observed split
def srm_p(n_a, n_b):
exp = (n_a + n_b) / 2
chi2 = (n_a - exp) ** 2 / exp + (n_b - exp) ** 2 / exp # df = 1
return chi2, 2 * (1 - N.cdf(math.sqrt(chi2)))
for a, b in ((50_120, 49_880), (50_900, 49_100)):
chi2, pv = srm_p(a, b)
print(f"SRM check {a:,}/{b:,}: chi2={chi2:.2f} p={pv:.2g} -> {'SRM! do not trust' if pv < 0.001 else 'ok'}")
# 4) Peeking: A/A tests (NO true difference). Look 10 times vs once at the end.
random.seed(42)
def aa_test(n_final=5000, p=0.10, looks=10):
a = b = 0
step = n_final // looks
hit_any = False
for i in range(1, n_final + 1):
a += random.random() < p
b += random.random() < p
if i % step == 0:
_, _, pv = two_prop_z(a, i, b, i)
if pv < 0.05:
hit_any = True
return hit_any, pv < 0.05 # (any peek significant, final-only significant)
runs = 2000
peek = final = 0
for _ in range(runs):
any_hit, fin = aa_test()
peek += any_hit; final += fin
print(f"A/A false positives over {runs} tests: single look {final/runs:.1%}, 10 peeks {peek/runs:.1%}")
# Real output from python3:
# z-test: diff=+0.0060 z=1.97 p=0.0484
# baseline 10%, MDE +1.00 pts -> n/arm = 14,749
# baseline 10%, MDE +0.50 pts -> n/arm = 57,760
# baseline 10%, MDE +0.25 pts -> n/arm = 228,552
# SRM check 50,120/49,880: chi2=0.58 p=0.45 -> ok
# SRM check 50,900/49,100: chi2=32.40 p=1.3e-08 -> SRM! do not trust
# A/A false positives over 2000 tests: single look 4.9%, 10 peeks 19.2%# Part 2 - a canary with an automatic rollback rule (three seeded scenarios)
# Python 3 stdlib only: canary rollout with an automatic rollback rule
import math, random
STAGES = [1, 5, 25, 50] # % to the canary (100% = promotion; no baseline left to compare)
REQS_PER_STAGE = 60_000 # total requests seen during each stage
CHECK_EVERY = 500 # evaluate guardrails every N canary requests
MIN_CANARY_ERRORS = 20 # do not judge an error rate from a handful of events
Z_ROLLBACK = 3.0 # one-sided z on error rate (strict, because we look many times)
P95_LIMIT = 1.25 # roll back if canary p95 latency > 1.25x baseline p95
def p95(xs):
xs = sorted(xs)
return xs[int(0.95 * (len(xs) - 1))]
def canary(base_err, can_err, base_mu, can_mu, seed, verbose=True):
rng = random.Random(seed)
base_lat = [rng.lognormvariate(base_mu, 0.4) for _ in range(2000)] # baseline latency reference (ms)
for pct in STAGES:
ce = cn = be = bn = 0
lat = []
for _ in range(REQS_PER_STAGE):
if rng.random() * 100 < pct: # routed to canary
cn += 1
ce += rng.random() < can_err
lat.append(rng.lognormvariate(can_mu, 0.4))
else:
bn += 1
be += rng.random() < base_err
if cn and cn % CHECK_EVERY == 0 and len(lat) == cn: # just crossed a check boundary
pc, pb = ce / cn, be / bn
pool = (ce + be) / (cn + bn)
se = math.sqrt(pool * (1 - pool) * (1 / cn + 1 / bn))
z = (pc - pb) / se
ratio = p95(lat) / p95(base_lat)
bad_err = z > Z_ROLLBACK and ce >= MIN_CANARY_ERRORS
if bad_err or ratio > P95_LIMIT:
if verbose:
why = f"error z={z:.1f}" if bad_err else f"p95 ratio={ratio:.2f}"
print(f" stage {pct:>3}%: ROLLBACK after {cn} canary reqs ({why}); "
f"canary err {pc:.2%} vs base {pb:.2%}")
return False
if verbose:
print(f" stage {pct:>3}%: ok canary reqs={cn:>6} err {ce/cn:.2%} vs {be/bn:.2%} "
f"p95 ratio {p95(lat)/p95(base_lat):.2f}")
if verbose:
print(" all stages passed -> promote to 100%")
return True
for name, args, seed in (("healthy canary", (0.010, 0.010, 4.0, 4.00), 1),
("error-rate bug", (0.010, 0.025, 4.0, 4.00), 2),
("latency regression", (0.010, 0.010, 4.0, 4.35), 3)):
print(f"--- {name} ---")
canary(*args, seed=seed)
# How often does the rule wrongly kill a HEALTHY canary? (100 seeded runs)
false_rb = sum(not canary(0.010, 0.010, 4.0, 4.0, seed=1000 + s, verbose=False) for s in range(100))
print(f"healthy canary rolled back by mistake: {false_rb}/100 runs")
# Real output from python3:
# --- healthy canary ---
# stage 1%: ok canary reqs= 610 err 1.48% vs 1.01% p95 ratio 1.02
# stage 5%: ok canary reqs= 3010 err 1.13% vs 0.95% p95 ratio 0.99
# stage 25%: ok canary reqs= 14881 err 1.01% vs 1.00% p95 ratio 1.03
# stage 50%: ok canary reqs= 29974 err 1.00% vs 1.00% p95 ratio 1.03
# all stages passed -> promote to 100%
# --- error-rate bug ---
# stage 1%: ok canary reqs= 608 err 2.14% vs 0.99% p95 ratio 1.03
# stage 5%: ROLLBACK after 1000 canary reqs (error z=3.8); canary err 2.50% vs base 1.16%
# --- latency regression ---
# stage 1%: ROLLBACK after 500 canary reqs (p95 ratio=1.43); canary err 0.60% vs base 1.10%
# healthy canary rolled back by mistake: 1/100 runsRead the output closely. The first experiment is "significant" only barely (p = 0.048), which is the normal shape of a real but small effect. The peeking result is the important one: identical arms, and yet ten looks flag a false winner about one time in five. In the canary run, the error-rate bug is invisible at 1% (under the minimum event count) and caught at 5%; the latency regression is caught immediately because latency is measured on every request, so it accumulates evidence far faster than a rare error. The final line is honest about the cost of automation: even a healthy canary is occasionally rolled back by mistake, and that false-alarm rate is a tunable trade-off against detection speed.
Q: Why is offline evaluation not enough to ship a model?
A: The offline set is a frozen sample with a proxy metric, so it cannot capture the live input mix, feedback loops, user reaction, or how latency and cost behave under load. A ranking model with better offline nDCG can still lower engagement because the metric did not reflect what users value. Offline is a cheap filter that stops obviously bad candidates; only live exposure tells you whether the business metric moved.
Q: Shadow, canary and A/B: how do they differ and when do you use each?
A: Shadow mirrors live requests to the candidate and discards its output, so it tests plumbing, latency and output differences at zero user risk but yields no user feedback. A canary gives a small real slice the new version to catch breakage quickly and roll back automatically. An A/B test gives a statistically sized randomised split to decide whether it is actually better. I normally chain them because each is cheap insurance for the next.
Q: What guardrail metrics and rollback rule would you set for a canary of a new fraud model?
A: Reliability (error and timeout rate), latency percentiles (p95, p99) against the concurrent baseline, decision-rate drift (approval and decline rate versus baseline), and downstream signals like manual-review volume; chargebacks are too delayed to gate a canary. Rollback triggers if the error-rate z-score exceeds a strict threshold with a minimum event count, or p95 exceeds, say, 1.25x baseline, or the decline rate moves outside a band. The trade-off is sensitivity versus false rollbacks, so thresholds are tuned on historical A/A comparisons.
Q: How do you decide the sample size for an A/B test?
A: Pick the baseline rate, the smallest effect worth acting on (MDE), alpha (commonly 0.05) and power (commonly 0.80), then use the two-proportion formula: n per arm is roughly (1.96 + 0.84)^2 times the summed variances divided by MDE squared. For a 10% baseline, detecting +1 point needs about 15k users per arm and +0.5 point about 58k, since halving the MDE roughly quadruples the sample. Then convert to days using eligible traffic and round to whole weeks.
Q: What is a sample-ratio mismatch and why does it invalidate a result?
A: It is when the observed split differs from the designed one by more than chance, detected with a chi-square test; 50,900 versus 49,100 on a 50/50 design gives p around 1e-8. It means users are being lost or misassigned non-randomly (bots, a variant that crashes before logging, a redirect), so the two groups are no longer comparable and the metric difference may be a selection artefact. You fix the pipeline and rerun rather than analyse the biased data.
Q: Why is peeking at an A/B test dangerous?
A: The 5% false-positive guarantee holds for a single pre-planned look; every extra look is another chance for noise to cross the threshold. In an A/A simulation of ten looks per test, about 19% of tests declared a winner versus about 5% for one look. If you need early stopping, use a sequential design such as group-sequential or SPRT-style boundaries, or spend alpha across planned looks.
Q: When would you use a bandit rather than an A/B test?
A: When the cost of showing a bad variant is high, the decision is short-lived, and there are many variants, such as picking among several prompt templates or headlines: Thompson sampling shifts traffic to better arms and lowers regret. When you need an unbiased effect size, a defensible decision record, or the reward drifts, a fixed-split A/B test is cleaner because adaptive allocation complicates inference.
Q: How would you A/B test a new prompt for an LLM support assistant?
A: Version the prompt, first gate it offline with a regression set and an LLM judge calibrated against a human-labelled sample, then canary it and split users by user id. Primary metric is task resolution or escalation rate; guardrails are p95 latency, time-to-first-token, cost per resolved conversation, refusal rate and schema validity. Outputs are stochastic so I size for higher variance and read per-segment results carefully rather than trusting one aggregate.
Follow-up: Your canary shows no problems at 5% but the model fails at 100%. How is that possible?
A: Several ways: load-dependent failures (GPU memory pressure, queueing, or a KV cache or connection pool that only saturates at full traffic), a shared cache or rate limit that the small slice never stressed, rare inputs that only appear at volume, or effects that take time (drift, data lag). Mitigations are load-testing the candidate at full scale in shadow, holding each canary step long enough to span peak periods, and keeping a fast automatic path back to the old version.
Follow-up: The test shows a significant win in one country and nothing overall. Ship to that country?
A: Not on that evidence alone. Slicing by many segments creates false positives (20 slices at 5% gives a 64% chance of at least one), so I would treat it as a hypothesis, then run a new pre-registered test in that market, or apply a multiple-comparison correction and check that the segment has a plausible mechanism. If it replicates, ship it there with monitoring.
Follow-up: How do you A/B test when users interact with each other, or when both arms share one GPU pool?
A: Interference breaks the independence assumption: if B consumes more GPU, A's latency worsens, so the measured gap can be exaggerated or hidden. Options are randomising by cluster (a region, a tenant, a graph community), time-slicing the treatment (switchback tests), or giving each arm reserved capacity. The trade-off is fewer independent units, so lower power, and you need more duration.
In an ML platform design, the deployment path is usually a model registry, a router (service mesh, API gateway or feature-flag service) that assigns traffic by hashed user id, per-version metrics (error rate, latency percentiles, quality proxies, cost) tagged by model version, and a controller that promotes or rolls back automatically; the CI/CD pipeline owns the stages and the monitoring layer supplies the guardrail signals. State the estimation: at, say, 200 requests per second, a 5% canary is 10 requests per second, so a 1% error rate yields about 0.1 errors per second, which is why detecting subtle regressions takes minutes to hours rather than seconds, and why an A/B test on a 10% baseline metric needing about 58k users per arm takes roughly two weeks at 10k eligible users per day (all ILLUSTRATIVE assumptions). Call out infrastructure constraints an interviewer likes: keeping both model versions warm doubles GPU memory during the rollout, shadow traffic adds real inference cost, routing must be sticky and idempotent, and rollback must be a config flip rather than a redeploy. For LLM products, add prompt and model versions to every log line so a regression can be attributed, and gate on cost per successful task alongside quality.
Every step of the ladder trades speed for confidence. Slow, staged rollouts limit blast radius but delay improvements and leave two versions running (more infrastructure, more operational complexity); fast rollouts ship value sooner but move failure risk to every user. Strict automatic rollback rules protect users but produce false alarms that erode trust, while permissive ones protect nobody. A/B tests give trustworthy causal answers but need traffic, time and discipline, and low-traffic products often cannot power them for small effects. Shadowing is safe but doubles compute and gives no behavioural feedback. Bandits reduce regret at the cost of clean inference; interleaving is efficient but only for ranking. The process is the wrong choice when the decision is trivially reversible and low stakes (just flip a flag and watch), when traffic is too small to reach any meaningful power (rely on offline evaluation, human review and staged rollout with strong monitoring instead), or when the metric you can measure quickly is a poor proxy for the outcome that matters, in which case a rigorous test of the wrong metric gives false confidence.