An ML model is not a build artifact you ship once; it is the output of a pipeline whose inputs (data, code, configuration, random seeds) all change, and whose quality decays after release because the world it models keeps moving. The lifecycle is the loop that manages this: collect and validate data, train, evaluate, register, deploy, monitor, and retrain. Interviewers probe it because the difference between a notebook model and a production model is almost entirely this loop, and because the failures are quiet: a model that ships with a broken feature still returns HTTP 200. For a Backend or AI engineer the useful frame is that ML CI/CD is ordinary CI/CD plus two extra things to test, data and model quality, and one extra thing to release safely, a model whose correctness you can only measure statistically.
The ML model lifecycle is the repeating set of stages a model passes through from raw data to a live prediction and back again: gather and check the data, train the model, test how good it is, save the approved version somewhere trusted, put it in front of real users, watch how it behaves, and use what you learn to train the next, better version. It is called a lifecycle rather than a one-time project because step seven always leads back to step one — the model keeps needing to be refreshed as the real world changes.
CI/CD (Continuous Integration/Continuous Delivery) is the automated pipeline that checks and ships changes without a person manually babysitting every step. For ordinary software, that means running tests and deploying code. CI/CD for ML does the same job but adds two things software CI never had to check: is the training data still good, and is the resulting model still accurate enough to trust — before it is ever allowed to replace the version currently serving users.
Think of a restaurant that keeps adjusting its menu based on what's selling. Someone tastes each new dish (evaluation) before it goes on the menu (deployment), the kitchen keeps a record of every past recipe in case a new one flops and they need to bring back the old favorite (rollback), and the whole process — taste-test, approve, publish, watch how customers react, tweak the recipe — repeats every season rather than happening once and being forgotten.
+--------------------- feedback: drift, errors, new labels ---------------------+
v |
data -> validate -> train -> evaluate -> register -> deploy -> serve -> monitor
(v_d) (schema, (code v_c, (frozen eval (model v_m + (staged: (feature (data drift,
freshness) config, set, slices, lineage: shadow, pipeline quality,
seed) gate) v_d, v_c, cfg) canary) same as latency)
training)
The tuple that defines a model: (data version, code version, config/hyperparameters, seed, environment)
Lose any one of them and "reproduce last month's model" stops being possible.The reason the loop exists is that a model is a function of its training data, so unlike application code its behaviour is not fully determined by the repository. Two consequences follow. First, everything that feeds a run has to be versioned and recorded together (see Data & Model Versioning and Lineage and Experiment Tracking), otherwise you cannot debug a regression or roll back with confidence. Second, the loop is closed by monitoring (see Model Monitoring & Drift Detection): production data drifts away from training data, labels arrive late, and the honest answer to "is the model still good?" comes from the right-hand end of the diagram, not the left. Google's widely cited MLOps write-up describes maturity as a progression from manual, notebook-driven training through automated training pipelines to automated build, test and deploy of the pipeline itself; treat those levels as a vocabulary, not a standard.
Layer What it checks Typical cost Catches
--------------- ----------------------------------------------- ------------- ---------------------------
Code tests unit tests for feature/pre-processing functions seconds off-by-one windows, wrong units,
(pure functions, deterministic) timezone bugs, null handling
Data tests schema, ranges, null rate, label rate, freshness, minutes upstream schema change, empty
row counts, duplicate keys, train/serve parity partition, leaked future data
Model tests invariance ("swap name -> same output"), minutes broken preprocessing, feature
directional ("more income -> not lower score"), order swaps, dead features
smoke train on 1k rows, output range/shape
Evaluation gate candidate vs baseline on a FROZEN eval set, minutes-hours quality regression, slice
overall + per-slice, with tolerances regression, latency/size budget
Reproducibility same data + seed + code => same metrics one extra run hidden nondeterminism,
(or within a stated tolerance) unpinned dependenciesThe cheap layers run on every commit; the expensive ones run on a schedule or when data or training code changes, which is why ML CI is usually split into a fast pull-request pipeline and a slower training pipeline. The feature code deserves ordinary unit tests because it is where most silent bugs live, and the highest-value data test is train/serve parity: the same input row must produce the same features in the offline pipeline and in the online service (see Feature Stores for why skew arises). Exact reproducibility is often unrealistic on GPUs, since some kernels and parallel reductions are nondeterministic unless you opt into deterministic modes at a speed cost, so many teams assert reproducibility within a tolerance instead. Where it breaks at scale: a full retrain on every pull request is unaffordable, so teams test on a small sample and rely on the gate for the real decision.
Gate rule used in the demo below (all must hold to promote):
1. overall_candidate - overall_baseline >= +0.010 (minimum improvement)
2. for every slice: slice_candidate - slice_baseline >= -0.010 (no-regression-on-slices)
Why a tolerance is not free -- the noise floor of a metric:
standard error of an accuracy p on n eval rows ~ sqrt(p * (1 - p) / n)
p = 0.75, n = 2,000 -> sqrt(0.1875 / 2000) = 0.0097 (~1 point)
p = 0.75, n = 200 -> sqrt(0.1875 / 200) = 0.031 (~3 points)
A 0.010 threshold on a 200-row slice is smaller than the noise. It will block or pass at random.An evaluation gate turns "the new model looks better" into a reproducible decision. The minimum-improvement rule stops you from paying rollout risk for a change that is within noise, and the per-slice rule stops the classic failure where a higher overall number hides a regression for a minority group, a region or a device type (a good average can be the sum of a large win on a big slice and a bad loss on a small one). The trade-off is that any fixed tolerance sits somewhere on a curve between false blocks and false passes, and it is only meaningful relative to the sampling noise above. Practical answers are a larger or paired eval set (the same rows scored by both models, then a paired bootstrap or McNemar-style test on the disagreements), minimum slice sizes, and tolerances set from measured seed-to-seed variance rather than guessed. Also gate on non-quality budgets such as latency, model size and cost per request, since a slightly better model that doubles p99 latency is often a regression.
Promotion policy sits on top of the registry: a registry stores model versions with their lineage and a label for where each version stands (for example candidate, staging, production, archived; some tools use stages, newer MLflow versions favour aliases, so check current docs). The policy states who or what may move a version between labels: automated on gate pass for staging, then a human approval or a successful canary for production. The registry entry must record the data version, code commit, config, seed and the evaluation report that justified the promotion, so a promotion is auditable and a rollback is just repointing the production label at the previous version.
Live traffic 100%
|
+-- SHADOW: copy of requests -> candidate; responses logged, NEVER returned to users
| checks: crashes, latency, output distribution vs production. No user risk.
|
+-- CANARY: 1% -> 5% -> 25% -> 100% of real users get the candidate
| advance only if guardrail metrics hold at each step; else auto-rollback
|
+-- ROLLBACK: repoint 'production' to the previous registry version
requires: old model artifact kept, old feature schema still served,
no irreversible data migration coupled to the release
ILLUSTRATIVE canary arithmetic (assumed): 10,000 req/s total, 1% canary = 100 req/s.
10 min bake = 60,000 requests: enough to see a crash-rate or p99 latency regression,
usually NOT enough to see a 0.5-point change in a conversion-style business metric.Deployment is where ML differs most from a web service: the failure signal is statistical, delayed and sometimes absent, so staged exposure matters more, not less. Shadow mode answers "does it run and does it behave sanely on real inputs" without user impact; canary answers "do real users and real latency stay healthy"; a proper A/B test answers "is it actually better on the business metric" and needs far more traffic and time (see A/B Testing & Canary Deployments for Models). Rollback is only cheap if you designed for it: the previous model and its feature schema must still be servable, and a model that needs a new feature is coupled to a data pipeline release, so ship the feature first, then the model. When models are large, note that rolling a pod-based service means loading multi-gigabyte weights, so a canary and a rollback have a cold-start cost that plain stateless services do not.
Continuous training (CT) is the automated retrain loop, and its trigger is a design choice. Schedule triggers (nightly, weekly) are simple and predictable; data triggers fire when enough new labelled data accumulates; drift triggers fire when monitoring shows the input or prediction distribution has moved; performance triggers fire when delayed ground truth shows quality has dropped. Each retrain still goes through the same gate, so CT should produce candidates, not deploy blindly. The classic trap is a feedback loop: if the model's own outputs decide which data gets labelled (recommendations, fraud blocks), retraining on that data can reinforce its own biases, so hold out an exploration slice or a randomized sample.
Silent failure Why nothing alarms Where it is caught
------------------------------------- -------------------------------------- ------------------------
Upstream column renamed / units change pipeline runs, feature = null or wrong data tests, schema contract
Training-serving skew both sides "work", just differently parity test, feature store
Label leakage (future info in features) offline metric looks great time-based split, review
Stale feature (pipeline late) serving reads yesterday's values freshness check, monitoring
Eval set contaminated by train data metric inflated dedupe, dataset versioning
Population drift code unchanged, world changed drift monitoring
Feedback loop model shapes its own training data exploration, holdoutsSculley et al., "Hidden Technical Debt in Machine Learning Systems" (2015), is the standard reference here: the model is a small box in a large system, boundaries erode because the model consumes whatever data it is given (the "changing anything changes everything" problem), and undeclared consumers and unstable data dependencies create coupling no compiler will flag. The practical lesson for CI/CD is that most production ML incidents are data and integration incidents, not algorithm incidents, so invest in data contracts, ownership, parity tests and monitoring before chasing a better architecture.
LLM applications reuse the same loop with different artifacts. The "model" you release is often a bundle of prompt template, model name and version, retrieval configuration, tool definitions and sampling parameters, and a change to any of them is a release. So the prompt lives in version control, and CI runs an evaluation suite on a versioned dataset of representative and adversarial cases with assertions where possible (JSON schema validity, required citations, refusal behaviour) and an LLM-as-judge or human rubric where not. Judges are themselves noisy and biased, and hosted models can change underneath you unless you pin a dated version, so run enough cases, repeat runs to estimate variance, and treat small score differences with the same suspicion as the accuracy noise floor above (see Prompt Engineering & Prompt Management and Observability for AI Systems).
# Python 3 (standard library only): a toy pipeline with data tests, a reproducibility check and an
# evaluation gate (minimum improvement + no regression on slices) that promotes or blocks candidates.
import math, random
def make_data(n, seed, mobile_share=0.5):
rng = random.Random(seed)
rows = []
for _ in range(n):
seg = "mobile" if rng.random() < mobile_share else "desktop"
x1, x2 = rng.gauss(0, 1), rng.gauss(0, 1)
signal = (1.2 * x1 + 0.5 * x2) if seg == "mobile" else (0.5 * x1 + 1.2 * x2)
y = 1 if signal + rng.gauss(0, 0.6) > 0 else 0
rows.append({"seg": seg, "x": (x1, x2), "y": y})
return rows
def train(rows, features, seed, epochs=30, lr=0.1):
rng = random.Random(seed)
w = [0.0] * (len(features) + 1) # last weight = bias
rows = rows[:]
for _ in range(epochs):
rng.shuffle(rows)
for r in rows:
z = w[-1] + sum(w[i] * r["x"][f] for i, f in enumerate(features))
p = 1 / (1 + math.exp(-max(-30, min(30, z))))
g = p - r["y"]
for i, f in enumerate(features):
w[i] -= lr * g * r["x"][f]
w[-1] -= lr * g
return {"features": features, "w": w}
def predict(m, r):
z = m["w"][-1] + sum(m["w"][i] * r["x"][f] for i, f in enumerate(m["features"]))
return 1 if z > 0 else 0
def evaluate(m, rows):
out = {"overall": 0.0}
by = {}
for r in rows:
by.setdefault(r["seg"], []).append(predict(m, r) == r["y"])
allc = [c for v in by.values() for c in v]
out["overall"] = sum(allc) / len(allc)
for s, v in sorted(by.items()):
out[s] = sum(v) / len(v)
return out
# ---- CI stage 1: data tests -------------------------------------------------
def data_tests(rows):
errs = []
if len(rows) < 1000: errs.append(f"too few rows: {len(rows)}")
pos = sum(r["y"] for r in rows) / len(rows)
if not 0.2 < pos < 0.8: errs.append(f"label rate {pos:.2f} outside 0.2-0.8")
segs = {r["seg"] for r in rows}
if segs != {"mobile", "desktop"}: errs.append(f"unexpected segments {sorted(segs)}")
return errs
# ---- CI stage 2: the evaluation gate ----------------------------------------
def gate(cand, base, min_gain=0.01, max_slice_drop=0.01):
reasons = []
gain = cand["overall"] - base["overall"]
if gain < min_gain:
reasons.append(f"overall gain {gain:+.3f} < required {min_gain:+.3f}")
for s in ("mobile", "desktop"):
d = cand[s] - base[s]
if d < -max_slice_drop:
reasons.append(f"slice '{s}' regressed {d:+.3f} (limit -{max_slice_drop:.3f})")
return (not reasons), reasons
if __name__ == "__main__":
train_rows = make_data(4000, seed=1)
test_rows = make_data(4000, seed=999) # frozen eval set, never used for training
print("data tests:", data_tests(train_rows) or "PASS")
# reproducibility check: same data + same seed must give identical weights
same = train(train_rows, [0, 1], seed=0)["w"] == train(train_rows, [0, 1], seed=0)["w"]
print("reproducible retrain:", same)
registry = {"production": "baseline-v1"}
baseline = train(train_rows, [0], seed=0)
base_m = evaluate(baseline, test_rows)
print("baseline-v1 (x1 only) ", {k: round(v, 3) for k, v in base_m.items()})
cands = {
"cand-1 same feats, new seed": train(train_rows, [0], seed=7),
"cand-2 x2 only": train(train_rows, [1], seed=0),
"cand-3 x1+x2": train(train_rows, [0, 1], seed=0),
}
for name, m in cands.items():
cm = evaluate(m, test_rows)
ok, why = gate(cm, base_m)
print(f"{name:28}", {k: round(v, 3) for k, v in cm.items()})
if ok:
registry["staging"] = name.split()[0]
print(" -> PROMOTE to registry stage 'staging'")
else:
print(" -> BLOCK: " + "; ".join(why))
print("registry:", registry)
# Real output from python3 (the toy data is seeded, so this is deterministic):
# data tests: PASS
# reproducible retrain: True
# baseline-v1 (x1 only) {'overall': 0.699, 'desktop': 0.608, 'mobile': 0.791}
# cand-1 same feats, new seed {'overall': 0.692, 'desktop': 0.607, 'mobile': 0.778}
# -> BLOCK: overall gain -0.007 < required +0.010; slice 'mobile' regressed -0.013 (limit -0.010)
# cand-2 x2 only {'overall': 0.715, 'desktop': 0.814, 'mobile': 0.615}
# -> BLOCK: slice 'mobile' regressed -0.176 (limit -0.010)
# cand-3 x1+x2 {'overall': 0.804, 'desktop': 0.815, 'mobile': 0.792}
# -> PROMOTE to registry stage 'staging'
# registry: {'production': 'baseline-v1', 'staging': 'cand-3'}
# Note: cand-1 is the SAME model family retrained with a different seed; it is blocked because seed
# noise alone moved the 'mobile' slice by -0.013, a hint that a 0.010 tolerance is near the noise
# floor of a 2,000-row slice. cand-2 wins overall (+0.016) but fails the slice rule badly.Q: How does CI/CD for ML differ from CI/CD for a normal service?
A: The artifact depends on data as well as code, so you have to version and test data, and "does it pass" becomes a statistical comparison against a baseline instead of a boolean unit test. It also needs an extra retrain-and-evaluate stage and post-deploy monitoring that can trigger new runs. The cost is longer, more expensive pipelines and fuzzier failure criteria, which is why teams split fast per-commit checks from slower training pipelines.
Q: What goes into an evaluation gate?
A: A frozen eval set never used for training or tuning, the candidate and the current baseline scored on identical rows, a minimum-improvement threshold, a no-regression tolerance on each important slice, and budgets for latency, size and cost. The catch is that thresholds must sit above the sampling noise of the metric or the gate is random, so I would set them from measured variance and enforce minimum slice sizes.
Q: Why check slices and not just the overall metric?
A: An average can rise while a subgroup gets worse, for example when a change helps a large segment a lot and hurts a small one a little. In the demo above the x2-only candidate improves overall accuracy from 0.699 to 0.715 while dropping the mobile slice from 0.791 to 0.615, which the overall check alone would have promoted. Slices should be chosen from business and fairness risk, not exhaustively, because every extra slice adds another chance of a noisy false block.
Q: Explain shadow, canary and A/B for a model rollout.
A: Shadow sends a copy of live requests to the candidate and logs its outputs without serving them, so it tests behaviour and latency at zero user risk but cannot measure user outcomes. Canary serves a small fraction of real users and watches guardrail metrics to catch crashes and regressions cheaply. A/B splits users to estimate the effect on a business metric with statistical rigor, which needs more traffic and time; each step trades speed for confidence.
Q: What triggers retraining, and how do you avoid retraining pointlessly?
A: Schedule, new-data volume, detected drift, or a measured performance drop once labels arrive. Pointless retrains waste compute and add rollout risk, so the pipeline should compare the candidate against the current model through the same gate and only promote when it is better, and drift alerts should be tuned because drift in an input does not always hurt quality.
Q: What would you test in the feature code?
A: Pure-function unit tests for windowing, unit conversions, time zones, null and unseen-category handling, plus a parity test that the offline and online implementations return the same features for the same row. Those bugs do not crash anything; they silently degrade predictions, so tests are the only cheap defence. A single shared implementation, often via a feature store, removes a whole class of them.
Q: How do you do CI for an LLM application where the "model" is a prompt?
A: Version the prompt, model name, retrieval and tool config together, and run a versioned eval set on every change with programmatic assertions (schema, citations, refusals) and a judge or rubric for quality. Because judges and hosted models are noisy, I run repeats to estimate variance and pin a dated model version, and I only trust differences larger than that variance. The trade-off is eval cost per change against the risk of shipping a silent quality regression.
Follow-up: Your gate blocks a candidate that looks better in offline metrics. Do you override it?
A: First find out which rule blocked it and whether the effect is real or noise: rerun with more seeds or a larger paired eval set and look at the slice sizes. If it is a real regression on an important slice, fix the model or change the product decision explicitly with sign-off; if it is noise, the fix is a better gate, not a silent override, because overrides train the team to ignore the gate.
Follow-up: How do you roll back a model when the new one needs a new feature?
A: Keep the previous model artifact and its feature schema servable, and release features before models so the feature exists for both versions. Then rollback is just repointing the production label, and the extra column is ignored by the old model. If a feature was removed or its meaning changed in place, rollback is no longer clean, which is why such changes need versioned features.
Follow-up: Labels arrive 30 days late. How do you know a deployed model is degrading?
A: Use proxy signals in the meantime: input and prediction distribution drift, feature null rates, confidence distributions, downstream behavioural metrics, and a small human-labelled sample. These are early warnings, not proof, so I would alert on them but confirm with real labels when they land, and I would design the canary around metrics that are available quickly.
In an ML platform or "design a recommendation / fraud / RAG system" interview, the lifecycle is the operational half of the design: name the data validation stage, the training pipeline with versioned inputs, a frozen eval set with slice metrics, the registry, a staged rollout, and the monitoring that closes the loop. Give the interviewer estimation with stated assumptions: say a nightly retrain on a day of data taking a few hours of a small GPU pool, an eval run of 50,000 held-out examples costing minutes, and a canary at 1% of an assumed 5,000 requests per second, which gives 50 requests per second and roughly 15,000 requests in five minutes, enough to catch a latency or error regression but not a small quality change, so quality confirmation needs either delayed labels or an A/B test. Call out coupling points: the feature pipeline has to be released ahead of the model, the registry has to retain the previous version for rollback, and large weights mean deployments have cold-start time that affects rollout speed and autoscaling. For LLM systems add the eval-in-CI loop for prompts and retrieval configuration, version pinning of hosted models, and cost and latency budgets in the gate.
The decision is how much automation and how strict a gate to put between a trained model and users. Strict gates, slice checks, staged rollouts and human approval reduce the chance of shipping a silently broken model, but they slow iteration, cost compute, and produce false blocks whenever thresholds sit near metric noise; loose gates ship faster and let regressions reach users, where they can persist unnoticed because nothing crashes. The right balance depends on blast radius: a low-stakes recommendation widget can auto-promote after a passing gate and a short canary, while a credit or medical model deserves human sign-off, fairness slices and longer bake time. Heavy MLOps machinery is the wrong choice for a single model retrained twice a year by one person, where versioned data, a reproducible script and a written checklist give most of the benefit; likewise continuous training is wrong when labels are scarce, slow or feedback-contaminated, because it automates the retraining of a model you cannot evaluate. Start with reproducibility and a simple gate, then add automation where an incident or a bottleneck justifies it.