Training a model usually means running the same pipeline dozens or hundreds of times with different hyperparameters, features, data slices and seeds. Experiment tracking is the discipline (and the tooling) of recording each run well enough that you can compare runs fairly, reproduce the winner, and explain months later why a particular model went to production. Interviewers probe it because it separates people who "tried some settings and picked the best" from people who understand variance, selection bias and reproducibility, and those are exactly the things that make an offline result fail to survive contact with production. For a Backend or AI engineer it is also plain data engineering: an append-only log of structured records with a schema, links to artifacts, and a query interface.
Experiment tracking is the practice of automatically writing down what you tried, what settings you used, and how well it worked, every time you test a new version of something — so that later you or a teammate can compare attempts fairly and know exactly how to reproduce the best one. Without it, teams end up with a pile of half-remembered attempts and no reliable way to say which one actually won, or why it won.
Think of a chef testing bread recipes. Every time they bake a loaf, they jot down the flour brand, the amount of yeast, the oven temperature, how long they kneaded it, and how the loaf turned out. Without that notebook, after the twentieth loaf the chef can't remember which exact combination produced the best bread, and can't recreate it for the restaurant menu. Experiment tracking is that notebook, kept automatically, for anyone building a model instead of baking bread.
A run record (one JSON line in the demo below; a row/document in MLflow-, W&B-style tools):
{"run_id": "50aef3e3",
"params": {"cfg": "A", "lr": 0.02}, <- everything you chose
"metrics": {"val_acc": 0.7925}, <- what you measured (often a time series per step)
"seed": 0, <- every source of randomness
"data_hash": "9f2920f6", <- WHICH data (version, split, filters)
"code_version": "demo-v1", <- git commit (+ dirty flag / diff)
"python": "3.14.6", <- environment: lockfile / container image digest
"artifacts": ["model.bin", "eval_report.json"]} <- pointers, not blobs, in the log itself
Category Examples If you skip it
-------------- ---------------------------------------------- ---------------------------------
Parameters hyperparameters, architecture, feature list cannot tell runs apart
Metrics train/val loss per step, final eval, slices, cannot compare; no learning curve
latency, cost, model size
Data version dataset hash/snapshot id, split seed, filters "same code" gives different results
Code version git SHA, uncommitted diff cannot rebuild the run
Environment library versions, CUDA/driver, hardware irreproducible numerical drift
Seeds init, shuffle, dropout, augmentation, sampling cannot separate signal from luck
Artifacts weights, eval report, plots, config file cannot serve or audit the winnerThe reason to log this much is that a metric is only meaningful together with the conditions that produced it. The metric is cheap to store; the expensive mistake is discovering that the best run of the quarter has no recorded data version. Tracking links naturally to the model registry and lineage (see Data & Model Versioning and Lineage): the tracker answers "what did we try and how did it score", the registry answers "what is deployed and where did it come from". The cost side is real: per-step metrics at high frequency add overhead and storage, and synchronous logging can slow a training loop, so tools generally batch or send asynchronously, and large artifacts such as full checkpoints are stored selectively in object storage with only a pointer in the log.
Unfair comparison Fair comparison
------------------------------------------------ ------------------------------------------------
different train/val splits per run one fixed split (or the same K folds) for all
one seed per config N seeds per config; report mean and spread
"B beat A" from a single number paired by seed: how many of N seeds does B win?
tune on the test set, report the test set tune on validation; touch the test set once
compare different training budgets same epochs / same compute budget
Demo result (10 seeds each, same data split, real output further down):
cfg A mean 0.789 std 0.005 per-seed B-A: -0.007 -0.012 -0.017 +0.022 +0.005
cfg B mean 0.791 std 0.011 -0.003 -0.003 +0.015 +0.015 -0.002
B wins on 4 of 10 seeds; mean gap +0.0013; standard error of the gap 0.0041Randomness enters training in several places (weight initialisation, data shuffling, dropout, augmentation, sampling), so one run of one config is one draw from a distribution. If the run-to-run spread is larger than the gap between two configs, the ranking you see is mostly luck, and picking the "winner" from single runs is a coin flip that the tracker happily records as a fact. The fixes are to hold the data split constant so you are not mixing split variance into the comparison, run several seeds per config, and prefer paired comparisons (same seed, same split, look at the difference) which cancel shared noise. In the demo B has a slightly higher mean but wins on only 4 of 10 seeds and its gap is about a third of one standard error, so the honest conclusion is "no reliable difference". The cost is compute: N seeds multiply the bill, so teams spend seeds where it matters (final candidates and close calls) and use cheap single runs only for coarse screening.
Two hyperparameters, one budget of 9 trials. Only the learning rate really matters.
GRID 3x3 RANDOM 9 points
l2 | x x x l2 | x x
| | x x
| x x x | x x
| | x x
| x x x | x
+------------------ lr +------------------ lr
tries only 3 distinct lr values tries 9 distinct lr values
BAYESIAN / model-based: fit a cheap surrogate to (config -> score) so far,
pick the next config by an acquisition rule (explore uncertain + exploit promising).
EARLY STOPPING (successive halving / Hyperband): start many, drop the weak early.Grid search scales exponentially with the number of hyperparameters and wastes budget when some are unimportant, because every grid point repeats the same values of the important ones; Bergstra and Bengio's random-search paper (2012) made this argument, and random search generally finds a good region faster for the same budget when few parameters matter. Sample scale-like parameters (learning rate, regularisation) log-uniformly, not linearly. Bayesian optimisation (Gaussian-process or tree-based surrogates such as those in Optuna or Hyperopt-style tools; check current docs for what each offers) spends compute on modelling to spend fewer expensive trials, which pays off when a single trial costs hours of GPU and the search space is modest; it is less compelling for cheap trials, high-dimensional or conditional spaces, or when you can run hundreds of trials in parallel. Early-stopping schemes trade a small chance of killing a slow starter for a large saving in budget. In every method, the search is itself a source of overfitting, which is the next section.
Every candidate has TRUE accuracy 0.700. Validation set = 300 rows (std error ~ 0.026).
Pick the candidate with the best validation score out of N trials:
trials best-of-N validation accuracy (average of 30 repeats, real output below)
------ -------------------------------
1 0.703
10 0.738
100 0.763
1000 0.781 <- looks like a 78% model; it is a 70% model that got lucky
Rough size of the inflation: sigma * sqrt(2 ln N) with sigma = sqrt(0.7 * 0.3 / 300) = 0.026
N = 100 -> ~0.080 upper-end estimate (observed +0.063)The validation set is reused every time you compare a config, so choosing the maximum over many trials is a selection process, and the maximum of many noisy estimates is biased upward even when nothing real improved. That is why the score you optimised is not a fair estimate of future performance and why "we ran 500 sweeps and got 81%" often ships as 76%. Defences: keep a separate test set that is evaluated once, at the end, for the final candidate only; use a larger validation set or cross-validation as trial counts grow; report the number of trials alongside the result; prefer configs that are good across seeds and neighbouring values (a flat optimum is more trustworthy than a sharp spike); and for time-ordered data split by time so you are not tuning on the future. The same applies to leaderboards and shared benchmarks: each look at the test set leaks a little information, which is the quiet reason benchmark scores drift optimistic.
Tool family (conceptual; details vary by version, check current docs)
MLflow open source: tracking (runs, params, metrics, artifacts), model registry,
packaging; self-hostable
Weights & Biases hosted (with self-host options): run dashboards, sweeps, artifact lineage, reports
Others TensorBoard (curves), DVC (data/pipeline versioning), Neptune, Comet, Aim ...
LLM experiment record -- what replaces "hyperparameters":
prompt_version (git SHA or content hash) model + dated version temperature / top_p / max_tokens
retrieval config (chunking, k, reranker) tool definitions eval_set_version
judge model + rubric version n_repeats cost, latency, tokens per caseConceptually the tools do the same three jobs: a client API that writes run records, a store for metrics and artifacts, and a UI or query layer to compare runs. The choice is mostly about hosting, cost, team workflow and integration, not about capability you could not rebuild in a table; the JSONL tracker below is a deliberate reminder that the underlying data model is simple. Whatever you pick, log the data version and code SHA explicitly, because tools record what you send them and cannot infer your dataset snapshot.
LLM experiments look different but the rules are the same. The tunable object is a prompt plus model plus retrieval and sampling configuration, so log a prompt version (a hash or commit), the exact model version (hosted models can change, so pin a dated version where the provider offers one), and the eval-set version. Two extra noise sources appear. First, sampling at non-zero temperature (and, on some providers, even at temperature zero) makes outputs nondeterministic, so run repeats. Second, an LLM-as-judge is itself a noisy, biased measurement (position, verbosity and self-preference biases are documented), so log the judge model and rubric version, check judge agreement against a human-labelled sample, and treat small score gaps as unproven. Everything in the previous sections still applies: same eval set, several runs, paired comparison, and a held-out set you did not iterate prompts against (see Prompt Engineering & Prompt Management and ML Model Lifecycle & CI/CD).
# Python 3 (standard library only). Part 1: a tiny JSONL run tracker, then a seeded experiment:
# does config B (lr=0.10) reliably beat config A (lr=0.02)? Same data split for all 20 runs.
import hashlib, json, math, os, platform, random, statistics, sys, tempfile, time
class Tracker:
"""Append-only JSONL run log: one line per run."""
def __init__(self, path):
self.path = path
def log_run(self, params, metrics, seed, data_hash, code_version="demo-v1", notes=""):
rec = {"run_id": hashlib.sha1(json.dumps([params, seed, data_hash], sort_keys=True).encode()).hexdigest()[:8],
"params": params, "metrics": metrics, "seed": seed, "data_hash": data_hash,
"code_version": code_version, "python": platform.python_version(), "notes": notes}
with open(self.path, "a") as f:
f.write(json.dumps(rec, sort_keys=True) + "\n")
return rec["run_id"]
def runs(self):
with open(self.path) as f:
return [json.loads(line) for line in f]
def make_data(n, seed):
rng = random.Random(seed)
rows = []
for _ in range(n):
x = [rng.gauss(0, 1) for _ in range(3)]
y = 1 if (x[0] + 0.5 * x[1] - 0.3 * x[2] + rng.gauss(0, 1.0)) > 0 else 0
rows.append((x, y))
return rows
def fit(rows, lr, seed, epochs=3):
rng = random.Random(seed)
w = [rng.gauss(0, 0.5) for _ in range(4)] # random init: a source of seed variance
rows = rows[:]
for _ in range(epochs):
rng.shuffle(rows) # random order: another source
for x, y in rows:
z = w[3] + sum(w[i] * x[i] for i in range(3))
p = 1 / (1 + math.exp(-max(-30, min(30, z))))
for i in range(3): w[i] -= lr * (p - y) * x[i]
w[3] -= lr * (p - y)
return w
def acc(w, rows):
return sum((1 if w[3] + sum(w[i] * x[i] for i in range(3)) > 0 else 0) == y for x, y in rows) / len(rows)
path = os.path.join(tempfile.mkdtemp(), "runs.jsonl")
t = Tracker(path)
train, val = make_data(200, 1), make_data(400, 2) # fixed split shared by ALL runs
dh = hashlib.sha1(json.dumps([train, val]).encode()).hexdigest()[:8]
for cfg_name, lr in (("A", 0.02), ("B", 0.10)):
for seed in range(10):
t.log_run({"cfg": cfg_name, "lr": lr}, {"val_acc": round(acc(fit(train, lr, seed), val), 4)}, seed, dh)
runs = t.runs()
print("log lines:", len(runs), "| data_hash:", dh)
print("first line:", json.dumps(runs[0], sort_keys=True))
by = {c: {r["seed"]: r["metrics"]["val_acc"] for r in runs if r["params"]["cfg"] == c} for c in "AB"}
for c in "AB":
v = list(by[c].values())
print(f"cfg {c}: seed0={v[0]:.3f} mean={statistics.mean(v):.3f} std={statistics.stdev(v):.3f} min={min(v):.3f} max={max(v):.3f}")
print("per-seed B-A:", " ".join(f"{by['B'][s]-by['A'][s]:+.3f}" for s in range(10)))
wins = sum(by["B"][s] > by["A"][s] for s in range(10))
ties = sum(by["B"][s] == by["A"][s] for s in range(10))
print(f"paired by seed: B beats A on {wins}/10 seeds, ties {ties}")
d = [by["B"][s] - by["A"][s] for s in range(10)]
print(f"mean diff {statistics.mean(d):+.4f}, std of diff {statistics.stdev(d):.4f}, "
f"standard error {statistics.stdev(d)/math.sqrt(10):.4f}")
# Real output from python3:
# log lines: 20 | data_hash: 9f2920f6
# first line: {"code_version": "demo-v1", "data_hash": "9f2920f6", "metrics": {"val_acc": 0.7925}, "notes": "", "params": {"cfg": "A", "lr": 0.02}, "python": "3.14.6", "run_id": "50aef3e3", "seed": 0}
# cfg A: seed0=0.792 mean=0.789 std=0.005 min=0.785 max=0.797
# cfg B: seed0=0.785 mean=0.791 std=0.011 min=0.775 max=0.807
# per-seed B-A: -0.007 -0.012 -0.017 +0.022 +0.005 -0.003 -0.003 +0.015 +0.015 -0.002
# paired by seed: B beats A on 4/10 seeds, ties 0
# mean diff +0.0013, std of diff 0.0129, standard error 0.0041
# Part 2: random vs grid with the same 9-trial budget, and validation overfitting from selecting the
# best of N equal candidates. The objective in the search demo is SYNTHETIC (an illustration of
# coverage), not a real model.
import math, random
# A synthetic objective standing in for "validation score": it peaks near log10(lr) = -1.3
# and barely depends on l2. Purely an illustration of search coverage, not a real model.
def score(lr, l2):
return 0.90 - 0.08 * (math.log10(lr) + 1.3) ** 2 - 0.002 * (math.log10(l2) + 3) ** 2
BUDGET = 9
grid_lr = [1e-4, 1e-3, 1e-2] # a 3 x 3 grid = 9 trials, but only 3 distinct lr values
grid_l2 = [1e-5, 1e-4, 1e-3]
grid_best = max(score(a, b) for a in grid_lr for b in grid_l2)
def random_search(seed):
rng = random.Random(seed)
best = 0
for _ in range(BUDGET):
lr = 10 ** rng.uniform(-4, -1) # sample log-uniformly, not linearly
l2 = 10 ** rng.uniform(-5, -3)
best = max(best, score(lr, l2))
return best
rs = [random_search(s) for s in range(200)]
print(f"grid (9 trials, 3 distinct lr): best = {grid_best:.4f}")
print(f"random (9 trials, 9 distinct lr): mean best over 200 seeds = {sum(rs)/len(rs):.4f}, "
f"beats grid in {sum(r > grid_best for r in rs)}/200 seeds")
# Overfitting the validation set: every candidate has the SAME true accuracy (0.70).
# Selecting the best of N by validation accuracy inflates the validation number.
def val_selection(n_trials, val_size=300, true_acc=0.70, seed=0):
rng = random.Random(seed)
vals = [sum(rng.random() < true_acc for _ in range(val_size)) / val_size for _ in range(n_trials)]
return max(vals)
print("\ntrials best-of-N val acc (true acc of every candidate = 0.700)")
for n in (1, 10, 100, 1000):
avg = sum(val_selection(n, seed=s) for s in range(30)) / 30
print(f"{n:6d} {avg:.3f}")
# Real output from python3:
# grid (9 trials, 3 distinct lr): best = 0.8608
# random (9 trials, 9 distinct lr): mean best over 200 seeds = 0.8935, beats grid in 195/200 seeds
#
# trials best-of-N val acc (true acc of every candidate = 0.700)
# 1 0.703
# 10 0.738
# 100 0.763
# 1000 0.781Q: What would you log for every training run, and why?
A: Parameters, metrics, seeds, the data version and split, the code commit, the environment (library and driver versions) and pointers to artifacts. Together they let you compare runs, reproduce the winner and audit a deployed model; skip any one and some class of question becomes unanswerable, such as why the "same" run gives different numbers. The cost is storage and some logging overhead, which is small for scalar metrics and manageable for artifacts if you store them selectively.
Q: Config B scored higher than config A. How do you decide whether B is really better?
A: Run several seeds of each on the same split and look at the paired differences, not the single numbers. In my demo B has a higher mean by 0.0013 but beats A on only 4 of 10 seeds and the gap is a third of its standard error, so I would call it a tie and choose on other criteria such as cost, latency or simplicity. If it matters, spend more seeds or a larger validation set on the close call.
Q: Grid search or random search?
A: For more than a couple of hyperparameters random search generally gives better coverage of the important ones for the same budget, since a grid repeats the same values of each parameter; a 3x3 grid tries only three distinct learning rates while nine random trials try nine. Grid is fine for one or two parameters with a few sensible values and gives easily readable results. Neither handles expensive trials well, where model-based search or early stopping earns its overhead.
Q: How does Bayesian optimisation work at a working level, and when is it worth it?
A: It fits a surrogate model (often a Gaussian process or a tree-based model) to the configs and scores seen so far, then uses an acquisition rule to choose the next config that balances trying promising regions with exploring uncertain ones. It is worth it when each trial is expensive and the space is low- to moderate-dimensional, because it needs fewer trials. It is less useful for very cheap trials, massively parallel search or highly conditional spaces, and its sequential nature limits parallelism.
Q: How can hyperparameter tuning overfit, and how do you protect against it?
A: Every comparison on the validation set leaks a bit of information, and the best of many noisy scores is biased upward; in my simulation, 1,000 candidates that were all truly 70% produced a best validation score of about 78%. Protect with a separate test set touched once at the end, cross-validation or a bigger validation set as trials grow, reporting the number of trials, and preferring stable regions over sharp peaks. The remaining honest caveat is that repeated projects on the same benchmark overfit it collectively.
Q: How would you make an experiment reproducible?
A: Version the data (snapshot or hash), pin the code commit and the dependency and container versions, set all seeds, and record them together in the run record. Exact bit-level reproducibility on GPUs may need deterministic kernel settings that cost speed, so I would define the requirement as identical results or within a stated tolerance. Reproducibility is verified by actually rerunning a past record, not assumed.
Q: How do you track experiments on an LLM application?
A: Treat prompt version, model version, retrieval and sampling settings and the eval-set version as the "hyperparameters", and log outputs and scores per test case, plus cost and latency. Add repeats to measure generation variance, and if a judge model scores outputs, log the judge and rubric and validate it against human labels, since judges have their own bias and noise. The trade-off is eval cost per experiment against the risk of shipping a prompt that only looked better on a lucky sample.
Follow-up: Your team ran 300 sweeps and the best model scored 4 points below its validation score on the test set. Is the tuning broken?
A: Probably not broken; that gap is what selection bias predicts when many candidates are compared on a modest validation set, and it can also include real distribution shift between splits. Check the validation size and the number of trials against the expected inflation, look at seed variance, and confirm the test set was not used during selection. Then report the test number as the estimate, and consider a larger validation set or fewer, better-motivated trials next time.
Follow-up: Do you need a hosted experiment tracker, or is a spreadsheet or a log file enough?
A: For a solo project with a handful of runs a structured log file with the run schema above is enough and has real advantages: no dependency and trivially diffable. Once several people, many runs, dashboards, artifact lineage or a registry are involved, a tracker such as MLflow or Weights & Biases saves more time than it costs. The wrong move is unstructured notes, because nobody can query them later.
Follow-up: Is it valid to pick the best seed and ship that model?
A: It is a valid way to choose a specific set of weights to deploy, but the reported performance must come from data the selection did not see, because the best seed on validation is the same selection bias as the best config. It also tells you the variance is large, so downstream results (calibration, tail behaviour) may differ from another seed, and a retrain will not necessarily reproduce it unless the seed and pipeline are fixed.
In an ML platform design, the tracking service is a small, write-heavy metadata system: many training jobs append run records and metric points, the UI reads and compares them, and artifacts go to object storage with pointers in the metadata store. Give the interviewer a concrete estimation with assumptions, for example 50 engineers each launching 20 runs a day is about 1,000 runs a day, each logging 5 metrics every 10 steps over 10,000 steps, which is about 5,000 points per run and around 5 million points a day, trivial for a relational or time-series store but a reason to batch writes and downsample old curves. The heavy cost is artifacts, since a single checkpoint can be gigabytes, so retention policy (keep the promoted model and top-k runs, expire the rest) is a real design decision. Tie it to the rest of the platform: the run record's data version and code SHA connect to the lineage system, the winner is promoted to the registry, and CI/CD gates re-evaluate it (see ML Model Lifecycle & CI/CD and Data & Model Versioning and Lineage). For LLM products the same store holds prompt and eval runs, and the interviewer will want to hear about eval-set versioning and judge variance.
The decision is how much rigor to spend per experiment. Logging everything and running several seeds per config gives trustworthy comparisons and reproducible winners, at the price of storage, logging overhead and multiplied compute; logging little and running single seeds is fast and cheap but produces confident-looking rankings that are mostly noise and results nobody can reproduce. Search strategy is a similar trade: random search is simple and robust, Bayesian optimisation and early stopping save budget on expensive trials but add complexity and can be misled by noisy scores, and more trials always increase the selection bias unless the evaluation protocol grows with them. Heavy tooling is the wrong choice for a one-off analysis with a dozen runs, where a structured log is enough, and elaborate sweeps are the wrong choice when the model is limited by data quality or a bad feature, where a day of error analysis beats a week of tuning. Decide the evaluation protocol before running the sweep, spend seeds on decisions that matter, and evaluate the final pick once on data it has never seen.