Data & Model Versioning / Lineage

"Why did the model's behavior change last Tuesday?" is a question a production ML team must be able to answer in minutes, and it is unanswerable unless every model version is tied to the exact data, code, configuration and environment that produced it. Software gets that from a git commit and a build hash. ML gets less for free: the training set drifts as pipelines append and correct rows, hyperparameters live in notebooks, random seeds are forgotten, and the artifact is a multi-gigabyte blob that nobody can diff. Versioning and lineage are the discipline that restores the "same inputs, same output, and I can prove which inputs" property.

Interviewers probe this because it is where ML systems differ from ordinary backends, and because it is the foundation of three things they care about: reproducibility (can I rebuild it?), safe operations (can I roll back, and what is affected when an upstream table turns out to be wrong?) and compliance (which data trained the model that made this decision?). A strong answer talks about content addressing, immutable snapshots, a registry with explicit promotion, and a lineage graph you can query in both directions. It is also worth linking this page to experiment tracking (which records runs) and the model lifecycle and CI/CD (which consumes registry versions).

What Is Data and Model Versioning?

Versioning means keeping a permanent, labelled snapshot of something every time it changes, so you can always go back and look at (or restore) an earlier copy. For an AI system this means saving a snapshot of the exact training data used and a snapshot of the exact model that came out of training, each one tagged so it can be found again later. It's the same idea as a document's "version history," just applied to the data and the model instead of a paragraph of text.

What Is Lineage?

Lineage is the record of where something came from and what it touched along the way — which raw data fed which processing step, which processing step fed which model, and which model produced which decision. It turns "the model gave a strange answer" into a traceable chain you can walk backward to find the cause, or walk forward to see everything affected if one piece turns out to be wrong.

Where This Is Used

• A bank using AI to approve or deny loans keeps versioning and lineage records so it can prove, months later, exactly which data and model version made a specific decision if a regulator asks.
• Companies like Netflix and Uber tag every model with the exact dataset it trained on, so if a model starts behaving oddly in production, engineers can trace it back to a specific data change.
• Tools such as MLflow, DVC and lakeFS are used by ML teams to store these snapshots and let anyone reload an older data or model version on demand.
• When a company discovers a bug in one of its data sources, lineage tells them instantly which downstream models and reports were built from that bad data.

A Day-to-Day Example

Think of a family recipe that's been tweaked over generations. Versioning is like keeping every past copy of the handwritten recipe card, so you can always see exactly what changed between grandma's version and this year's. Lineage is being able to trace a disappointing batch of cookies back to "we switched brands of flour last month" — following the thread from a bad result back to the specific ingredient that changed.

Deep Dive: Why Data, Code, Config and Model Must Be Versioned Together

A trained model is a function of MANY inputs, not just code:

  model_v7  =  train( code@commit,  data@snapshot,  config,  environment,  seed )
                        |             |               |         |            |
                        git          DVC/lakeFS/     YAML/     lockfile /   int
                                     Delta/Iceberg   params    container
                                     snapshot id               image digest

  Change ANY one input  ->  a different model (in general).
  Version only the code ->  "git checkout v7 && python train.py" gives you a DIFFERENT model,
                            because the table it reads has moved on, so you cannot reproduce or bisect.

  Manifest for one registered model version (the unit of provenance):
  {
    "model":   "churn:7",
    "weights": "sha256:9f2c...",          # content hash of the artifact
    "data":    "sha256:41ab...",          # snapshot id of the training set
    "code":    "git:a1b2c3d",
    "config":  "sha256:77e0...",
    "image":   "sha256:c0de...",          # container digest = libs + CUDA + Python
    "seed":    1234,
    "metrics": {"auc": "recorded by the eval step"},
    "parent":  "churn:6"
  }

The reason to bind them into one manifest rather than versioning each separately is that the questions you ask in production cut across them. "Which model was serving when the complaint arrived?" needs the registry. "What did it train on?" needs the data snapshot. "Can I retrain it exactly?" needs code, config, environment and seed. If these live in four systems with no common key, the join is done by a human reading Slack history. The pragmatic rule is that a training run writes one manifest, the registry stores it beside the artifact, and deployment references the version, never a mutable path such as models/latest.

Be honest about limits: bit-exact reproduction is not always achievable. GPU kernels with non-deterministic reductions, different hardware or library versions, data-loader ordering and mixed precision can change results slightly even with everything pinned. The realistic goal is "reproducible within a tolerance and fully explainable", which is why the container digest and seed are recorded too. Distinguish this from the database notion of snapshots in MVCC: there a snapshot is transient and garbage-collected; here snapshots are deliberately retained as evidence.

Deep Dive: Content-Addressed Storage and Dataset Snapshots

Content-addressed store (CAS): key = hash(bytes).  Same bytes -> same key -> stored once.

  dataset v1 = manifest [ h(chunk0), h(chunk1), h(chunk2) ]     snapshot id = h(manifest)
  dataset v2 = manifest [ h(chunk0), h(chunk1), h(chunk2') ]    only the changed chunk is new

        v1 manifest  --> chunk0 --+
                                  +--> shared, stored once
        v2 manifest  --> chunk1 --+
        v1 --> chunk2      v2 --> chunk2'   (new blob)

  Properties:
   - immutable: changing content changes the key, so an id can never silently mean two things
   - integrity for free: re-hash on read; mismatch = corruption
   - dedup: unchanged chunks are not copied
   - cheap "copy": a new version is a new small manifest pointing at existing blobs (Merkle-style)

  Illustrative arithmetic (assumptions: 100 GB dataset, 1 GB of chunks change per day, 30 days):
   - full copy per version : 100 GB x 30        = 3,000 GB
   - chunk-level dedup     : 100 GB + 30 x 1 GB =   130 GB
   Caveat: if daily edits are scattered rows touching every chunk, dedup savings shrink toward zero.

Content addressing is the idea behind git objects, Docker image layers and (conceptually) tools such as DVC, which keeps a small pointer file in git while the data lives in a remote object store keyed by content hash. lakeFS presents a git-like branch/commit model over object storage and is generally described as doing zero-copy branching by manipulating metadata rather than copying objects. Table formats such as Delta Lake and Apache Iceberg reach a similar goal a different way: each commit produces a new table snapshot over immutable data files, enabling "time travel" reads by version or timestamp. Exact mechanics, hash algorithms and retention defaults differ by tool and version, so check current docs before making claims in an interview; the transferable idea is immutable objects plus small mutable pointers. See also object storage and data lakes for where the blobs physically live.

Where it breaks at scale: hashing terabytes is I/O-bound and slow on first ingest (tools cache hashes by file metadata), millions of tiny files make the manifest itself large, and immutable snapshots never free space until you define a retention policy and a garbage collector that understands which snapshots are still referenced by a registered model. A GDPR "delete this user" request collides with immutability: you need either a rewrite that produces a new snapshot plus deletion of old blobs, or a documented retention exception. Plan for that deletion path before you need it.

Deep Dive: Model Registry, Stage Promotion and Rollback

Registry entry:   name + version  ->  artifact URI + manifest + metrics + stage/alias

  churn:1   staging
  churn:2   production   <- serving traffic
  churn:3   staging      <- candidate: offline eval passed, needs shadow/canary

  promote(churn:3, production):
     1. gate checks: eval metrics vs current production, schema compatibility, bias/safety tests
     2. move the pointer: production -> churn:3   (churn:2 becomes archived/previous)
     3. serving layer reloads by alias or version, ideally through canary traffic first

  rollback = move the SAME pointer back to churn:2.   No retrain, no rebuild, seconds not hours.

  Bad:  deploy "s3://models/churn/latest"     (mutable; what is running? which data? unknowable)
  Good: deploy "churn:3"  -> resolves to immutable artifact hash recorded in the manifest

The registry separates two lifecycles that otherwise get tangled: building versions (many, automatic, most discarded) and choosing what serves (few, deliberate, audited). Stages such as staging, production and archived, or aliases such as champion and challenger, are just mutable pointers to immutable versions; MLflow's model registry, for example, has historically used stages and more recently favors aliases and tags, so verify the current API rather than quoting a fixed vocabulary. What matters in a design discussion is the promotion policy: automated gates on offline metrics, then shadow or canary exposure (see A/B testing and canary deployments), then full rollout, with an audit record of who or what promoted it.

Rollback is only fast if the previous version is still loadable: keep its artifact, its feature definitions and its serving container. The classic trap is a model rolled back while the feature pipeline has already moved to a new schema, so the old model now receives inputs it was not trained on. Registering the feature-set version in the manifest, and rolling back the two together when needed, closes that gap. Also remember that rollback of a model does not undo its effects: predictions already written to a database or recommendations already shown stay.

Deep Dive: Lineage Graph and Impact Analysis

Lineage is a DAG of derivations (edges point from input to output):

  raw_events_2026-09 --> cleaned_events --> features_v4 --> training_set@41ab --> churn:7 --> endpoint /predict
                                                                 ^                    ^
                                                          label_rules.py         config@77e0

  Backward query ("root cause"):   churn:7 <- training_set@41ab <- features_v4 <- cleaned_events <- raw_events
  Forward query  ("impact"):       raw_events found corrupt -> which datasets, models, endpoints consumed it?

  Impact analysis = graph reachability from the bad node.
   affected = descendants(bad_node)   (transitive closure, cycle-free because it is a DAG)
   then filter to models with stage == production  ->  the incident scope, with an owner per node

Lineage answers two operationally different questions. Backward tracing is debugging: given a misbehaving model, walk to its inputs and diff them against the previous version. Forward tracing is blast-radius analysis: given a corrupted upstream table, find every downstream dataset and model to retrain, quarantine or notify. Real platforms capture this either by instrumentation (jobs emit events describing inputs and outputs, as the OpenLineage specification standardises) or by parsing query plans and pipeline definitions. Accuracy varies: lineage captured from code you did not instrument, such as an analyst's ad-hoc notebook, is silently missing, so treat the graph as a lower bound.

Granularity is the cost knob. Dataset-level lineage is cheap and usually enough for incident response; column-level or row-level lineage needed for privacy or fairness audits is far more expensive to capture and store. A graph of millions of nodes is still small to store, but keeping it correct as pipelines change is the real work, which is why it usually belongs to the platform, not to every team's notebook.

Deep Dive: Versioning Prompts and Embeddings

Prompts are code-like:  version them, review them, tie them to an eval score.
   prompt_id=support_answer  v12  sha=ab34..   used with model="<provider model id, pinned>"  eval=recorded
   The pair (prompt version, model version) is the unit of behaviour; changing either changes outputs.

Embeddings are DERIVED DATA: vectors are only comparable within one (model, dims, preprocessing, chunking) tuple.

  index_v1 = embed( chunker@1, model_A, dims=1024, normalize=true )
  index_v2 = embed( chunker@2, model_B, dims=1024, normalize=true )    <- NOT compatible with v1 queries

  Never mix: query embedded by model_B against index built by model_A  ->  garbage similarity scores.
  Migration (blue/green):  build index_v2 in the background -> evaluate recall@k on a golden set
                           -> switch alias "active" from v1 to v2 -> keep v1 for rollback -> delete later.

  Illustrative storage (float32, 50M chunks x 1024 dims x 4 bytes):
    50,000,000 x 1024 x 4 = ~205 GB per index copy;  two copies during the switch = ~410 GB (before index overhead).

Prompts change behaviour as much as weights do, yet are often edited in a dashboard with no history. Treat them like code: store in git or a prompt registry, attach an evaluation result to each version, and log the prompt version with every request so a regression can be bisected (see prompt management). For hosted LLMs, floating model aliases can change underneath you depending on the provider; where a dated or pinned model identifier exists, prefer it for reproducibility and check provider docs for deprecation schedules.

Embeddings are the trap specific to AI backends. Because a vector only means something relative to the model that produced it, changing the embedding model, dimension, normalisation or even the chunking strategy means re-embedding the whole corpus and rebuilding the index in a vector database. That is a cost (compute proportional to total corpus tokens, plus double storage during cut-over) and a risk (ANN recall may shift), so it is done as a versioned, evaluated migration with an alias flip, not an in-place overwrite.

Key Pointers

• A model is a function of data, code, config, environment and seed; version all of them and link them in one manifest per registered model version.
• Content addressing (key = hash of bytes) gives immutability, integrity checks and deduplication; versions are small manifests pointing at shared blobs.
• Data snapshots need retention and garbage collection tied to registry references, and a plan for deletion requests that conflict with immutability.
• A registry maps name and version to an artifact plus metadata; stages or aliases are mutable pointers to immutable versions, moved through gated promotion.
• Rollback is a pointer move, but only works if the old artifact, its feature schema and its serving image are still available.
• Lineage is a DAG: trace backward to debug a model, forward from a bad input to find the blast radius; captured lineage is usually a lower bound.
• Version prompts with their eval results and pin model identifiers where possible; the unit of behaviour is prompt plus model.
• Embeddings are only comparable within one model/preprocessing tuple: changing any of them means a versioned re-embedding and index migration.

How It Works

1. The pipeline snapshots its training data (chunk hashes into a manifest, or a table-format snapshot id) and records that id, so the input can never change under the run.
2. The training job records code commit, config hash, container digest and seed, then writes the weights into a content-addressed artifact store.
3. A manifest tying data, code, config, weights and parent version together is registered under a new model version with its evaluation metrics.
4. Promotion moves a stage or alias pointer only after gates pass (metrics vs the incumbent, schema and safety checks), ideally through shadow or canary traffic.
5. Serving resolves a version to an immutable artifact hash and logs that version with each prediction, so any output can be traced to a manifest.
6. Lineage edges (input to output) accumulate across jobs; incident response queries the graph backward for the cause or forward for affected models, and rollback moves the pointer to the parent.

Try It Yourself

A standard-library sketch: a content-addressed store that dedups unchanged dataset chunks, a registry of manifests that tie dataset, code and config to model versions, stage promotion with an automatic archive of the previous production model, a lineage impact query and a rollback. Hashes are real SHA-256 prefixes; the "training" is faked since the point is provenance.

import hashlib, json

class CAS:
    """Content-addressed store: the key IS the SHA-256 of the bytes."""
    def __init__(self):
        self.blobs = {}
    def put(self, data: bytes) -> str:
        h = hashlib.sha256(data).hexdigest()
        self.blobs.setdefault(h, data)          # same bytes -> same key -> stored once
        return h
    def get(self, h: str) -> bytes:
        data = self.blobs[h]
        assert hashlib.sha256(data).hexdigest() == h, "corrupt blob"
        return data

cas = CAS()
short = lambda h: h[:8]

def snapshot(rows):
    """A dataset snapshot = manifest of row-chunk hashes; the snapshot id hashes the manifest."""
    chunks = [cas.put(json.dumps(rows[i:i+2]).encode()) for i in range(0, len(rows), 2)]
    manifest = json.dumps(chunks).encode()
    return cas.put(manifest), chunks

rows_v1 = [{"id": i, "label": i % 2} for i in range(6)]
rows_v2 = rows_v1[:5] + [{"id": 5, "label": 0}]        # one row relabelled
ds1, c1 = snapshot(rows_v1)
ds2, c2 = snapshot(rows_v2)
print("dataset v1", short(ds1), "chunks", [short(c) for c in c1])
print("dataset v2", short(ds2), "chunks", [short(c) for c in c2])
print("shared chunks (deduped):", len(set(c1) & set(c2)), "of", len(c1), "| blobs stored:", len(cas.blobs))

code = cas.put(b"def train(rows, lr): ...  # commit a1b2c3")
cfg1 = cas.put(json.dumps({"lr": 0.1, "epochs": 5}, sort_keys=True).encode())
cfg2 = cas.put(json.dumps({"lr": 0.05, "epochs": 5}, sort_keys=True).encode())

registry = {}     # model_version -> manifest
def register(name, ver, data_id, code_id, cfg_id, weights, parent=None):
    w = cas.put(weights)
    manifest = {"data": data_id, "code": code_id, "config": cfg_id, "weights": w, "parent": parent, "stage": "staging"}
    registry[f"{name}:{ver}"] = manifest
    return manifest

register("churn", 1, ds1, code, cfg1, b"weights-A")
register("churn", 2, ds2, code, cfg1, b"weights-B")
register("churn", 3, ds2, code, cfg2, b"weights-C", parent="churn:2")
for k, m in registry.items():
    print(k, "data", short(m["data"]), "code", short(m["code"]), "cfg", short(m["config"]), "weights", short(m["weights"]))

def promote(key, stage):
    if stage == "production":
        for k, m in registry.items():
            if m["stage"] == "production": m["stage"] = "archived"
    registry[key]["stage"] = stage

promote("churn:2", "production")
promote("churn:3", "production")          # v2 archived automatically
print("stages:", {k: m["stage"] for k, m in registry.items()})

def impacted_by(artifact_hash):
    return sorted(k for k, m in registry.items() if artifact_hash in (m["data"], m["code"], m["config"]))
print("bad dataset v2 impacts:", impacted_by(ds2))
print("bad dataset v1 impacts:", impacted_by(ds1))
print("cfg1 impacts          :", impacted_by(cfg1))

def rollback():
    prod = [k for k, m in registry.items() if m["stage"] == "production"][0]
    prev = registry[prod]["parent"]
    registry[prod]["stage"] = "archived"; registry[prev]["stage"] = "production"
    return prev
print("rollback ->", rollback(), {k: m["stage"] for k, m in registry.items()})

# reproducibility check: same inputs -> same manifest hash
h = lambda m: hashlib.sha256(json.dumps({k: m[k] for k in ("data","code","config")}, sort_keys=True).encode()).hexdigest()[:8]
print("inputs hash v2/v3:", h(registry["churn:2"]), h(registry["churn:3"]))

# Output:
# dataset v1 779d62e2 chunks ['fe05f9ea', 'b399af18', 'db0678f1']
# dataset v2 f9b26fe3 chunks ['fe05f9ea', 'b399af18', '78154345']
# shared chunks (deduped): 2 of 3 | blobs stored: 6
# churn:1 data 779d62e2 code d2a8a518 cfg 79022445 weights 7a2cf92f
# churn:2 data f9b26fe3 code d2a8a518 cfg 79022445 weights a9e542d5
# churn:3 data f9b26fe3 code d2a8a518 cfg 171f7e04 weights cebe82f0
# stages: {'churn:1': 'staging', 'churn:2': 'archived', 'churn:3': 'production'}
# bad dataset v2 impacts: ['churn:2', 'churn:3']
# bad dataset v1 impacts: ['churn:1']
# cfg1 impacts          : ['churn:1', 'churn:2']
# rollback -> churn:2 {'churn:1': 'staging', 'churn:2': 'production', 'churn:3': 'archived'}
# inputs hash v2/v3: 77f25d81 606940d8

Notice that dataset v2 shares two of three chunks with v1, so only one new chunk and one new manifest are stored (6 blobs, not 8), and that a bad v2 dataset impacts exactly the models whose manifests reference its hash. In real tools the same ideas are spelled differently; the commands below are illustrative and not executed here (flags vary by version):

# DVC (conceptual): track a dataset by content hash, keep the pointer in git
#   dvc add data/train.csv          -> writes data/train.csv.dvc (hash + size); the file itself goes to a DVC cache/remote
#   git add data/train.csv.dvc && git commit -m "dataset v2"
#   dvc checkout                    -> restores the data version that matches the current git commit
# MLflow (conceptual): log params/metrics/artifact, register a model version, then point an alias at it
#   mlflow.log_params(cfg); mlflow.log_metric("auc", auc); mlflow.register_model(uri, "churn")
# No output is shown because these need the tools installed; verify current syntax in their docs.

Interview Q&A

Q: Why is git alone not enough to reproduce an ML model?

A: The code commit fixes only one input. The training data usually lives outside git and changes as pipelines append or correct rows, hyperparameters and seeds may be in notebooks, and library or CUDA versions affect numerics. Reproducing a model needs the data snapshot id, config, container digest and seed recorded alongside the commit, otherwise checking out the same commit trains a different model.

Q: What is content-addressed storage and why does ML use it?

A: The storage key is the hash of the content, so identical bytes are stored once and a key can never refer to two different things. That gives immutability, cheap integrity verification by re-hashing, and deduplication across dataset versions, because a new version is just a small manifest referencing mostly existing chunks. The cost is that the first hash of very large data is I/O heavy and that immutable data needs an explicit garbage collection and retention policy.

Q: How would you design a model registry and promotion flow?

A: Store immutable versions with their manifest and metrics, and expose mutable pointers (stages or aliases such as production). Promotion is gated: compare offline metrics against the incumbent, check input schema and safety tests, then expose the candidate through shadow or canary traffic before moving the pointer, recording who or what did it. Serving pins to a version, never to "latest", so what runs is always known and rollback is just moving the pointer back.

Q: A source table was found to have a labelling bug from three weeks ago. What do you do?

A: Run a forward lineage query from the bad table to find every dataset, model version and endpoint that consumed data after the bug began, then filter to the ones in production or in use. Roll back or quarantine the affected models, retrain from a corrected snapshot, and notify downstream owners. The quality of the answer depends on how complete lineage capture is, so also state which consumers might be invisible (ad-hoc notebooks, uninstrumented jobs).

Q: Why can rolling back a model still break production?

A: The old model expects the feature schema and preprocessing it was trained with. If the feature pipeline or tokenizer has since changed, the rolled-back model receives different inputs and may silently degrade. Record feature and preprocessing versions in the manifest, keep the old serving image, and roll the dependent components back together, testing the rollback path before an incident.

Q: How do you version an embedding model change in a RAG system?

A: Treat the vector index as derived data keyed by model, dimension, preprocessing and chunker version. Build the new index in parallel (illustratively about 205 GB per copy for 50M chunks of 1024 float32 dims, so budget double storage), evaluate recall@k on a golden query set against the old index, then flip an alias and keep the old one for rollback. Mixing query vectors from the new model with the old index gives meaningless similarity scores.

Q: What should be stored for each prediction in production?

A: At minimum the model version (and prompt version for LLM calls), a request id, a timestamp and enough input reference to replay it, subject to privacy limits. That lets you attribute a complaint to a manifest, join outcomes back for monitoring, and compare versions during a canary. The trade-off is storage volume and PII exposure, so sample or hash inputs where policy requires.

Follow-up: How do you keep versioned data from growing without bound?

A: Define retention by reference: keep any snapshot referenced by a registered model version that is production or within a retention window, plus a sliding window of recent snapshots, and garbage-collect unreferenced chunks with a mark-and-sweep over manifests. Tiering old snapshots to cold storage is a middle option. The risk to manage is deleting a blob a rollback candidate still needs, so GC should be a dry-run-first, audited job.

Follow-up: A user invokes a right-to-erasure request; how does immutable versioning cope?

A: Immutability is a policy, not physics: you produce a new snapshot with the rows removed, delete the old blobs that contain them, and retrain or flag models trained on the old snapshot according to legal advice. Lineage tells you which models are affected. It is easiest if you designed for it, for example by keeping identifiers pseudonymous and separable, rather than discovering it during a request.

Follow-up: Two training runs with identical manifests give different metrics. Is versioning broken?

A: Not necessarily. The manifest proves the inputs match, so the difference comes from unrecorded sources of nondeterminism such as GPU reduction order, thread scheduling, data-loader shuffling or library differences. Pin seeds and enable deterministic algorithms where available (accepting a speed cost), record the hardware, and define an acceptable tolerance instead of expecting bit-exactness.

System Design Tie-in

In an ML platform design, the versioning layer sits between the data pipelines and serving: pipelines publish immutable dataset and feature snapshots, training jobs read a snapshot id and emit a manifest, the registry gates promotion, and the serving fleet pulls by version. Scale intuition to state as ESTIMATION with assumptions: a manifest is a few kilobytes, so even one million registered versions is only gigabytes of metadata, while the artifacts dominate (a few GB per large model version; assume 5 GB and 20 versions per model for 50 models is about 5 TB, which is trivial in object storage but worth a retention policy). Dataset storage depends on churn and chunk locality, as the 100 GB illustration above showed. For RAG systems add index versions and prompt versions to the manifest, and budget re-embedding as tokens times the provider's current price, or GPU-hours for a self-hosted encoder, plus double index storage during cut-over. Incident response and audits, not daily development, are what justify the investment, so the design should make lineage queries cheap: a graph or relational store keyed by artifact hash, fed by job events.

Trade-offs

The decision is how much provenance to capture and enforce, at what friction. Strict versioning (immutable snapshots, mandatory manifests, gated registry) costs storage, tooling and per-run overhead, and can slow exploratory work, so a common compromise is to keep notebooks loose while making anything that reaches staging or production go through the pipeline that emits the manifest automatically. Coarse dataset-level lineage is cheap and covers most incident response; column-level lineage is expensive and only worth it where regulation or fairness audits demand it. Tool choice matters less than discipline: git-style tools like DVC or lakeFS suit file and object datasets, table formats with time travel suit warehouse-style data, and a registry such as MLflow's covers models, but each has gaps that a homegrown manifest often fills. The wrong choice is heavy governance for a prototype that will be thrown away, or, at the other extreme, running a business-critical model with no answer to "what exactly is deployed and what trained it", which turns every incident into an archaeology project.