Lineage, Metadata & Reproducibility

The question every incident asks

Three very different people ask the same question. The on-call engineer, at 2 a.m., when ranking quality drops: what changed? The auditor, six months later, about a fraud decision: what did the model know when it decided? The data scientist, trying to beat last quarter's model: what exactly did the baseline train on? All three are asking for the same artifact — a complete, versioned description of the inputs to a training run — and most platforms cannot produce it, because the pieces live in five systems that never learned each other's identifiers.

Analytics lineage stops at the mart. ML lineage has to go two hops further: from the feature tables into the dataset version a model consumed, and from that dataset into the model run itself. This chapter is about making those two hops first-class, so the reproducibility promise from chapter 05 is something the platform enforces rather than something a diligent engineer remembers.

The reproducibility chain

Five links, each with a versioning mechanism that already exists on a well-run platform. The work is connecting them.

  ┌─────────────┐   git sha       ┌───────────────────┐   registry version   ┌──────────────────┐
  │  CODE       │ ──────────────▶ │  FEATURE DEFS      │ ───────────────────▶ │  FEATURE TABLES   │
  │  training   │                 │  host_reliability  │                      │  (Iceberg, SCD2)  │
  │  pipeline,  │                 │  _7d  v3           │                      │  snapshot 60198…  │
  │  preprocess │                 └───────────────────┘                      └─────────┬────────┘
  └──────┬──────┘                                                                      │ point-in-time join
         │                                                                             ▼
         │                        ┌───────────────────┐   manifest           ┌──────────────────┐
         │                        │  LABELS            │ ───────────────────▶ │  DATASET VERSION  │
         │                        │  fraud_confirmed   │                      │  fraud-signup/    │
         │                        │  snapshot 60198…   │                      │  v2026.03.01-a1f9c│
         │                        └───────────────────┘                      └─────────┬────────┘
         │                                                                             │
         ▼                                                                             ▼
  ┌──────────────────────────────────────────────────────────────────────────────────────────┐
  │  MODEL RUN  (MLflow run 7c2e…)                                                            │
  │    params · metrics · artifact (model + preprocess spec) · code sha · dataset version ·   │
  │    feature registry versions · label snapshot · env (container digest, CUDA, seed)         │
  └───────────────────────────────────┬──────────────────────────────────────────────────────┘
                                      │ promote
                                      ▼
  ┌──────────────────────────────────┐        ┌────────────────────────────────────┐
  │  MODEL REGISTRY  fraud-signup v14 │ ─────▶ │  SERVING  (feature log carries v14) │
  └──────────────────────────────────┘        └────────────────────────────────────┘

Read it top to bottom and it is a build. Read it bottom to top and it is an investigation: from a served prediction (logged with the model version) to the registry entry, to the run, to the dataset version, to the snapshots and feature definitions, to the source tables and the code. Every arrow is an identifier stored on the downstream side. That storage is the entire design.

Experiment tracking: what to record

Experiment trackers — MLflow, Weights & Biases, Neptune, Vertex/SageMaker experiments — record parameters, metrics and artifacts per run. Most teams log the first two carefully and the inputs casually ("dataset: features_v2"). The platform's job is to make the inputs precise and automatic, so a run cannot be logged without them.

Automate it. The training pipeline's entry point should resolve the dataset version, read feature versions off the manifest, capture the git sha and image digest, and attach all of it as run tags before a single epoch runs. A data scientist should be unable to produce an untraceable run from the platform's tooling; the notebook path is for exploration, and its runs are not promotable.

manifest = load_manifest("s3://griddp-ml-datasets/fraud-signup/v2026.03.01-a1f9c/manifest.json")

with mlflow.start_run(run_name="fraud-signup-xgb") as run:
    mlflow.set_tags({
        "dataset.name":        manifest["dataset"],
        "dataset.version":     manifest["version"],
        "dataset.schema_hash": manifest["schema_hash"],
        "features.versions":   json.dumps({k: v["registry_version"] for k, v in manifest["sources"].items()}),
        "labels.snapshot":     str(manifest["sources"]["labels.fraud_confirmed"]["iceberg_snapshot"]),
        "code.sha":            git_sha(), "code.dirty": str(git_dirty()),
        "env.image":           os.environ["IMAGE_DIGEST"], "env.cuda": cuda_version(),
        "seed":                str(SEED),
    })
    mlflow.log_input(mlflow.data.from_pandas(train_df, source=manifest["version"]), context="training")
    model = fit(train_df, params)                      # deterministic given seed + env
    mlflow.log_params(params); mlflow.log_metrics(evaluate(model, valid_df, test_df))
    mlflow.sklearn.log_model(model, "model", code_paths=["preprocess/"])   # transform ships with the model

The model registry as the join point

The registry (MLflow Model Registry, SageMaker/Vertex registries, Unity Catalog models) is where a run becomes a versioned, deployable thing with a lifecycle — staging, production, archived. It is also the natural join point for the chain: the registry version is what the serving path logs with every prediction, and the run it points to carries every upstream identifier. Two rules make it work.

• Promotion requires provenance. A run without a dataset version, feature versions and a code sha cannot be registered. Enforce it in the promotion step, not in a wiki.
• Serving logs the registry version. Every prediction, every logged feature vector (chapter 08) carries model=fraud-signup:v14. This is the link from a business outcome back into the chain.

Lineage graphs that reach the model

Data catalogs (DataHub, OpenMetadata, Unity Catalog, Atlan) build lineage graphs from job metadata: this dbt model reads those tables, that Spark job writes this one. The open standard for emitting those events is OpenLineage — a run-level event with input and output datasets, supported natively by Airflow, Dagster, Spark, Flink and dbt. The gap on most platforms is that the graph ends at the last table. Extend it two hops:

• Dataset builds emit lineage. The training-set job is a run whose inputs are the feature tables (at snapshot IDs) and the label table, and whose output is the dataset version. Emit the OpenLineage event with the snapshot IDs as facets.
• Training runs emit lineage. Input: the dataset version. Output: the model version. MLflow and the catalogs have integrations; if yours does not, a twenty-line emitter at the end of train.py is enough.
• Serving emits lineage. Input: the model version and the online feature tables. Output: the prediction log. Now the graph runs from source CDC topic to business decision.

With the graph complete, impact analysis works in both directions. Upstream: a schema change to host_agent.telemetry highlights every feature, dataset, model and product surface downstream — before the change ships. Downstream: a bad ranking decision traces to a model version, its dataset, and the specific telemetry rollup snapshot it trained on.

Reproducing a run a year later

The test of the chain is running it. Reproduction has three levels, and it is worth being precise about which one you promise.

Most audit and incident needs are met at retrieve and rebuild. Retrain matters when you must prove the process, not just the artifact — and it is the level where environment capture (image digest, CUDA version) stops being pedantry. Set retention policies from the level you promise: a fraud model with a seven-year regulatory tail needs its dataset exports kept seven years, which is a storage line item to budget, not an afterthought.

Worked example: "why did ranking change on Tuesday?"

Tuesday 09:40, the marketplace team notices offers from one region ranking lower than Monday. The investigation, with the chain in place: the search service logs show ranking:v22 — promoted Monday 18:00. Its MLflow run points at dataset rank-train/v2026.04.06-c81e; the manifest lists feature snapshots; the lineage graph shows features.price_vs_market at that snapshot was built from fct_offer_snapshot, which had a schema change on Sunday: the pricing service began emitting price_hr in the customer's currency instead of USD for one region. The feature was computed correctly on wrong inputs; v22 learned that region's offers were expensive. Fix: revert serving to v21 (registry alias), correct the source, rebuild the dataset from corrected snapshots, retrain, compare metrics against v22's run, promote. Elapsed: two hours. Without the chain the same investigation is a week of "what changed?" across five teams.

When to choose what

Next: the monitoring that catches bad inputs before they become bad models — 10 · Data Quality, Validation & Drift Monitoring.

✓ Check yourself

Answer from memory before opening each one.

1. Serving logs the model as fraud-signup:production. A month later a regulator asks which model made a decision in week 2. Why can you not answer, and what should the log hold?

   Show answer

   An alias is a pointer that moves when a new version is promoted, so the log no longer says which model scored the request. Log the resolved registry version for every prediction and use the alias only for routing. See the model registry as the join point.

2. Name the identifiers a training run must carry so its inputs can be reconstructed, and say who should attach them.

   Show answer

   Dataset name and version with a content hash, feature registry versions, the label snapshot and labelling policy, the git sha and dirty flag, the container digest with framework versions, and the seed. The platform's training entry point attaches them before training starts, so a scientist cannot produce an untraceable run. See experiment tracking.

3. Ranking quality drops on Tuesday and you have the full chain. Walk the investigation from the symptom to a source change, naming each hop.

   Show answer

   Read the logged model version from the serving log, open its run, then its dataset version and manifest, then the feature table snapshot, then lineage back to the source table where a schema change landed. Each arrow works because the downstream artifact stored the upstream identifier. See lineage graphs that reach the model.

4. A schema change is proposed on a source table. How does the lineage graph change the review, compared with lineage that stops at the last table?

   Show answer

   With the graph extended through datasets, runs and serving, you can list every downstream feature, dataset, model and product surface before the change ships. A graph that ends at the mart shows only tables, so model owners learn of it from an incident. See lineage graphs that reach the model.

5. A fraud model has a seven-year regulatory tail. Which reproduction level do you promise, and what do you have to keep for it?

   Show answer

   Promise retrieve and rebuild at least: the model artifact and metrics, the dataset export kept for seven years, and the snapshots or exports and feature versions needed to regenerate it. Promise retrain only if you must prove the process, and then capture image digest, CUDA version, seed and hardware class, accepting small GPU nondeterminism. See reproducing a run a year later.

6. What does a quarterly rebuild rehearsal do that dashboards and documentation do not?

   Show answer

   It rebuilds the dataset of a randomly chosen production model from its manifest and compares hashes. A match shows the chain works. An expired snapshot or missing feature version shows a break you can fix on your schedule, not an auditor's. See reproducing a run a year later.

Takeaway

• Reproducibility is a join across code, data, features and models on identifiers each registry records, not a saved notebook.
• Every arrow in the chain is an identifier stored on the downstream artifact, and the training entry point should attach them automatically.
• Gate registry promotion on provenance, and log the resolved model version at serving, never an alias.
• Extend lineage two hops past the mart, through dataset builds, training runs and serving, so impact analysis works upstream and downstream.
• Promise a reproduction level per model class, retrieve, rebuild or retrain, and set retention to match it.
• Rehearse a rebuild every quarter and treat it as a platform objective on the same dashboard as freshness.

A chain that is traceable still carries bad data if nothing checks it on the way in. → Data Quality, Validation & Drift Monitoring

Further reading

• MLflow Tracking — MLflow documentation, accessed 2026. Runs, parameters, tags, metrics and artifacts for recording a training run's inputs.
• MLflow Model Registry — MLflow documentation, accessed 2026. Model versions, aliases and lifecycle, and how a version links back to its run.
• OpenLineage object model — OpenLineage documentation, accessed 2026. Runs, jobs and datasets with facets, the event shape for lineage from builds and training.
• Hidden Technical Debt in Machine Learning Systems — Sculley et al., NeurIPS 2015. Why undeclared data dependencies make a model hard to debug or roll back.
• Model Cards for Model Reporting — Mitchell et al., FAT* 2019. A structure for documenting what a model was trained and evaluated on.