Section B · DS using neoclouds

Experiment Tracking & Reproducibility

How to track ML experiments across heterogeneous neocloud infrastructure. The tools, the metadata that matters, and the binding of runs to compute / data / code that makes work reproducible months later.

Why this matters more on neoclouds

Experiment tracking matters more when:

Compute is heterogeneous (different providers, different GPU SKUs, different cluster configurations).
Workloads run on ephemeral infrastructure (instance gone, no way to reconstruct from logs alone).
Many people are running many experiments in parallel.
Cost attribution matters and requires linking experiments to compute spend.

All of these are typical on neoclouds. Bad experiment tracking results in research that can't be reproduced and production models with no traceable lineage.

Tools

MLflow: Open-source, broadly compatible. Standard for many shops.
Weights & Biases (W&B): Hosted; rich visualizations; widely adopted in modern ML.
Comet ML: Hosted; similar feature set to W&B.
Neptune.ai: Hosted with strong metadata capabilities.
Aim: Open-source; lightweight.
TensorBoard: Built into PyTorch/TensorFlow; useful but less of a full tracking solution.

Most modern teams use W&B or MLflow as the tracking surface. Pick the one your org standardizes on; the differences are smaller than the value of consistency.

What to track

Each run should record:

Configuration: All hyperparameters and configuration values.
Code version: Git SHA of the training code; also the framework versions.
Data version: Dataset version / hash / path used.
Compute environment: Provider, region, SKU, cluster size, container image.
Metrics: Training metrics (loss, etc.) over time and final evaluation metrics.
Artifacts: Saved checkpoints, eval outputs, sample generations.
Cost: Compute hours used; estimated cost.
Notes / tags: Human context — what hypothesis being tested.

The configuration and metric tracking is straightforward; binding to compute and cost requires explicit work.

Binding runs to compute, data, code

The hard part of reproducibility is the joining keys. Strategies:

Code

Either run from a clean Git working directory and log the SHA, or log the SHA plus a diff of any uncommitted changes. The "clean working directory" discipline is best practice but routinely violated.

Data

Log the dataset version explicitly. If the dataset is content-addressed (hash), log the hash. If it's path-based, log the immutable path (training data must not be mutated after a run uses it).

Compute

The job submission layer logs provider, instance type, cluster size. The training code logs the actually-observed configuration (sometimes different from requested).

Cost

Either query the provider's billing API after the run, or estimate from instance type × runtime. Production-grade attribution comes from the DE-side billing pipelines (see DE guide).

Reproducibility checklist

Can you reproduce a 3-month-old training run? If yes, you have:

The Git SHA of the training code.
The container image used (with frameworks pinned).
The dataset version and path.
The complete hyperparameter configuration.
The cluster configuration (size, GPU type, network).
Confidence that randomness is seeded.
Confidence that floating-point determinism (where required) is enabled.

If any of these are missing, reproducibility breaks. In practice, perfect reproducibility is rare for distributed training (floating-point determinism is hard), but coming close is achievable.

✓ Check yourself

Answer from memory before opening each one.

Six months after a run, a production model regresses and the training instance is gone. Why is the instance's own log not enough, and what should have been recorded elsewhere?

Show answer
Ephemeral infrastructure takes its logs with it, and a heterogeneous fleet means the same config can behave differently across SKUs. Record the provider, region, SKU, cluster size and container image with the run in a tracking system. See why this matters and what to track.
A run started from a working directory with uncommitted edits. You have the Git SHA. Can you reproduce the run?

Show answer
Not reliably, because the SHA does not contain the edits. Log the SHA plus a diff of uncommitted changes, or enforce a clean working directory. See binding runs to code.
Someone refreshes the training file at the same path that three old runs logged. What breaks, and what rule prevents it?

Show answer
The logged path now points to different data, so those runs can no longer be reproduced or compared. Treat training data as immutable once a run uses it, and log a content hash or immutable path. See binding runs to data.
You requested 64 GPUs and the job started with 60. Which record do you trust, and what do you log?

Show answer
Log both. The submission layer records what you asked for, and the training code records what it actually observed, which can differ. Reproduce from the observed value. See binding runs to compute.
You rerun a 128-GPU job with the same SHA, image, data, config and seeds, and the loss curve is slightly different. Is the record incomplete?

Show answer
Not necessarily. Floating-point reductions across many devices are not deterministic unless you ask for it, so perfect reproducibility is rare for distributed training. Decide a tolerance from run-to-run variance, and enable deterministic modes only where you need exact equality. See the reproducibility checklist.
Your org argues between MLflow and Weights & Biases. What decides it?

Show answer
Consistency. The differences are smaller than the value of everyone logging to one place. Exceptions: choose the open-source, self-hostable option if data cannot leave your environment. See tools.

Takeaway

Tracking matters more on neoclouds because compute is heterogeneous and ephemeral, so the instance cannot explain a run later.
Each run records its config, code version, data version, compute environment, metrics, artifacts, cost and a note on the hypothesis.
Reproducibility lives in the joining keys: Git SHA plus any diff, an immutable dataset hash or path, and requested and observed compute.
Cost attribution needs explicit work, either the provider's billing data or instance type times runtime.
A three-month-old run is reproducible if you have the SHA, pinned image, data version, full config, cluster configuration and seeds.
Exact reproducibility is rare for distributed training, but you can get within the normal run-to-run variance.

Tracked runs also show where the money goes, which is the next constraint. → Cost-Aware ML Experimentation