Experiment Tracking & Reproducibility

Why this matters more on neoclouds

Experiment tracking matters more when:

• Compute is heterogeneous (different providers, different GPU SKUs, different cluster configurations).
• Workloads run on ephemeral infrastructure (instance gone, no way to reconstruct from logs alone).
• Many people are running many experiments in parallel.
• Cost attribution matters and requires linking experiments to compute spend.

All of these are typical on neoclouds. Bad experiment tracking results in research that can't be reproduced and production models with no traceable lineage.

Tools

• MLflow: Open-source, broadly compatible. Standard for many shops.
• Weights & Biases (W&B): Hosted; rich visualizations; widely adopted in modern ML.
• Comet ML: Hosted; similar feature set to W&B.
• Neptune.ai: Hosted with strong metadata capabilities.
• Aim: Open-source; lightweight.
• TensorBoard: Built into PyTorch/TensorFlow; useful but less of a full tracking solution.

Most modern teams use W&B or MLflow as the tracking surface. Pick the one your org standardizes on; the differences are smaller than the value of consistency.

What to track

Each run should record:

• Configuration: All hyperparameters and configuration values.
• Code version: Git SHA of the training code; also the framework versions.
• Data version: Dataset version / hash / path used.
• Compute environment: Provider, region, SKU, cluster size, container image.
• Metrics: Training metrics (loss, etc.) over time and final evaluation metrics.
• Artifacts: Saved checkpoints, eval outputs, sample generations.
• Cost: Compute hours used; estimated cost.
• Notes / tags: Human context — what hypothesis being tested.

The configuration and metric tracking is straightforward; binding to compute and cost requires explicit work.

Binding runs to compute, data, code

The hard part of reproducibility is the joining keys. Strategies:

Code

Either run from a clean Git working directory and log the SHA, or log the SHA plus a diff of any uncommitted changes. The "clean working directory" discipline is best practice but routinely violated.

Data

Log the dataset version explicitly. If the dataset is content-addressed (hash), log the hash. If it's path-based, log the immutable path (training data must not be mutated after a run uses it).

Compute

The job submission layer logs provider, instance type, cluster size. The training code logs the actually-observed configuration (sometimes different from requested).

Cost

Either query the provider's billing API after the run, or estimate from instance type × runtime. Production-grade attribution comes from the DE-side billing pipelines (see DE guide).

Reproducibility checklist

Can you reproduce a 3-month-old training run? If yes, you have:

1. The Git SHA of the training code.
2. The container image used (with frameworks pinned).
3. The dataset version and path.
4. The complete hyperparameter configuration.
5. The cluster configuration (size, GPU type, network).
6. Confidence that randomness is seeded.
7. Confidence that floating-point determinism (where required) is enabled.

If any of these are missing, reproducibility breaks. In practice, perfect reproducibility is rare for distributed training (floating-point determinism is hard), but coming close is achievable.

Takeaway

Experiment tracking and reproducibility are the analytical hygiene that lets DS teams operate confidently at scale. Without them, knowledge gained is fragile. The next chapter addresses cost-aware experimentation — managing the budget side of ML work.