Training & Fine-Tuning Workflows

Common workflow patterns

Three workflow classes capture most ML work:

Pattern 1: Single-GPU experimentation

Pull a GPU instance, SSH in, run notebook or script, iterate. Best for early exploration. Marketplace providers excel here.

Pattern 2: Distributed training run

Launch a multi-node cluster, run distributed training (PyTorch DDP / FSDP, DeepSpeed, Megatron-LM), monitor, tear down. Dedicated providers with InfiniBand fabric required.

Pattern 3: Programmatic experiment management

Submit many experiments through an orchestrator (Ray Tune, MLflow, custom). Each runs on appropriate compute. Useful for sweeps and ablations.

Hyperparameter sweeps

For sweeping hyperparameters across many runs:

• Tooling: Ray Tune, Optuna, Weights & Biases Sweeps, MLflow.
• Compute: Often many smaller GPU instances in parallel. Marketplaces work well.
• Strategy: Successive halving / Hyperband to terminate underperforming trials early.
• Budget: Cap total compute; the sweep terminates when budget exhausted.

The DS work is choosing what to sweep (parameter ranges, objective). The infrastructure work is making the sweep run cleanly across heterogeneous provider supply.

Multi-node training

Multi-node distributed training requires:

• Coordinated cluster startup with all nodes available simultaneously.
• Inter-node network configuration (InfiniBand setup).
• Distributed-training framework (PyTorch DDP/FSDP, DeepSpeed, Megatron).
• Coordinated data loading (each node reads its shard).
• Distributed checkpointing.
• Graceful handling of node failures.

Dedicated providers (CoreWeave, Crusoe, Nebius, Lambda) provide reference cluster configurations. Setting one up well takes time; running a training campaign at scale takes infrastructure expertise alongside DS expertise.

Distillation

Distillation (teaching a smaller model from a larger one) is increasingly common as customers want production-affordable model sizes:

• Run inference on the teacher (using API or self-hosted) to generate training signals.
• Train the student model on the teacher's outputs.
• Evaluate that the student matches the teacher on the relevant metric.

Inference on the teacher is sometimes a meaningful cost. Together.AI or similar managed inference can be cheaper than running the teacher yourself.

Fine-tuning flavors

The common fine-tuning approaches differ in compute and quality:

• Full fine-tune: Update all parameters. Most compute. Best quality. Use when sufficient training data and serious customization needed.
• LoRA: Low-rank adapters; small fraction of parameters updated. Much less compute. Slight quality tradeoff. Default for many customization workloads.
• QLoRA: LoRA on quantized base model. Lowest compute. Quality good enough for many cases.
• RLHF / DPO / preference optimization: Reward-model-guided adjustment. Compute varies; complexity high.
• Continued pre-training: Domain adaptation via further pre-train on domain data. Significant compute; quality benefit depends on data.

The DS choice between flavors is driven by data availability, target quality, and budget.

Failure handling

For long-running training, failures are routine:

• Node failures (worst case for distributed training).
• Network partitions.
• OOM (out-of-memory) due to memory leak or workload growth.
• Hyperparameter divergence (loss explodes).
• Storage failures.

Defensive practices:

• Frequent checkpointing — at least every 30 minutes.
• Atomic checkpoint writes — partial writes shouldn't corrupt the latest.
• Distributed checkpointing for very large models.
• Resume-from-checkpoint as the default mode of starting any run.
• Monitoring on loss curves to catch divergence early.

Takeaway

Training and fine-tuning workflows on neocloud infrastructure require DS instinct, framework expertise, and operational discipline in equal measure. The next chapter goes into experiment tracking — the infrastructure that makes the work reproducible.