Training & Fine-Tuning Workflows

Common workflow patterns

Three workflow classes capture most ML work:

Pattern 1: Single-GPU experimentation

Pull a GPU instance, SSH in, run notebook or script, iterate. Best for early exploration. Marketplace providers excel here.

Pattern 2: Distributed training run

Launch a multi-node cluster, run distributed training (PyTorch DDP / FSDP, DeepSpeed, Megatron-LM), monitor, tear down. Dedicated providers with InfiniBand fabric required.

Pattern 3: Programmatic experiment management

Submit many experiments through an orchestrator (Ray Tune, MLflow, custom). Each runs on appropriate compute. Useful for sweeps and ablations.

Hyperparameter sweeps

For sweeping hyperparameters across many runs:

• Tooling: Ray Tune, Optuna, Weights & Biases Sweeps, MLflow.
• Compute: Often many smaller GPU instances in parallel. Marketplaces work well.
• Strategy: Successive halving / Hyperband to terminate underperforming trials early.
• Budget: Cap total compute; the sweep terminates when budget exhausted.

The DS work is choosing what to sweep (parameter ranges, objective). The infrastructure work is making the sweep run cleanly across heterogeneous provider supply.

Multi-node training

Multi-node distributed training requires:

• Coordinated cluster startup with all nodes available simultaneously.
• Inter-node network configuration (InfiniBand setup).
• Distributed-training framework (PyTorch DDP/FSDP, DeepSpeed, Megatron).
• Coordinated data loading (each node reads its shard).
• Distributed checkpointing.
• Graceful handling of node failures.

Dedicated providers (CoreWeave, Crusoe, Nebius, Lambda) provide reference cluster configurations. Setting one up well takes time; running a training campaign at scale takes infrastructure expertise alongside DS expertise.

Distillation

Distillation (teaching a smaller model from a larger one) is increasingly common as customers want production-affordable model sizes:

• Run inference on the teacher (using API or self-hosted) to generate training signals.
• Train the student model on the teacher's outputs.
• Evaluate that the student matches the teacher on the relevant metric.

Inference on the teacher is sometimes a meaningful cost. Together.AI or similar managed inference can be cheaper than running the teacher yourself.

Fine-tuning flavors

The common fine-tuning approaches differ in compute and quality:

• Full fine-tune: Update all parameters. Most compute. Best quality. Use when sufficient training data and serious customization needed.
• LoRA: Low-rank adapters; small fraction of parameters updated. Much less compute. Slight quality tradeoff. Default for many customization workloads.
• QLoRA: LoRA on quantized base model. Lowest compute. Quality good enough for many cases.
• RLHF / DPO / preference optimization: Reward-model-guided adjustment. Compute varies; complexity high.
• Continued pre-training: Domain adaptation via further pre-train on domain data. Significant compute; quality benefit depends on data.

The DS choice between flavors is driven by data availability, target quality, and budget.

Failure handling

For long-running training, failures are routine:

• Node failures (worst case for distributed training).
• Network partitions.
• OOM (out-of-memory) due to memory leak or workload growth.
• Hyperparameter divergence (loss explodes).
• Storage failures.

Defensive practices:

• Frequent checkpointing — at least every 30 minutes.
• Atomic checkpoint writes — partial writes shouldn't corrupt the latest.
• Distributed checkpointing for very large models.
• Resume-from-checkpoint as the default mode of starting any run.
• Monitoring on loss curves to catch divergence early.

✓ Check yourself

Answer from memory before opening each one.

1. A 32-node run loses one node nine hours in, and the last checkpoint is from hour six. What is lost, and which practices cap that loss?

   Show answer

   The run stops, because a node failure is the worst case for distributed training, and you redo the work since the last checkpoint, here three hours across all 32 nodes. Checkpoint at least every 30 minutes and make resume-from-checkpoint the default way to start any run. See failure handling.

2. A node dies halfway through writing a checkpoint. What goes wrong if the write is not atomic, and what is the fix?

   Show answer

   The newest checkpoint file is truncated, so the resume either fails or loads corrupt weights, and you may have overwritten the good one. Write to a temporary name and rename on completion, so the latest checkpoint is always whole. See failure handling.

3. You have 5,000 labeled examples, one 80 GB GPU and a deadline next week. You want to customize a large model. Which fine-tuning flavor do you start with?

   Show answer

   QLoRA, or LoRA on a smaller model. It updates a small fraction of parameters on a quantized base, so it fits one GPU and finishes fast, and the quality is good enough for many cases. A full fine-tune needs more data and compute than you have. See fine-tuning flavors.

4. A sweep uses successive halving. Why does it cut spend, and what does it assume that can fail?

   Show answer

   It runs many trials briefly, keeps the top fraction and gives only those more budget, so most trials stop early. It assumes early metrics rank trials the way final metrics would. Trials that start slowly but finish well, such as those with a slow warmup, are cut by mistake. See hyperparameter sweeps.

5. A distillation project is cheaper in training than expected but the bill is high. Where is the cost likely to sit, and what can you do about it?

   Show answer

   In inference on the teacher, which generates every training signal. Compare the price of managed inference with self-hosting the teacher at your utilization, and run it as batch work on cheap capacity. See distillation.

6. What does a multi-node run need that a single-GPU experiment does not?

   Show answer

   All nodes up at the same time, a configured inter-node network, a distributed-training framework, a data shard per node, distributed checkpointing and a plan for node failure. This is why a dedicated provider with a reference cluster configuration is the usual choice. See multi-node training.

Takeaway

• Most ML work fits three patterns: single-GPU experimentation, distributed training and programmatic experiment management.
• A sweep is a DS decision about ranges and objective plus an infrastructure job to run it cleanly on mixed supply, capped by a compute budget.
• Multi-node training needs simultaneous nodes, a fast inter-node network and distributed checkpointing, not just more GPUs.
• Distillation shifts cost to inference on the teacher, so price the teacher's inference before you start.
• Pick the fine-tuning flavor by data, target quality and budget; LoRA and QLoRA are the default for many jobs.
• Plan for failure: checkpoint at least every 30 minutes, write atomically and start every run as a resume.

Long runs on shifting supply are only useful if you can track and reproduce them. → Experiment Tracking & Reproducibility

Further reading

• LoRA: Low-Rank Adaptation of Large Language Models — Hu et al., 2021. The method behind adapter fine-tuning, with the parameter and memory savings.
• QLoRA: Efficient Finetuning of Quantized LLMs — Dettmers et al., 2023. How fine-tuning on a 4-bit base fits large models on a single GPU.
• Hyperband — Li et al., 2016. The early-stopping bandit method behind most budgeted sweeps.
• Distilling the Knowledge in a Neural Network — Hinton et al., 2015. The original statement of teacher-student distillation.
• Distributed checkpoint — PyTorch documentation. Saving and resuming sharded state across many ranks.
• The Llama 3 Herd of Models — Meta, 2024. Includes an account of interruptions and hardware faults on a very large training cluster.