Production ML

The "end-to-end ownership" bar

the JD: "This is a full-stack data science role, involving model development, analysis, and writing production code." And: "Write production-ready code that can be relied on for real-time decision making by our partners."

What that means in interviews: you'll be asked about how your code looks, how it's tested, how it deploys, how it's monitored, and what happens when it breaks. Not just "show me an AUC curve."

Production code quality

The dividing line between research code and production code:

Modules, not notebooks

Code lives in .py files, organized into modules. Notebooks are exploration; the deliverable is importable code.

Types

Type hints everywhere. Modern Python: list[int], dict[str, float], Optional[X]. mypy or pyright in CI.

Small functions

One responsibility per function. Pure where possible. Mocking and testing become easy when functions don't have hidden dependencies.

Configuration over hard-coded values

Thresholds, model paths, feature flags — externalize. Read from environment or a config file. Don't edit the model_path = "..." line every deploy.

Error handling

Be explicit about what fails how. Validate inputs at module boundaries; trust within the module. Don't catch broadly when you should let the failure propagate.

Logging

Structured logging (JSON) at appropriate levels. INFO for normal operation, WARNING for unusual but handled, ERROR for failures that need attention. Include correlation IDs so a single request can be traced across services.

import logging
from dataclasses import dataclass
from typing import Optional

logger = logging.getLogger(__name__)

@dataclass(frozen=True)
class ScoringResult:
    score: float
    decision: str
    model_version: str
    features_used: dict[str, float]
    correlation_id: str

class Scorer:
    def __init__(self, model, threshold: float, version: str):
        self.model = model
        self.threshold = threshold
        self.version = version

    def score(self, features: dict[str, float], correlation_id: str) -> ScoringResult:
        if not features:
            raise ValueError("empty features dict")
        probs = self.model.predict_proba([list(features.values())])[0]
        score = float(probs[1])
        decision = "flag" if score >= self.threshold else "pass"
        logger.info("scored", extra={
            "correlation_id": correlation_id,
            "score": score,
            "decision": decision,
            "model_version": self.version,
        })
        return ScoringResult(score, decision, self.version, features, correlation_id)

Testing

Production ML needs three layers of tests:

Unit tests

Each function tested in isolation. Feature transformations, scoring logic, threshold decisions. Use pytest. Aim for high coverage on critical paths.

Integration tests

The scoring pipeline end-to-end on a known input. "Given this raw application, the service produces this score." Catches issues at module boundaries that unit tests miss.

Model-quality regression tests

Special to ML. Hold a frozen labeled test set. On every model change, score the test set and require AUC (or lift at K%) to be at least X. Prevents accidental quality regressions from deploy.

Real-time inference

the company's APIs run in the milliseconds. Patterns that matter:

Feature precomputation

For features that can't be computed in the request budget (cross-entity aggregates, graph features), precompute and cache. The request looks up the cached value.

Model serialization

Don't load Pickled models per request. Load once at service startup; keep the model in memory; reuse across requests.

Batching

When latency permits, batch multiple requests into one model call. Gradient boosting and neural network inference benefit substantially from batching.

Cold-path vs hot-path

Some features need synchronous lookup (call a bureau API). Some don't (lookup in a precomputed feature store). Separate the two paths; budget the latency accordingly.

Fallbacks

If a downstream feature service times out, return a conservative default rather than failing the request. The decision has to happen; missing a feature isn't a reason to deny the application.

Monitoring

Four layers of monitoring for a deployed model:

1. Service health

Latency p50/p95/p99, error rate, throughput. SRE-style metrics, alertable.

2. Input drift

PSI per feature in a rolling window. Alert when distribution shifts beyond a threshold. Catches upstream data issues fast.

3. Output drift

Score distribution shifts. Mean, median, P95, fraction above the threshold. Alert when these drift past historical bands.

4. Outcome metrics

When labels arrive (chargebacks, manual reviews), score the model on the new labels and confirm AUC, calibration, and lift haven't degraded. This is the slow but ground-truth signal.

The dashboard

A senior DS owns the model's monitoring dashboard. It shows all four layers on one page, with thresholds and recent alerts. Stakeholders can glance at it and see model health.

Retraining cadence

How often to retrain depends on the rate of drift:

• Stable problems (legal-text classification, equipment-failure prediction): quarterly retrain is plenty.
• Moderate drift (lending propensity, churn): monthly.
• High drift / adversarial (fraud detection): weekly to bi-weekly. fraud-detection-scale.
• Continuous: online learning with frequent micro-updates. Rare in production fraud because validation rigor is harder; usually a batch retrain on a fast cycle is sufficient.

Triggered retraining

Beyond cadence, retrain when monitoring triggers fire: significant input drift, output drift, outcome regression. Build the path for triggered retraining alongside the scheduled one.

Rollouts & canaries

A new model isn't a deploy you flip in one shot. Stages:

Shadow mode

New model scores every request alongside the current model. Decisions still made by the current model. Compare new scores to old; check for unexpected behavior. Run for days to weeks.

Canary

New model makes decisions on a small slice (1–5%) of traffic, in addition to its shadow scoring on the rest. Monitor outcome metrics on the canary. Promote to full rollout if metrics hold.

A/B rollout

50/50 split. Compare outcome metrics. Promote winner. Useful when you want a clean lift estimate, not just a quality regression check.

Full rollout

New model is the decision-maker. Old model kept in shadow for a while for comparison and easy revert.

On-call considerations

A production ML service is on the on-call rotation if it's serving live decisions. The senior DS owns:

• Runbooks: when X alert fires, do Y. Covers common failures (input feature stale, model not loaded, downstream timeout).
• Rollback procedure: how to revert to the previous model version in under a minute.
• Escalation: who do you wake up at 3 AM? When?
• Post-mortems: every incident gets one. Root cause, contributing factors, action items, and (critically) what evaluation gap let the issue ship.

Interview probes

✓ Check yourself

Answer from memory before opening each one.

1. A teammate changes a feature transformation. Unit tests pass and the service returns scores. Which test catches a silent loss of model quality, and how does it work?

   Show answer

   A model-quality regression test. Keep a frozen labelled set, score it on every change and require AUC or lift at the top K% to stay above a floor. Unit and integration tests prove the code runs, not that the model is still good. See testing.

2. Profiling a scoring API shows p99 latency of 400 ms against a 100 ms budget, and the model call takes 4 ms. Where do you look?

   Show answer

   At feature lookup, serialisation and any synchronous call to a slow downstream. Precompute heavy features into a fast store, load the model once at startup instead of per request, and move slow calls off the hot path with a tight timeout and a fallback. See real-time inference.

3. The feature service times out on one request. Should the scorer deny the application or return a default? What decides it?

   Show answer

   Return a conservative default, because a decision has to be made and a missing feature is not a reason to deny. What counts as conservative depends on the relative cost of a false approval and a false denial. Log the fallback and watch its rate. See real-time inference.

4. Latency, error rate, input drift and score distribution all look normal, yet fraud losses are rising. Which layer would show it, and why does it arrive last?

   Show answer

   Outcome metrics. Input and output monitoring see only distribution shifts, and a change in the feature-to-target relationship can leave both flat. Outcomes need labels, which arrive late, so AUC, calibration and lift on recent matured labels are the slow but decisive signal. See monitoring.

5. A shadow run shows the new model’s scores look sensible. Why not go straight to full rollout, and what does each later stage tell you that the one before cannot?

   Show answer

   Shadow scoring never affects a decision, so it reveals no outcomes. A canary on 1 to 5% of traffic shows real outcomes with limited harm. An A/B split gives a clean lift estimate. Skip stages only when the stakes and the label delay allow it. See rollouts and canaries.

6. Your fraud model retrains monthly. Why add triggered retraining, and what should trigger it?

   Show answer

   A schedule cannot react to an abrupt shift between runs. Trigger on significant input drift, output drift or an outcome regression, and build that path next to the scheduled one. Fraud drifts fast enough that a weekly to bi-weekly cadence is typical. See retraining cadence.

Takeaway

• Production code means modules, type hints, small functions, external configuration, explicit error handling and structured logs.
• Test at three levels: unit, integration on a known input and a model-quality regression test on a frozen labelled set.
• Meet a millisecond budget by precomputing expensive features, loading the model once, avoiding slow synchronous calls and returning a fallback.
• Monitor four layers: service health, input drift, output drift and outcome metrics, the last being slow but decisive.
• Retrain on a cadence that matches drift and also on triggers from monitoring.
• Roll out through shadow, canary and A/B stages with a fast rollback, and write a post-mortem that names the evaluation gap.

The same standards show up under time pressure in live-coding rounds, which the next chapter covers. → Coding Fundamentals

Further reading

• Rules of Machine Learning — Google, accessed 2026. Practical rules on serving skew, monitoring and launch discipline
• The ML Test Score — Breck et al., Google, 2017. A rubric of tests and monitors for production readiness
• Postmortem Culture: Learning from Failure — Google SRE book. How to run blameless post-mortems that produce action items
• Logging Cookbook — Python documentation. Structured logging and passing correlation context through log records
• Model persistence — scikit-learn documentation. Why loading pickles from untrusted sources is unsafe and what to use instead
• pytest documentation — pytest project. Fixtures, parametrisation and layout for unit and integration tests