Labels, Ground Truth & Feedback Loops

Labels are data with latency and bias

Features get most of the architectural attention; labels get a column name. Yet the label pipeline decides what the model can learn at all, and it has two properties features don't. Latency: the truth about a prediction arrives later — a click within seconds, a rental's actual duration in hours, a chargeback in up to 120 days, "did the forecast hold" in a week. Bias: you only observe outcomes for the actions you took. GridDP's fraud model sees chargebacks only on payments it approved; the ranker sees clicks only on listings it showed. The model's own decisions carve the shape of tomorrow's training data.

Treating labels as "whatever column happens to be there" produces models that are confidently wrong about the part of the world they never see. Treating them as a pipeline with an explicit delay, a censoring rule and a bias-correction strategy is what this chapter is about.

Where labels come from

Most real models blend these: authoritative delayed labels for training, implicit signals for fast proxy metrics, human review to label the band where the model is unsure, and weak labels to bootstrap a cold-start. The architecture question is how those blend without contaminating each other — which needs a label store with provenance, below.

Delay, censoring and the label window

Chargebacks can be filed up to 120 days after a payment, with most arriving in the first 45. A payment from 20 days ago has no chargeback yet; calling it a negative is the most common label bug in fraud systems, and it biases the model toward approving recent-looking behaviour. The row is censored, not negative.

  payment at T                                             today
  │◀────────────── label window (90 d) ──────────────▶│
  │                                                    │
  ├─ chargeback filed at T+31 ─▶ label = 1 (known at T+31)
  ├─ no chargeback by T+90     ─▶ label = 0 (known only at T+90)
  ├─ today < T+90              ─▶ label = NULL (censored; exclude or model survival)
  │
  practical rule: a row is trainable only when  now() ≥ prediction_ts + window
  → the newest 90 days of the spine are unlabeled, always. Design for it.

Three consequences for the pipeline. The label table must be append-only with a known_at timestamp, so a training set built on 1 June and one built on 1 September can both be reproduced (a label that flipped from 0 to 1 on 15 July is two rows, not an update). The training window must end at least one label-window before the build date, and the validation split should be the most recent mature period, not the most recent period. And for models where waiting 90 days is unacceptable, use a shorter proxy label (chargeback within 30 days captures ~70% of eventual chargebacks at GridDP) and track its correlation with the true label so you know when the proxy drifts.

Survivorship bias and the feedback loop

The fraud model approves 97% of payments and declines 3%. Chargebacks — the only fraud signal — exist only for the 97%. The declined 3% have no label ever. Retrain on this and the model learns that the patterns it already declines are "not fraud" (it never sees them lose money), and it slowly drifts toward approving them. Rankers have the same problem in mirror image: a listing that is never shown is never clicked, so it looks unattractive forever.

Whichever you use, the requirement on the data plane is the same: log the decision, the score, the policy version and the propensity alongside every prediction. Without that log, no correction is possible after the fact. This is the same served-vector log that chapter 08 needs for skew analysis — one table, several consumers.

Human-in-the-loop and weak supervision

Human labels are the most accurate and the most expensive, so route them where the model is least certain. A review queue that pulls payments with scores in the 0.35–0.65 band and returns labels within 24 hours gives you high-value examples at the decision boundary and a fast proxy for delayed chargebacks. Engineering requirements: a queue with assignment and SLA, a labeling UI that shows the served feature vector (so the reviewer and the model see the same thing), reviewer identity and rubric version on every label (reviewers disagree, rubrics change), and periodic double-labeling to measure inter-annotator agreement — if two analysts agree on 80% of cases, your label noise floor is 20%, and the model's precision can't be trusted above it.

Weak supervision goes the other way: many cheap, noisy labeling functions ("more than three distinct cards in ten minutes", "IP in a known proxy range", "email domain created this week") combined by a label model that estimates each function's accuracy from their agreements. It is the right tool for cold starts and for classes too rare to review, and the wrong tool as a permanent replacement for ground truth — its errors are correlated, and the model will learn the heuristics rather than the fraud.

The label store

The pipeline above needs a home. A label store is a small, opinionated table family, not a product:

CREATE TABLE ml.labels (
  task          STRING,      -- 'fraud_chargeback_90d', 'ranking_click', 'capacity_actual_7d'
  entity_id     STRING,      -- payment_id, (session_id, listing_id), (region, gpu_model, hour)
  prediction_ts TIMESTAMP,   -- the decision moment this label belongs to
  label         DOUBLE,      -- 0/1, a value, or a rank position
  known_at      TIMESTAMP,   -- when this label became known — append-only
  source        STRING,      -- 'chargeback_feed' | 'reviewer' | 'weak:lf_card_velocity' | 'exploration'
  source_ref    STRING,      -- chargeback_id, reviewer_id + rubric@v, labeling-function version
  weight        DOUBLE       -- propensity / sampling weight, 1.0 by default
) PARTITIONED BY (task, days(known_at));

-- current label for a task, as of a build date (reproducible):
SELECT entity_id, prediction_ts, label
FROM ml.labels
WHERE task = 'fraud_chargeback_90d' AND known_at <= :build_date
QUALIFY ROW_NUMBER() OVER (PARTITION BY entity_id ORDER BY known_at DESC, source_rank(source)) = 1;

source is the column that keeps blends honest: a training set can choose "authoritative only", "authoritative + reviewer", or "everything with weights", and the manifest (ch. 02) records which. When a reviewer rubric changes, the old labels keep their source_ref and can be excluded or re-weighted rather than silently mixed.

Worked example: the ranker's labels

GridDP's ranker predicts which listings a renter will rent. The implicit label is "rented after being shown". The spine is (search_id, listing_id, position, shown_at); the label is rented_within_24h, known at most 24 hours later — fast, plentiful, and biased three ways: position (row 1 gets clicked more regardless of quality), exposure (unshown listings have no rows), and the loop (the current ranker decides what's shown). The label pipeline therefore logs position and the ranker's score and policy_version with every impression, reserves two of the ten slots on 5% of searches for randomly sampled eligible listings (exploration), and stores the label with weight = 1/propensity for those slots. Offline evaluation uses the exploration slice only; training uses everything with weights. Nothing about this is a modeling trick — it is table design plus a serving-side logging contract, which is why it lives in this guide.

When to choose what

With spine and labels in hand, the columns that do the work: 04 · Feature Engineering at Scale & the Feature Store.

✓ Check yourself

Answer from memory before opening each one.

1. A fraud model approves 97% of payments and declines 3%. Retrained on chargebacks, it slowly starts approving patterns it used to decline. Explain the mechanism.

   Show answer

   Chargebacks exist only for approved payments, so declined payments never get a label. The model never sees those patterns lose money, so it learns they are safe, and its own decisions have shaped its training data. See survivorship bias and the feedback loop.

2. Your label table has one row per payment and you update the label to 1 when a chargeback arrives on day 45. Why can you no longer rebuild last quarter's training set, and what is the fix?

   Show answer

   The update destroys what was known on any earlier build date, and it hides how label maturity behaves. Make the table append-only with a known_at timestamp, so a label that flipped is two rows, and derive the current label with a window as of the build date. See the label store.

3. Chargebacks arrive up to 120 days after a payment. Your validation split is the most recent three months of data. What is wrong with it, and what do you use?

   Show answer

   The newest three months are mostly censored, so many true fraud cases still look negative and validation understates the positive rate. Validate on the most recent mature period, one label window before the build date. A short proxy label can supplement it if you track its agreement with the real one. See delay, censoring and the label window.

4. The ranker only logs which listing was clicked. A year later you want to correct for position and for listings never shown. What should have been logged at serving time, and why can you not add it afterwards?

   Show answer

   Log the position, the score, the policy version and the probability the action was taken, with every impression, and reserve some slots for random listings. Propensities describe the policy at the moment of the decision, so they cannot be reconstructed later. See survivorship bias and the feedback loop.

5. A review queue labels payments scored between 0.35 and 0.65. Two reviewers disagree on 25% of cases after a rubric change. What do you do, and why not just average the labels?

   Show answer

   Version the rubric on every label, measure agreement with a double-labelled sample, and keep old labels excluded or re-weighted by source instead of mixing them. Averaging hides a systematic shift that came from the rubric, not from the data. See human-in-the-loop and weak supervision.

6. A new product has no ground truth, and an engineer proposes weak labels from heuristics for good. When is that right, and what limits it?

   Show answer

   It is right for a cold start and for classes too rare to review, with a plan to replace it as real labels arrive. Labelling functions make correlated errors, so the model learns the heuristics rather than the fraud. Keep the source on each label so training can exclude them later. See human-in-the-loop and weak supervision.

Takeaway

• Labels are data with latency and bias, and the label pipeline decides what the model can learn at all.
• A row whose label window has not closed is censored, not negative, and a row is trainable only when now is past prediction time plus the window.
• Keep labels append-only with known_at and source, so any past dataset can be rebuilt and blended sources stay separable.
• You only observe outcomes for the actions you took, so a model trained on its own decisions drifts toward approving or showing what it already did.
• Exploration traffic and propensity logging must start on day one, because they cannot be added retroactively.
• Human review belongs in the uncertain band with rubric versions and agreement checks, and weak supervision is a way to bootstrap, not a permanent replacement for ground truth.

With spine and labels fixed, the next question is where the feature columns come from and how one definition serves both paths. → Feature Engineering at Scale & the Feature Store

Further reading

• Data Programming: Creating Large Training Sets, Quickly — Ratner et al., NeurIPS 2016. The method behind combining noisy labelling functions with a label model.
• Unbiased Learning-to-Rank with Biased Feedback — Joachims et al., 2016. Inverse propensity weighting to correct position bias in click data.
• Rules of Machine Learning: Best Practices for ML Engineering — Google, accessed 2026. Rules on feedback loops and on logging what the model saw at serving time.