Section F · Production and debugging

The System You Shipped

The behavioural spine of every loop: how to walk through an AI system you built, its worst production failure, and how you measured the fix, in a form interviewers can score.

The question, and what is being scored

The interviewer says: "Tell me about an AI system you shipped to production. What went wrong, and how did you fix it?" It comes in many costumes: "your biggest failure", "a time a model surprised you", "walk me through your résumé project". Every loop has one, and it is the question where the other fifteen chapters get tested against your real experience.

Interviewers score it against a rubric, usually informal, with four items:

Ownership. Did you decide things, or did you attend? The answer should make clear what you did and what the team did.
Diagnosis. Did you find the cause from evidence, and can you say what the evidence was?
Measurement. Did you know the fix worked? Before and after, over a stated period, with a stated method.
Judgement. What would you do differently, and what trade-off did you accept?

You cannot be coached into a good story you do not have. This chapter gives a structure that lets a true story be scored fairly. Everything you say must be a fact you can defend under follow-up, so this chapter uses placeholders like [the system] and [N%] and never a made-up history. Where it needs a setting, it borrows Harbourline's, and you replace that with yours.

The six beats

A strong answer runs four to five minutes and has six parts in a fixed order. The order matters because the interviewer is building a picture and interrupts when a part is missing.

1 Context and constraint who used it, what volume, what a wrong answer cost ~30 s 2 What you built and why the shape, and the alternative you rejected ~60 s 3 The failure what happened, and the signal that revealed it ~60 s 4 The fix the root cause, and the guard against a repeat ~45 s 5 The measurement metric, before and after, period, method ~45 s 6 What you would change one honest thing, with the reason ~20 s

1. Context and constraint

Say what the system did, for whom, at what scale, and what made it hard. "A claims-intake agent reading [N] first notices of loss a day, where a wrong coverage decision is a regulatory issue" tells the interviewer which trade-offs were real. Name your role in the same breath: "I was the [role] and owned [the retrieval layer / the eval harness / the rollout]."

2. What you built and why that shape

Give the design in two or three sentences and then the choice: the alternative you did not take and why. "We made it a workflow with one agentic step, not a free agent, because [the path was known and we needed an audit trail]." This is where chapters 04 and 05 show up in your own words. If you cannot name a rejected alternative, the interviewer will suspect you did not make the decision.

3. The failure, with its signal

Choose a failure that cost something and say what it cost: wrong answers to customers, a spike in spend, an escalation, a missed deadline. Then say how you found out. The signal is the most scored detail in the answer, because it shows what you were watching: "an alert on cost per session", "a rise in the human-escalation rate", "a complaint that I traced", "the nightly eval dropped [N] points on one slice". A failure found by luck is a weaker story than one found by a metric you had set up.

4. The fix

State the root cause in terms of the system, not the model. Then give two parts: the change that fixed this case, and the change that makes the class of case harder to repeat, such as a new eval slice, a harness guard or an alert. Compare chapter 01, where the fix is chosen by locating the fault first.

5. How you measured it

Give the metric, the value before and after, how long you watched, and how you measured it. "[Metric] went from [N%] to [N%] over [N weeks] on a [sample / A/B split], graded by [rule / judge calibrated against human labels]." Include a number that did not improve or a cost you paid. That is what makes the improvement credible. The grading method is in chapter 07 and for agents chapter 08.

6. What you would change

One thing, specific, with the reason: "I would have built the [slice] into the eval set before launch, because the failure was one we could have seen offline." Avoid a list of five, and avoid a flaw that is secretly a virtue ("I work too hard").

How the answer fails

The same mistakes recur, and each maps to a rubric item.

No numbers. "It improved a lot" scores as unmeasured. Give a value, a period and a method, even if the value is small or the method imperfect. If you truly have no number, say that, and say what you would measure.
Blaming the model. "The model hallucinated" names a symptom. Interviewers want to hear what in your system allowed it to reach the customer: no grounding check, no abstention path, no eval on that slice. Root causes in your own system are the ones you can fix, which is why the interviewer wants them.
"I improved the prompt." This is true of nearly every fix and tells the interviewer nothing. Say what was wrong in the prompt, what you changed, and what stops the next change from undoing it.
No signal. A failure that "was reported" with no account of how you saw it in data suggests you had no monitoring.
The hero story. If every verb is "I", the interviewer cannot tell what was yours. Use "we" for the team's work and "I" for your decisions, and say the difference out loud.
The fake failure. A failure that costs nothing ("it was slow once") is read as evasion. Pick one where something real was at stake, even if modest.
Architecture tour. Beats 1 and 2 are a quarter of the time. If you spend the first three minutes on boxes, the failure and the measurement, which are scored hardest, get squeezed.

Never state a figure you cannot source if asked

Interviewers probe the number: "How did you measure that?" A figure you cannot trace to a dashboard, an eval run or a log is worse than no figure. Say "roughly [N%], from [the source], and I would not rely on the second decimal."

Two templates to fill in

These are fill-in forms, not scripts to memorise. The setting is Harbourline's; replace the setting and every bracket with your own facts. If a bracket has no true value for you, choose a different story.

Template A: a retrieval system gave a wrong answer

Template A · about 4 minutes

"[The system] was a [help-centre assistant] over [policy documents], used by [N] customers a day, where a wrong [deductible figure] becomes a complaint. I owned [the retrieval layer and its evals]. We chose [hybrid search with a reranker] over [dense only] because [exact policy numbers were missing]. About [N weeks] after launch, [the escalation rate on one topic] rose from [N%] to [N%]. We saw it on [the dashboard I had set up], not from a complaint. I pulled [N] traces and the cause was [a superseded policy wording ranked above the current one], because [ingestion kept both versions with no effective-date filter]. The fix had two parts: [filter by effective date], and [a new eval slice of superseded-wording questions that runs on every change]. After the fix, [that topic's escalation rate] fell from [N%] to [N%] over [N weeks], measured on [the live sample], and [latency rose by N ms]. What I would change is [to have built the slice before launch]."

Template B: an agent did the right thing the wrong way

Template B · about 4 minutes

"[The system] was a [claims-intake agent] handling [N] claims a day, where [a coverage decision with no record] is a regulatory problem. I owned [the agent loop and its tool boundaries]. We made it [a workflow with one agentic step] because [the path was known and we needed an audit trail]. In [week N] a [daily check over live traces] showed that [N%] of routed claims had no [coverage record]. Outcomes were still correct, so [the answer-based eval never flagged it]. The cause was [the model inferring the queue from short, clear loss descriptions]. The fix was [to make routing a tool that refuses to run without a coverage decision ID] plus [a trace-level eval scored for evidence, not only for outcome]. Within [N weeks] the share fell from [N%] to [0%], measured on [all live traces]. I would change [the eval design from day one]."

Both templates hide the same moves: a signal you owned, a root cause in the system, a two-part fix, and a measured result with a cost.

Variants, and the take-home debrief

The same six beats cover most of the neighbouring questions. What changes is which beat gets the time.

"Tell me about a time you disagreed with a teammate or stakeholder." Keep beats 1, 2 and 5. Make the disagreement a technical choice (agent against workflow, or ship against hold) and say what evidence settled it. Beat 3 becomes "what each side was afraid of".
"Your biggest failure." Spend the most time on beat 6 and on what you changed in your own practice, not just in the system. Choose a failure where you were partly responsible and say which part.
"A time you shipped something that did not work." Stretch beat 5: the interviewer wants to know how you knew it did not work, and what you did about the users in the meantime.
"How did you decide to build rather than buy, or to change models?" Beat 2 becomes the whole answer. State the options, the criteria, the eval you ran and the number that decided it.

The take-home debrief is a short version of the same story told about a system you built in a few hours. There is no production failure, so beat 3 becomes "the failure modes I found or expect", and honesty about limits is the thing scored. Walk through what you built, what you verified and how, what you cut and why, which failures you know about and how you would measure them, and what you would do with another day. Say clearly what you did not test. A debrief that claims it works everywhere loses to one that says "I checked [these slices] on [N] items, and the rest is untested."

Interview probes

Drill mode — hide all answers

Q1: "What exactly was your part, and what was the team's?"

"The team built [the ingestion and the serving layer]. I owned [the evaluation harness and the retrieval changes]. I made the decision to [change the ranking] and I wrote [the eval slice] that found the problem. [A colleague] owned [the rollout]. I can tell you what I would have decided differently in their area, but I did not own it."

Q2: "How do you know the fix, and not something else, caused the improvement?"

"I looked for other changes in the same window. [We shipped nothing else touching that path.] I also compared [the affected slice] against [a slice the fix did not touch], and only the first moved. Better still, [I ran the old and new versions side by side on the same traffic for N days]. If I had only a before-and-after, I would say so and treat the result as likely, not proven."

Q3: "Why didn't you catch it before production?"

"Because [the eval set had no questions of that kind]. The set came from [sampled logs by topic], and [superseded wording] was a rare case that sampling did not produce. That was a gap in how I built the set, not bad luck, and the change I made was to add [synthetic rare cases] and a rule that every incident becomes an eval item."

Q4: "Was this the model's fault?"

"The model produced the bad output, but the system allowed it to reach the user. [A check on the cited source was missing / no abstention path existed.] I can only change the system, so that is where I looked. If I found that a different model avoided it, I would still want the check, because the next model will fail somewhere else."

Q5: "What would you do differently if you built it again?"

"One thing: [build the eval slice and the live trace check before launch, not after the incident]. It would have cost about [N days] and it would have found the problem offline. I would also [define the rollout gate in advance], but the first is the one that would have changed the outcome."

✓ Check yourself

Answer from memory before opening each one.

A candidate says, "Our assistant started giving wrong answers, so I rewrote the prompt, and it got much better." Which three things would an interviewer ask for next, and which rubric items do they map to?

Show answer
How the failure was detected (diagnosis), what was wrong in the system and what exactly changed (diagnosis and ownership), and the before and after figures with period and method (measurement). The answer as given has no signal, no cause and no number. See How the answer fails.
Why does the order of the six beats matter, and what happens to an answer that spends three of its five minutes on architecture?

Show answer
The interviewer builds a picture in sequence and interrupts when a piece is missing, so a fixed order keeps the answer under control. Spending three minutes on architecture squeezes the failure and the measurement, which are the beats scored hardest, and leaves under two minutes for them. See The six beats.
Your failure story is that your agent looped for 40 steps on a malformed tool response. Where in the six beats does "we added a hard step cap" go, and what else must the fix beat contain?

Show answer
In the fix, as the guard against the class of failure. The fix beat also needs the root cause (the tool error was handed back to the model, which retried the same call), and the case-level change. Then beat 5 must show the loop rate before and after. See The six beats and chapter 14.
You are asked to debrief a take-home you built in one evening. You have no production failure. What replaces beat 3, and what single quality is scored most?

Show answer
The failure modes you found or expect, with how you would measure each. The quality scored most is honesty about limits: what you tested, on how many items, and what you did not test. Claiming it works everywhere is the common error. See Variants and the debrief.
Your honest measurement is a before-and-after over four weeks with no side-by-side run, and a model upgrade shipped in week three. How do you present the result, and would you still use the story?

Show answer
State the figures, name the confound (the model upgrade), and compare the affected slice with an untouched one to separate the two. Say the result is likely but not proven. Yes, the story is still usable: interviewers reward a stated limit and penalise a hidden one. See How the answer fails.

Takeaway

The system-you-shipped question scores ownership, diagnosis, measurement and judgement, and the six beats are built so each can be seen.
The signal that revealed the failure is the most scored detail, because it shows what you were watching.
A root cause in your own system is a stronger answer than a root cause in the model, because it is the one you can fix.
A fix has two parts: the change for this case and the guard against the class of case, such as an eval slice or a harness rule.
Every claimed improvement needs a metric, a before and after, a period and a method, plus one cost or one number that did not move.
Never state a figure you cannot trace to a source, and say plainly what you did not test.

That completes the sixteen questions; the hub lists them all, with the study paths for the time you have left. → AI Engineer Interview Prep