Common Deployment Failures
The dozen patterns that kill FDE-DE deployments — what they look like, what causes them, how to prevent and recover. Treat as the diagnostic catalog; when something feels off, scan this first.
How to read this catalog
Each entry has the same shape: name, symptoms, root cause, prevention, recovery. The most senior FDEs have seen most of these at least once and most of the rest from a colleague's deployment. Reading the catalog is cheaper than living through any single one of them.
Political failures
1. The disappearing sponsor
- Symptoms: Executive sponsor misses two monthly reviews. Stops responding to direct emails. Delegates a deputy who can't make decisions.
- Root cause: Sponsor's priorities shifted; internal reorg; the project lost board-level visibility; sponsor is exiting the company.
- Prevention: Cultivate at least one secondary sponsor early — usually one tier below the primary. Don't bet everything on a single relationship.
- Recovery: Escalate via your AE; if a real change happened, you may need to re-anchor on the secondary. If sponsor is exiting, find the inheritor of their portfolio fast.
2. The political stalemate
- Symptoms: Internal customer stakeholders disagree about what success means. Conversations loop. Working agreement keeps getting redrafted with no signoff.
- Root cause: Two senior stakeholders (procurement vs finance, procurement vs legal) want fundamentally different outcomes from the platform.
- Prevention: In discovery, surface the disagreement explicitly. Refuse to start work until it's resolved.
- Recovery: Escalate to your AE. They own the conversation about whose mandate prevails. Don't ship into stalemate.
3. The reorganization
- Symptoms: Customer announces a restructuring. Your sponsor's team moves under different leadership. New leader doesn't see the deployment's value.
- Root cause: Standard enterprise dynamics. Happens to most multi-quarter deployments.
- Prevention: Watch for leading indicators in the customer's industry news. Have artifacts ready to show new leadership: business outcomes, end-user quotes, ROI.
- Recovery: Schedule an introduction with the new leader within 30 days. Ship a one-page outcomes summary. Re-validate scope under the new mandate.
Scope failures
4. The scope creep that wasn't tracked
- Symptoms: By week six, you're delivering work you didn't agree to. Original timeline slips. Customer is happy but you're burning out.
- Root cause: Adjacent customer asks were absorbed without renegotiating timeline or scope. The "out of scope" section of the working agreement was vague.
- Prevention: Be ruthless about the out-of-scope list. When a new ask comes up, write it down explicitly: "this is in scope" or "this is a new scope conversation."
- Recovery: Re-anchor the working agreement. Name what's been added; renegotiate timeline. Honest is uncomfortable but recoverable.
5. The shrinking scope
- Symptoms: Customer keeps removing classes from scope. By week 8, you're delivering 30% of what was originally signed for.
- Root cause: Customer's situation changed; or the original scope was unrealistic; or extraction quality is below what they want and they're cutting losses.
- Prevention: Pre-flight data quality check in discovery. Set realistic SLAs against the real corpus.
- Recovery: Surface to your AE — shrinking scope often correlates with churn risk. Renegotiate the working agreement and the commercial terms together.
Data-reality failures
6. The customer's data is worse than they said
- Symptoms: OCR quality on a meaningful portion of the corpus is below threshold. Documents are scanned faxes. Amendments aren't linked to MSAs. Customer's "supplier master" has 4× the entries they claimed.
- Root cause: Customer didn't know. Or knew and didn't say. Either way, you have it now.
- Prevention: Pre-flight quality sample in discovery. Set the SLA expectations against what you saw, not what they described.
- Recovery: Bring the evidence to the customer. Reshape scope — narrow to the data that works. Negotiate carve-outs on the SLA.
7. The hidden document class
- Symptoms: Week six: customer mentions in passing that "we also have these 50,000 amendments separately." Wasn't in the original corpus walkthrough.
- Root cause: Discovery didn't go deep enough on the document inventory. Or different teams own different document classes and only one team was in the room.
- Prevention: Ask "what other contracts exist that you didn't think to mention?" in discovery, multiple times, to multiple stakeholders.
- Recovery: Treat it as a new scope conversation. Don't just absorb it. Renegotiate timeline.
Integration failures
8. The identity-IT bottleneck
- Symptoms: Week six and you still don't have SSO set up. SCIM provisioning ticket has been "in review" for a month. Service-account credentials require a security committee that meets monthly.
- Root cause: Identity tickets were filed late; or filed but not escalated when they sat. Customer's IT team has more work than capacity.
- Prevention: File identity tickets in week one. Escalate when they sit. Maintain a "what's blocked by IT" running list in your Friday status.
- Recovery: Escalate via your AE. Sometimes a sponsor needs to call IT directly. Identify a service-account workaround that gets you partial value while waiting.
9. The ERP that won't be touched
- Symptoms: Customer wanted ERP writeback. Two months in, the SAP team has not approved any integration. CAB reviews keep being deferred.
- Root cause: ERP writeback was scoped without IT/Finance team's actual buy-in. The political weight was underestimated.
- Prevention: Treat ERP writeback as a Phase 2 conversation. Defer until you've earned trust with warehouse-only delivery first.
- Recovery: Pivot scope to warehouse + BI only; reframe ERP integration as a Phase 2 expansion. Don't argue with the SAP team; absorb the constraint.
Operational failures
10. The quality-drift you didn't see
- Symptoms: Customer notices extracted renewal dates are wrong before you do. SLA dashboard hadn't fired because measurement sample was stale.
- Root cause: Eval cadence too slow; sample size too small; the dashboard you built wasn't actually being watched by anyone.
- Prevention: Daily eval against a rolling sample. Alert on any class trending toward SLA threshold. Confirm with the customer that they're checking the dashboard.
- Recovery: Apologize specifically — this is a trust hit. Diagnose, fix, ship a remediation plan, document the gap that let it slip. Tighten monitoring.
11. The status-doc decay
- Symptoms: Three weeks of identical "in flight" items. Customer stops reading the Friday status. Your manager doesn't know what's happening.
- Root cause: The deployment has gotten stuck and the status doc reflects it. The doc-writing habit decayed when the work got hard.
- Prevention: Never skip the Friday status doc. The week it gets hard to write honestly is the week you most need to write it.
- Recovery: Write the status doc honestly this Friday. Name what's stuck. Use the kill-decision worksheet if appropriate.
Meta-failures
12. The FDE who can't leave
- Symptoms: Day 90 came and went. You're still primary on-call. Runbooks were written but not tested. The customer's data team hasn't independently resolved an incident.
- Root cause: Handoff didn't start until the last week. Training didn't include real incidents. The customer wasn't pushed to take ownership.
- Prevention: Start handoff in week 6, not week 12. Write runbooks during deployment, not after. Bring the customer into incident response actively, not as observers.
- Recovery: Explicitly schedule a 30-day handoff sprint. Schedule pair sessions on real tasks. If a real incident happens, step back even if it feels uncomfortable. Let the customer drive.
You'll see most of these once across a few years of FDE work. Senior FDEs have a story for each. Treat the catalog as inoculation: knowing the pattern doesn't make you immune, but it shortens the time-to-recognition when you're in it. The two-week-faster recognition is often the difference between a recoverable deployment and a lost one.
✓ Check yourself
Answer from memory before opening each one.
-
Your sponsor's team has moved under a new leader who does not see the deployment's value. Which failure is this, and what do you do in the first 30 days?
Show answer
A reorganization. Schedule an introduction with the new leader within 30 days, ship a one-page outcomes summary with user quotes and ROI, and re-validate the scope under the new mandate. Prevention is to have those artifacts ready before the announcement. See political failures.
-
In week six the customer mentions another 50,000 amendments kept elsewhere. What do you do, and which discovery question would have caught them?
Show answer
Treat it as a new scope conversation and renegotiate timeline rather than absorbing it. The discovery question is "what other contracts exist that you did not think to mention?", asked repeatedly and to several stakeholders, because different teams own different classes. See data-reality failures.
-
The customer removes a document class from scope every few weeks and you are down to 30% of the original. Why is this more than a scope issue?
Show answer
Shrinking scope often tracks churn risk, since the customer may be cutting losses on quality or priority. Tell the AE and renegotiate the working agreement and the commercial terms together. See scope failures.
-
The customer finds wrong renewal dates before you do, and the SLA dashboard never fired. Give three likely reasons and the repair.
Show answer
The evaluation runs too rarely, the sample is too small or stale, or nobody is watching the dashboard. Move to a daily rolling sample, alert on any class trending toward its threshold, confirm the customer checks it, and apologize specifically. See operational failures.
-
In week nine the Friday status shows the same three in-flight items as three weeks ago, and the service-account approval has sat with customer IT for a month. Which failures are in play and what links them?
Show answer
Status-doc decay and the identity-IT bottleneck. The blocked approval is the stuck item. Write the status honestly this Friday, name what IT is holding, escalate through the AE and keep a running list of what is blocked. See integration failures.
-
It is day 90, you are still primary on-call and your team wants you on the next deployment. Do you start a handoff sprint?
Show answer
Yes. Schedule a 30-day sprint of pair sessions on real tasks and step back at the next incident even if it feels uncomfortable. The cost to the next deployment is real, so tell leadership the date. Staying primary only delays the same work. See meta-failures.
Takeaway
- Most deployments are lost to political, scope and data-reality problems rather than technology.
- Cheap prevention happens in discovery: a second sponsor, disagreements surfaced, a preflight data sample and a repeated document-inventory question.
- Absorbing new work without renegotiating the agreement is how scope creep and hidden document classes become timeline failures.
- Shrinking scope often precedes churn, so bring it to the AE early.
- File identity tickets in week one and treat ERP writeback as a later phase.
- If the customer finds a quality drop first, you lose trust, so evaluate daily on a rolling sample and alert on trends.
- An FDE who cannot leave started handoff too late; begin in week six and let the customer drive incidents.
The technical craft behind these deployments, from extraction to warehouse integration and SLA-backed evaluation, is in the sibling guide. → FDE Data Engineering guide
Further reading
- Postmortem Culture: Learning from Failure — Google SRE book, 2016. A model for writing up a failure without blame, and for turning it into changes.
- Emergency Response — Google SRE book, 2016. Real cases of operational failures and how the responders worked through them.
- Monitoring Distributed Systems — Google SRE book, 2016. What to monitor and alert on, relevant to catching quality drift before the customer does.
- Source freshness — dbt Labs documentation, accessed 2026. How to detect stale source data, one way to catch a silent pipeline problem.