Section D · Production

Data & Pipelines

The off-chain half of security work. Monitoring transactions in flight, learning from past exploits, and where the data feedback loop sits in your daily workflow.

Off-chain security tooling

You can't formally verify your way out of every incident. A serious security program treats the on-chain code as one layer; the off-chain monitoring, alerting, and dataset work as another. The bar a senior security engineer should clear:

  • Real-time alerts when something out-of-distribution happens on-chain.
  • A maintained dataset of past exploits you can pattern-match against.
  • Indexed access to on-chain data so you can investigate alerts in minutes, not hours.
  • A clear path from "alert fires" to "human is paged" to "war room opens."

Transaction monitoring stack

The product landscape for protocol monitoring is healthy. The major vendors:

ToolStrengthNotes
Tenderly AlertsBest dev UX, alerting on events / fn calls / state changesPairs with Tenderly's debugger
HypernativePredictive ML on suspicious patterns; pre-exploit alertingCommercial, used by larger protocols
CyversReal-time threat detection, on-chain forensicsStrong post-incident workflow
HexagateThreat detection, oracle deviation alertsAcquired by Chainalysis
FortaDecentralized detection-bot network; OSS detectorsYou can write and deploy your own
OpenZeppelin DefenderMonitor + Sentinel + admin actions in onePairs with OZ contracts

Most protocols run 2-3 in parallel. The ones at minimum:

  • One vendor with predictive / ML alerts (catch novel patterns).
  • One protocol-specific custom detector (you know your invariants; encode them).
  • One operational alerting layer (PagerDuty / Opsgenie) that pages humans.

Mempool monitoring

The mempool is the queue of pending transactions before block inclusion. Watching it lets you:

  • See exploits forming in pending txs before they confirm.
  • Whitehat-rescue: if you can submit a transaction faster, you might frontrun the exploit and rescue funds.
  • Detect MEV patterns (sandwich, JIT, atomic arbitrage).

Tools: BloXroute mempool, Flashbots Protect API, public Ethereum nodes with mempool exposed. Note that private mempool / order flow is increasingly common; you can't see everything from a public node.

Realism

"Whitehat rescue via mempool monitoring" is hard. The exploiter's bot is also watching. To beat them you need lower-latency infra and a pre-rehearsed playbook. Don't promise this to leadership as a capability you can deliver on cold.

Forta detection bots

Forta runs as a decentralized network of bots that scan every block. Each bot is a small program that emits alerts when its conditions fire. You can publish bots to the network, subscribe to others' bots, and route alerts to your protocol's pagerduty.

Patterns of useful bots:

  • Invariant violation bot — checks your protocol's stated invariants every block; alerts if violated.
  • Large-outflow bot — alerts if outflow exceeds 10% of TVL in a single block.
  • Privileged-address-active bot — alerts when admin / multi-sig sends a transaction (you should always know).
  • Oracle-deviation bot — alerts on price moves > threshold per block.
  • Flash-loan use bot — alerts when known flash-loan providers are involved in a tx that interacts with your protocol.
// Forta bot pseudocode (TypeScript SDK)
const handleTransaction: HandleTransaction = async (txEvent) => {
  const findings: Finding[] = [];
  if (txEvent.to !== MARKET_ADDR) return findings;

  // Read total borrow / supply after this tx via web3
  const totalBorrow = await market.totalBorrowAssets();
  const totalSupply = await market.totalSupplyAssets();
  if (totalBorrow.gt(totalSupply)) {
    findings.push(Finding.fromObject({
      name: "Bad debt invariant violated",
      severity: FindingSeverity.Critical,
      type: FindingType.Exploit,
    }));
  }
  return findings;
};

Postmortem datasets

The cheapest way to learn the bug taxonomy: read postmortems. Maintain a curated list. The major sources:

  • Rekt News — narrative-style breakdowns of major exploits with USD impact tracker.
  • DeFiLlama hacks page — structured dataset of exploits with date, amount, protocol, attack type.
  • SlowMist hacked database — well-tagged, searchable by attack vector.
  • Audit firm blogs — Trail of Bits, OpenZeppelin, Spearbit, Zellic all publish detailed postmortems.
  • Code4rena / Sherlock contest reports — public dataset of vulnerabilities found in audits.
  • Immunefi disclosures — public bounty reports.
  • Security Alliance (SEAL) — incident response collective; their public writeups are gold.
A specific habit

Pick one major incident per week. Read every postmortem you can find. Write a 500-word summary in your own notes, including: root cause class, the missing invariant, how it would have been caught. Six months of this and you have a serious bug catalogue in your head.

Maintain your own dataset of past exploits

A spreadsheet or sqlite file with columns:

  • Date
  • Protocol
  • Chain
  • USD loss
  • Root cause class (from your taxonomy)
  • Specific bug
  • Public postmortem URL
  • Was an audit done pre-deploy? Which firm?
  • Was the bug findable with [Slither / fuzzing / FV]?
  • Notes

This becomes invaluable when:

  • You're asked "what's the most common root cause in lending protocols?" — you can answer with data.
  • You're reviewing a new feature — you can pattern-match to past similar bugs.
  • You're justifying audit scope — "the audit firm should focus on X because three of the last five lending exploits were that class."

Where the security feedback loop sits

The mature feedback loop for a DeFi protocol's security:

  1. Monitoring sees a real-world signal (alert, exploit attempt, novel pattern).
  2. Triage determines if it's a false positive, a near-miss, or an active incident.
  3. Investigation finds the underlying cause.
  4. Mitigation goes into the codebase or operational SOP — new CVL invariant, new Slither detector, new Forta bot, new pause condition.
  5. Documentation updates the internal bug taxonomy and the threat model.
  6. Drill tests the new SOP at least once before the next incident.

The loop is owned by you. If alerts fire and no one updates the spec or the runbook, the alert is just noise. The discipline is connecting alert → investigation → permanent improvement.