Async: Webhooks & Streaming

When request/response fails

Request/response assumes the caller waits and the answer is ready in a few hundred milliseconds. Two situations break that assumption.

• The work outlasts the connection. Importing 50,000 links takes minutes. Load balancers, proxies and clients time out long before that, and a held connection ties up a worker for the whole wait.
• The event starts on the server. A link crosses a click threshold, an import finishes, a domain fails verification. No caller is waiting, so there is no response to carry the news.

Each has a standard shape. Slow work gets an operation resource the client polls. Server-originated events get a webhook (push to a URL the client owns) or a stream (push down a connection the client opened). This chapter covers all three, then how to design the events so they survive change.

Snip is the running example. Recall the split from the scale guide: the redirect GET snip.ly/{code} is the data plane; https://api.snip.ly/v1 is the control plane. Everything here sits on the control plane, fed by the click pipeline described in Async Work & Queues.

Long-running operations

When a request cannot finish in the time a client will wait, accept it, say so, and give the client something to ask about. Return 202 Accepted with a Location header pointing at an operation resource. This is the same move as in Resource Modeling: when an action takes time, model the action as a noun.

POST /v1/imports HTTP/1.1
Authorization: Bearer eyJ…
Idempotency-Key: 7c1e9a52-3b0d-4c7e-9a41-0f5d2b6e8a10
Content-Type: application/json

{"source_url": "https://files.example.com/links.csv"}

HTTP/1.1 202 Accepted
Location: /v1/operations/op_8Hk2
Retry-After: 5

{"id": "op_8Hk2", "status": "pending", "created_at": "2026-10-12T09:14:03Z"}

The client polls the operation:

GET /v1/operations/op_8Hk2

HTTP/1.1 200 OK
{
  "id": "op_8Hk2",
  "status": "succeeded",
  "progress": {"done": 50000, "total": 50000},
  "result": "/v1/imports/imp_41x",
  "error": null
}

status moves one way: pending to running to succeeded or failed. On failure, error holds an RFC 9457 problem object, the same shape as every other error in the API. On success, result links to the created resource rather than embedding it, so there is one place to read the truth.

Four details decide whether this works in practice:

• Return Retry-After. It tells clients how often to poll, and lets you slow them down under load.
• Keep the operation after it finishes. A client that was offline for an hour must still find the outcome. Expire operations after days, not seconds.
• Make the POST idempotent. A retry of a slow request must not start a second import. See Caching, Concurrency & Idempotency.
• Offer a callback as well. Polling is the floor. A client that registers a webhook for operation.completed stops polling altogether, which is the next section.

Do not block instead. A synchronous import that "usually" finishes in 20 seconds will one day take 200, and you will find out from the support queue. Exception: if the work has a hard, small upper bound (a few hundred milliseconds at the 99th percentile), stay synchronous. An operation resource is extra surface the client must learn.

The webhook model

A webhook is an HTTP POST from your server to a URL the consumer registered. The consumer does not poll; you call them when something happens. The cost is that you now run a delivery system, and the consumer runs a public endpoint.

Registration is a resource like any other:

POST /v1/webhook-endpoints
{
  "url": "https://hooks.acme.example/snip",
  "events": ["link.clicked", "operation.completed"]
}

HTTP/1.1 201 Created
{
  "id": "whe_3Qa9",
  "url": "https://hooks.acme.example/snip",
  "events": ["link.clicked", "operation.completed"],
  "secret": "whsec_k3J…",
  "status": "enabled"
}

Three rules for the registration:

• The consumer names the event types. Never send everything by default; a consumer who wants import results should not receive click summaries.
• HTTPS only. Reject http:// URLs at registration. Also reject URLs that resolve to private or link-local addresses, or your delivery workers become a way to probe your own network from inside it.
• Show the secret once. It signs every delivery (see Signing and verifying). Offer rotation, with both old and new secrets valid for a short overlap.

Disable endpoints that fail for days, and say so in the API ("status": "disabled") and by email. A dead endpoint otherwise burns retries forever.

Designing the event

Every event has the same envelope, whatever it carries:

{
  "id": "evt_01JA4Z8M2K",
  "type": "link.clicked",
  "created": "2026-10-12T09:15:00Z",
  "data": {
    "link": "https://api.snip.ly/v1/links/q7Xk2",
    "code": "q7Xk2",
    "window": {"start": "2026-10-12T09:14:00Z", "end": "2026-10-12T09:15:00Z"},
    "clicks": 18342,
    "top_countries": [{"country": "US", "clicks": 9120}, {"country": "BR", "clicks": 2210}]
  }
}

id is unique and stable across retries; it is what consumers de-duplicate on. type is a dotted name, resource.past-tense-verb. created is an RFC 3339 timestamp for when the event happened, not when it was delivered. data holds the payload.

The design question is how much goes in data.

• Thin events carry only identifiers: "link q7Xk2 changed." The consumer calls GET /v1/links/q7Xk2 to learn how. Always current, but every event costs a round-trip, and a burst of events becomes a burst of API calls.
• Fat events carry the new state. The consumer can act at once. But the state is a snapshot, and with no ordering guarantee (next section) a snapshot can arrive after a newer one. A consumer who writes it straight to their database overwrites newer data with older.

My default: carry enough to act on without a call, and always include the resource URL (data.link above) so the consumer can fetch current truth. Document the snapshot as "state as of created". A consumer who cares about correctness compares timestamps or re-fetches; one who only wants to notify a human reads the payload and moves on. For fields that change often, such as total_clicks, which is eventually consistent anyway, lean thin.

Delivery: retries, replay and order

Delivery over HTTP can fail in a way the sender cannot tell apart: the consumer may have processed the event and crashed before replying, or never seen it. A sender can then pick one of two promises. At-most-once drops events silently. At-least-once retries until it gets a 2xx, and may deliver the same event twice. Choose at-least-once; losing events is worse than seeing them twice. Exactly-once delivery is not available over a network, and a vendor who claims it means at-least-once plus de-duplication on their side.

Snip's rules:

• Any 2xx within 10 seconds is success. Anything else, including a timeout, is a failure.
• Retry with exponential backoff and jitter: roughly 1 minute, 5 minutes, 30 minutes, 2 hours, 8 hours, then every 24 hours.
• After 8 failed attempts the delivery goes to a dead-letter state and is no longer retried.
• Consumers can list deliveries and replay any of them:

GET /v1/webhook-endpoints/whe_3Qa9/deliveries?status=failed
POST /v1/webhook-endpoints/whe_3Qa9/deliveries/dlv_77/replay

The delivery log is not a nicety. A consumer that was down for a weekend needs a way to catch up that is not "ask support". Replay re-sends the same event with the same id.

Two obligations follow for the consumer.

De-duplicate on id. Because delivery is at-least-once, the same evt_… will sometimes arrive twice. The consumer records processed ids and skips repeats, exactly the pattern in Caching, Concurrency & Idempotency, with the event id as the idempotency key. Make the record and the side effect commit together, or a crash between them re-opens the gap.

Do not assume order. A retry of event 1 can land after event 2 was delivered first time. Snip does not promise order and consumers must not rely on it. Use created to detect staleness, or treat the event as a hint and re-fetch. Respond 2xx fast and process the work on your own queue; a consumer that does slow work inline will time out and trigger the retries it dreads.

Signing and verifying

The consumer's endpoint is on the open internet. Anyone can POST to it. The consumer needs proof that Snip sent a request and that nobody altered it.

This is a different problem from request auth in Auth & Scopes. There, the client proves who it is to Snip with a bearer token. Here, the roles reverse and Snip proves itself to the client. Tokens are the wrong tool, because a bearer token is replayable by anyone who sees it. Use a message signature instead: an HMAC (RFC 2104) over the request body with a secret only the two parties know.

Snip signs the raw bytes of the body together with a timestamp:

POST /snip HTTP/1.1
Content-Type: application/json
Snip-Signature: t=1760260500,v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd

{"id":"evt_01JA4Z8M2K","type":"link.clicked", …}

The consumer verifies in five steps:

header  = request.headers["Snip-Signature"]
t, v1   = parse(header)                      # t=timestamp, v1=hex signature
if abs(now() - t) > 300 seconds:  reject    # stops replay of a captured request
signed  = t + "." + raw_body                 # raw bytes, before any JSON parsing
expect  = hex(HMAC_SHA256(secret, signed))
if not constant_time_equal(expect, v1): reject
accept; then de-duplicate on event id

Each step closes a specific hole:

• Raw body. Parse the JSON first and re-serialize it, and key order or whitespace changes; the signature no longer matches. Verify before parsing.
• Timestamp inside the signed string. Without it, an attacker who captures one valid delivery can replay it forever. With it, a replay older than five minutes fails. The attacker cannot edit t without breaking the signature.
• Constant-time compare. An ordinary == exits at the first differing byte, and the timing leaks the signature one byte at a time.
• Replays inside the window. Five minutes is still a window. The event-id de-duplication closes it.

The v1= prefix is deliberate. It lets you add a new scheme later, sending both signatures during the change. The Standard Webhooks project specifies a similar design if you would rather not invent one.

Why link.clicked is batched

The obvious design is one link.clicked delivery per redirect. For Snip that is a mistake. The data plane serves 100 million redirects a day, about 1,160 a second on average and several times that at peak. One webhook per click would hand each consumer a firehose their endpoint cannot absorb, and make Snip run a second system as large as the redirect service just to send the news. Retries on top multiply it.

So link.clicked summarizes. An event says "link q7Xk2 had 18,342 clicks between 09:14 and 09:15", as in the payload above, or flags a notable event: a spike over a threshold, the first click from a new country. The click pipeline in Async Work & Queues already aggregates clicks off a queue for analytics. The webhook sender reads those aggregates; it never sits in the redirect path, and a slow consumer cannot slow a redirect.

Document the sampling and batching in the event's reference page: the window size, the threshold, and the fact that counts are eventually consistent, like total_clicks on the link. Consumers who need every click should pull an export or the analytics API, not wait for a webhook that was never designed to carry them. Say what an event is for. A webhook is a notification channel, not a replication log.

Streaming: SSE, WebSockets and long-polling

Webhooks push to a server. A dashboard open in a browser has no URL to push to; it wants to read live from you, and the client starts the conversation. Three tools fit.

Default to SSE. It is plain HTTP, so it passes proxies, reuses your auth and logging, and needs no new protocol. The browser's EventSource reconnects on its own and sends the last event id it saw in a Last-Event-ID header, so the server can resume where the stream broke. WebSockets cost more: a separate protocol, its own auth story (browsers cannot set headers on the handshake), sticky connections, and your own reconnect and resume logic. Take that cost only when the client really sends a stream back. Long-polling is the fallback, and it is what a poll against an operation resource already is.

A live click stream for the Snip dashboard:

GET /v1/links/q7Xk2/clicks/stream HTTP/1.1
Accept: text/event-stream
Authorization: Bearer eyJ…
Last-Event-ID: 1042

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache

id: 1043
event: clicks
data: {"window_end":"2026-10-12T09:15:05Z","clicks":312}

id: 1044
event: clicks
data: {"window_end":"2026-10-12T09:15:10Z","clicks":297}

: keep-alive

Each message is a block of field: value lines ended by a blank line. id is what the client echoes back on reconnect; the line starting with : is a comment, sent every 15 seconds or so to stop idle proxies from closing the connection. Note that the stream sends five-second aggregates, for the same reason as the webhook: the dashboard needs a number it can draw, not 1,160 messages a second.

Three things to plan for. The server must keep a short buffer of recent events, or Last-Event-ID has nothing to resume from. Each open stream holds a connection, so cap streams per token. And streams are not a place to hide the data plane: serve them from the control plane's own tier, not the redirect fleet.

Events are a contract

A webhook payload is an API response you cannot see the reader of. Consumers wrote parsers against it, deployed them, and will not touch them for a year. Everything in Versioning & Evolution applies, with the extra difficulty that you cannot ask the consumer to send a version header with each delivery.

• Pin the version on the endpoint. Record the API version when the consumer registers ("api_version": "2026-10-01"), and render events in that shape until they upgrade. Include the version in the payload as well.
• Add fields, never change or remove them. New optional fields are free. Renaming clicks to click_count breaks everyone.
• Tell consumers to ignore what they do not know. Unknown fields and unknown event types must be skipped, not rejected. State this in the docs and in every sample. Otherwise adding link.expired next year breaks the consumers who subscribed to everything.
• Treat enum values as additive too. A new status value is a breaking change for a consumer with a strict switch; warn them in the docs.

If you would rather adopt a shared envelope than design your own, CloudEvents defines id, source, type and time fields. It earns its place when events cross many systems and tools; for a single product API, the plain envelope above does the same job with less to explain.

✓ Check yourself

Answer before you open the folds. The answers are short and each links back to its section.

1. A client retries POST /v1/imports after a network timeout. What stops Snip from starting a second import, and what does the client get back?

   Show answer

   The Idempotency-Key header. The server stored the first request under that key, so the retry returns the original 202 and the same operation id. See Long-running operations.

2. Your fat link.updated event arrives at a consumer twice, and an older one arrives after a newer one. Name the two defences the consumer needs, and what each relies on.

   Show answer

   De-duplicate on the event id (at-least-once delivery), and compare created or re-fetch the resource URL (no ordering guarantee). See Delivery.

3. Why must the consumer verify the signature against the raw body, and what goes wrong if it parses the JSON first?

   Show answer

   The HMAC covers exact bytes. Parsing and re-serializing can change whitespace or key order, so a genuine delivery fails verification. See Signing and verifying.

4. An attacker records a valid delivery and re-sends it ten minutes later. Which two checks stop it, and which one covers a re-send inside the first minute?

   Show answer

   The timestamp window rejects the ten-minute-old copy, because the timestamp is part of the signed string. Event-id de-duplication catches a re-send inside the window. See Signing and verifying.

5. A customer asks for a webhook on every click so they can count them. What do you say, and what do you offer instead?

   Show answer

   Not as one delivery per redirect: 100M a day would swamp them and Snip. Offer the batched link.clicked summary, or an export or analytics query for exact counts. See Why link.clicked is batched.

6. The dashboard needs live click counts and never sends data back. Which transport do you choose, and how does it recover from a dropped connection?

   Show answer

   SSE. EventSource reconnects and sends Last-Event-ID; the server replays from its short buffer. See Streaming.

Takeaway

• When work outlasts a connection, return 202, a Location, and an operation resource that ends in a link to the result.
• Webhooks are at-least-once and unordered. The consumer de-duplicates on event id, as in idempotency, and never trusts arrival order.
• Sign the raw body and a timestamp with HMAC-SHA256, verify in constant time, reject old timestamps. This is separate from request auth.
• Do not mirror your data plane into events. Batch and sample, and say what the event is for.
• For a client that only listens, use SSE. Pay for WebSockets only when the client streams back.
• Events are a contract: version them, add fields only, and tell consumers to ignore what they do not know.

Next, every idea on one page — the full worked design. Design-First: A Worked API →

Further reading

• HTML Standard: Server-sent events — the text/event-stream format, EventSource and Last-Event-ID, from the source.
• RFC 6455: The WebSocket Protocol — the handshake and framing, to judge what you take on.
• RFC 2104: HMAC — the construction behind webhook signatures.
• Standard Webhooks — a shared specification for signing and delivering webhooks.
• CloudEvents specification — a common event envelope and its attribute rules.