Asynchronous Work & Queues

The move: a queue between two speeds

A redirect must finish in tens of milliseconds, and it also has a second job: record that the click happened. Doing that write inline ties the redirect's speed and availability to the analytics store. A queue breaks the tie. The producer (the redirect handler) appends a message to a durable log and returns. A consumer, running in a separate process at its own pace, reads the message later and does the work. The producer never waits for the consumer and never learns whether it succeeded.

The queue works as a shock absorber because it decouples arrival rate from service rate. Suppose a viral link pushes Snip to its 12,000 clicks per second peak, ten times the 1,200 average, for 5 minutes. A database that can apply 3,000 writes per second cannot take that directly; Little's Law says its in-flight requests balloon until connections run out. Behind a queue, the 9,000 per second of excess accumulates as backlog: 9,000 × 300 s = 2.7 million messages, about 270 MB at 100 B each. When the spike ends, consumers keep draining at 3,000 per second against 1,200 arriving, a net 1,800 per second, and the queue empties in about 25 minutes. The database saw a flat 3,000 per second the whole time. You traded latency on the background work for protection of the thing that cannot scale quickly.

That trade defines what belongs off the request path. Move work when the caller does not need its result to build the response, and when it is slow, bursty, or depends on a flaky third party: sending email, delivering webhooks, generating thumbnails, writing analytics, updating a search index. Keep work on the path when the response depends on it. Checking that a short code exists, verifying a password, or charging a card before showing "paid" must be synchronous, because the user is waiting for that answer. The test is simple: if you can return the response without knowing the outcome, queue it. If you would have to lie in the response, do not.

Backpressure and the infinite queue

A queue absorbs a burst only if consumers are, on average, faster than producers. When arrivals exceed capacity for long enough, the backlog grows without limit, and a growing backlog is growing latency. Little's Law applies to the queue as to a server: with L messages waiting and a drain rate of λ per second, a new message waits L / λ. A backlog of 2.7 million at 3,000 per second means the newest click is recorded 15 minutes after it happened. The queue did not fail; it is working, and the dashboards are 15 minutes stale.

Bound the queue in size or age so overload shows at a defined point. At the bound you have three options, and you pick one on purpose.

• Slow the producer (backpressure). The queue refuses or delays the write, and the producer waits or returns an error such as HTTP 429 or 503.
• Shed load. Drop the lowest-value messages, newest or oldest, and count what you dropped. Right when freshness beats completeness.
• Add consumers. Scale out the readers. This helps only until the shared downstream (the database) is the limit; ten consumers feeding one saturated database make the backlog worse, not better.

Alert on the age of the oldest message, not its depth: age matches the promise you made ("clicks appear within 5 minutes"). An unbounded queue never refuses a write, which is the trap: the system keeps accepting work it cannot finish, and you learn from latency, not from an error.

Delivery semantics

A consumer receives a message, does the work, and tells the broker it is done. The order of those steps, and the fact that the network can fail between any two of them, decides what guarantee you get.

The broker keeps each message until the consumer acknowledges (acks) it. If the consumer crashes or the ack is lost, the broker never learns the work finished and redelivers the message. In SQS this is the visibility timeout: a received message is hidden from other consumers for a set time (30 seconds by default), and if no delete arrives in that window it reappears. In Kafka the consumer commits an offset, a bookmark in the log, and after a crash it resumes from the last committed one. A negative ack (nack) tells the broker to redeliver immediately, because this attempt failed.

Where you ack decides the guarantee:

• At-most-once. Ack first, then process. A crash after the ack loses the message. Nothing is ever processed twice, and sometimes nothing is processed at all. Fine for a metric sample.
• At-least-once. Process first, then ack. A crash before the ack means redelivery, so every message is processed one or more times. Assume this from any queue.
• Exactly-once. Each message has exactly one effect. This is what everyone wants, and as a property of delivery over an unreliable network it is not achievable. When the ack is lost, the broker cannot tell "the consumer died before working" from "it worked and the ack vanished". It must pick redelivery or loss.

Idempotency in practice

An operation is idempotent when running it twice equals running it once. Setting a field to a value is idempotent; incrementing a counter is not. Three techniques turn a non-idempotent effect into a safe one.

Idempotency keys. The producer attaches a unique id to each message when it creates the event, not when it sends it, so a retried send carries the same id. The consumer records ids it has handled and skips repeats.

Dedup records. Store the ids in a table or set, and check-and-apply in one atomic step. The dedup check and the effect must commit together, or a crash between them recreates the problem. You also need not remember ids forever: only as long as a duplicate can arrive, which is bounded by the broker's retention and your retry policy. Keep 48 hours, then delete.

Here is Snip's click consumer. It reads a batch of events, inserts their ids into a dedup table, and applies counter increments only for ids that were new, all in one transaction. A redelivered batch inserts no new ids, so it adds nothing.

BEGIN;

-- 1. Claim the event ids. Only first-time ids come back.
WITH fresh AS (
  INSERT INTO processed_clicks (event_id, seen_at)
  SELECT event_id, now() FROM unnest($1::uuid[]) AS t(event_id)
  ON CONFLICT (event_id) DO NOTHING
  RETURNING event_id
)
-- 2. Count fresh events per code, then add to the counters.
INSERT INTO link_counts (code, clicks)
SELECT e.code, count(*)
FROM unnest($1::uuid[], $2::text[]) AS e(event_id, code)
JOIN fresh USING (event_id)
GROUP BY e.code
ON CONFLICT (code) DO UPDATE SET clicks = link_counts.clicks + EXCLUDED.clicks;

COMMIT;  -- then ack the batch / commit the Kafka offset

Order matters in the last line: commit the transaction, then ack. If the process dies between them, the broker redelivers, step 1 finds every id already present, and the batch is a no-op. Prune the dedup table after 48 hours; at 100M clicks per day it holds a few gigabytes of ids. A cheaper variant for Kafka stores the last processed offset per partition in the same transaction and ignores anything at or below it.

Poison messages, retries and ordering

Some messages fail every time: malformed JSON, a code that references a deleted record, a bug in your handler that one input triggers. With at-least-once delivery, a failing message is redelivered, fails, and is redelivered again. If the consumer retries immediately and in order, one such poison message blocks everything behind it, or burns the whole consumer fleet's capacity on a loop that cannot succeed.

Two mechanisms contain it. First, limit attempts: SQS calls the limit maxReceiveCount, and after that many receives the broker moves the message to a dead-letter queue (DLQ), a separate queue you inspect by hand. Kafka has no built-in DLQ; the consumer writes the failing record to a "dead letter" topic itself. Alert when the DLQ is non-empty: each message is work that never happened, to replay after the fix.

Second, space the retries. When a failure is transient (a downstream timeout), retrying instantly makes it worse. Use exponential backoff: wait 1 s, 2 s, 4 s, 8 s, up to a cap. Add jitter, a random fraction of the delay, because without it every consumer that failed at the same moment retries at the same moment, and the recovering dependency sees a synchronized spike, the same stampede the caching chapter described. "Full jitter" picks a random wait between 0 and the backoff ceiling. Retry only errors that can heal; a validation failure goes straight to the DLQ.

Ordering is the last thing the queue does not promise. A plain queue such as SQS Standard gives best-effort order at most, and redelivery makes it worse: message 1 fails and returns after message 2 has been handled. Kafka orders messages within a partition only; across partitions, none. To keep all events for one entity in order, send them to one partition by hashing the entity id, the same idea as the shard keys in the database chapter. The cost is that a hot key means a hot partition. So ask whether you need order at all. Counters commute (adding 1 then 2 equals adding 2 then 1), so click counting needs none. A "link edited, then deleted" sequence does.

The dual-write problem and the outbox

Now the producer side. When a user creates a link, Snip must insert the row and also publish a link.created event so an abuse scanner can check the target URL. That is two systems, the database and the broker, and no transaction spans both. Any order fails:

• Write the row, then publish. A crash between them leaves a link that exists and was never scanned. The event is lost.
• Publish, then write the row. If the insert then fails or rolls back, the scanner reviews a link that does not exist. The event is a phantom.

The fix is to make it one write to one system. In the transactional outbox pattern, the service inserts the link row and an outbox row describing the event in the same database transaction. Both commit or neither does. A separate relay process reads unsent outbox rows, publishes them to the broker, and marks them sent. The relay can crash after publishing and before marking, so it will sometimes publish twice. That is acceptable, because you already built idempotent consumers: the outbox gives you at-least-once publishing, the event id gives you dedup.

With Snip's sharded links table, put the outbox table on the same shard as the row it describes; a transaction cannot span shards cheaply. The relay can poll (SELECT … WHERE sent_at IS NULL ORDER BY id LIMIT 500, simple and a little delayed) or read the database's write-ahead log, which is change data capture (CDC). CDC tails committed changes with no polling load and no outbox table needed for plain row changes. It is the same mechanism that feeds a warehouse, and the data systems design guide covers it.

Snip's click pipeline

This is the pipeline the caching chapter and the database chapter deferred. The redirect path does its lookup, builds the response, and as its last step appends a click event to a durable log such as Kafka or SQS. The event is about 100 B: an event id (a UUID generated in the handler), the code, a timestamp, and coarse fields such as country and user agent. The handler sends it asynchronously and does not wait for the broker; the 302 goes out regardless.

Peak is absorbed by the queue, not the database. At 12,000 events per second, the broker takes about 1.2 MB per second of appends, trivial for a log. Retaining 7 days costs 100M × 100 B × 7 ≈ 70 GB. The counter store sees only what consumers choose to send it.

Batching beats a row per click. The aggregator reads a few thousand events, groups them by code in memory, and issues one update per distinct code. A viral link receiving 10,000 clicks per second turns into a single clicks = clicks + N per batch instead of 10,000 competing updates on one row, which would queue behind that row's lock. If even one update per batch per code is too much, the consumer keeps counters in memory and flushes every second. Partition by event id rather than by code so that a hot code spreads across all consumers; since counts commute, no ordering is needed. The idempotent upsert from the sketch above keeps redelivery from inflating counts.

Counts are eventual, as the consistency chapter set out. The dashboard shows a number that trails reality by the queue's age, usually seconds, up to 15 minutes in the spike worked above. Say so in the product ("updated within 5 minutes") and alert when queue age passes that number.

The raw events leave through a second consumer group. Each click is archived to object storage and rolled up into the analytics store, where "clicks by country by day" queries belong, as described in the data systems design guide. At 100 B per click, that is the 3.6 TB per year from the sizing exercise. Because a separate consumer reads the queue, a slow warehouse load just grows its own backlog. It cannot slow a redirect, which never touches it. That separation is the point of the design: the redirect's only dependency on analytics is an append to a log.

The costs: a click is lost if an app server dies with unsent events buffered, which Snip accepts because it does not bill by the click. A duplicate from a consumer crash is removed by the event id.

✓ Check yourself

Answer from memory before opening each one.

1. A consumer processes a message, writes to the database, and crashes before sending the ack. What does the broker do next, and what must be true of the consumer for the system to stay correct?

   Show answer

   The broker cannot tell that the work finished, so it redelivers (after the visibility timeout in SQS, or from the last committed offset in Kafka). The consumer must be idempotent, so the second run leaves the same state as one run. See Delivery semantics.

2. A vendor's page says its queue offers "exactly-once delivery". What do you ask before relying on it, and why is the claim suspect as stated?

   Show answer

   Ask where the guarantee ends. Over an unreliable network a lost ack makes "consumer died" and "ack lost" indistinguishable, so the broker must choose redelivery or loss. Real offerings deduplicate within a window or inside the broker (Kafka transactions, SQS FIFO), and side effects outside it, like an email, are still yours to make idempotent. See Delivery semantics.

3. Consumers drain 3,000 messages per second. A 10-minute surge brings 9,000 per second against that capacity, then arrivals fall to 1,000 per second. About how long after the surge ends does the backlog clear, and what number should you alert on during it?

   Show answer

   The backlog is (9,000 − 3,000) × 600 = 3.6 million. After the surge the net drain is 2,000 per second, so about 1,800 seconds, or 30 minutes. Alert on the age of the oldest message, since that is what the user feels. The tempting answer, queue depth, hides whether the drain is working. See Backpressure.

4. A handler runs INSERT order and then publish("order.created"). Describe two different ways production can leave the two systems disagreeing, and how an outbox removes both.

   Show answer

   A crash after the insert but before the publish loses the event; publishing first and failing the insert sends an event for an order that does not exist. With an outbox, the order row and the event row commit in one transaction, so both exist or neither does, and a relay publishes later with at-least-once semantics that idempotent consumers absorb. See the outbox.

5. Your colleague wants one Kafka partition per link code so each link's clicks stay in order. What do you ask, and what would you propose instead for counting?

   Show answer

   Ask whether counting needs order at all. Increments commute, so it does not. Keyed partitioning would also send a viral link's 10,000 events per second to one partition and one consumer, a hot spot. Partition by event id to spread the load, and keep per-key ordering for sequences where order matters, like edit then delete. See Ordering and Snip's pipeline.

6. A message in your queue fails with a validation error on every attempt, while another fails with a database timeout. How should the retry policy treat each?

   Show answer

   The validation failure cannot heal, so retrying wastes capacity: send it to the dead-letter queue and alert. The timeout is transient, so retry with exponential backoff and jitter so that many consumers do not hit the recovering database in the same instant. Cap the attempts and dead-letter at the limit. See Poison messages.

Takeaway

• A queue protects a slow component by turning a burst of arrivals into backlog that drains at the consumer's pace; the cost is latency on the queued work, not on the caller.
• An unbounded queue hides overload until latency or memory fails, so bound it, choose between backpressure and shedding in advance, and alert on the age of the oldest message.
• Over an unreliable network you get at-most-once or at-least-once delivery; exactly-once effects come from at-least-once delivery plus an idempotent consumer.
• Idempotency means the event id is fixed at the source and the dedup check commits in the same transaction as the effect.
• Retry only errors that can heal, with exponential backoff and jitter, cap attempts, and send what remains to a dead-letter queue someone watches; do not assume ordering beyond a single partition.
• Writing to a database and a broker cannot be atomic, so write the event to an outbox table in the same transaction and let a relay publish it.
• Snip's redirect only appends a click event to a log; batched, idempotent consumers turn it into eventual counts and feed the analytics store, so neither can slow a redirect. The next question is where each kind of data should live: Choosing a Data Store by Shape.

Further reading

• Designing Data-Intensive Applications — Martin Kleppmann, O'Reilly, 2017. Chapter 11 on stream processing and messaging covers delivery guarantees, ordering and idempotence in depth; chapter 9 explains why exactly-once is subtle.
• Apache Kafka documentation: Design — see "Message Delivery Semantics" for offsets, the idempotent producer and transactions, and what they do and do not guarantee.
• Amazon SQS Developer Guide: Visibility timeout — the redelivery mechanism in concrete numbers; read alongside the dead-letter queue pages for maxReceiveCount.
• Timeouts, retries and backoff with jitter — Marc Brooker, Amazon Builders' Library. Why retries amplify outages and how jitter fixes it.
• Transactional outbox — Chris Richardson, microservices.io. The pattern, its polling and log-tailing relays, and the trade-offs.