How to design a Notification System

When a provider caps your sending rate, that cap is a shared budget. Every email of every kind draws from it: security alerts, comment notifications and marketing alike. The first numbers to work out are how long the big jobs occupy that budget, and what is left for everything else meanwhile.

An idempotency key is an id you send with a request so the receiver can recognise a retry: "you already did 812, here's the same result". Payment APIs usually accept one. This email provider doesn't.

Without it, a request that times out is ambiguous: the email may or may not have been sent, and the provider gives you no way to ask.

When a request does two things, the slower and less reliable one sets the pace for both. If saving a comment also sends emails, the comment is as slow as the email provider and fails whenever the provider fails.

The fix is to make the comment record that notifications are owed, and let something else do the sending later.

A dual write updates two systems separately, say the database and then a queue. A crash between the two leaves them disagreeing: a comment with no notification, or a notification for a comment that was rolled back.

The transactional outbox avoids it: write the comment and an event row ("comment 991 created") in the same database transaction. Either both exist or neither does. A relay later reads new outbox rows and publishes them. See Transactional outbox.

Event streams deliver at least once. A consumer that processes an event and crashes before recording its progress will see that event again after restarting. Two consumer instances can also briefly process the same event during a rebalance.

So the planner must produce the same result however many times it sees an event.

The most robust way is to give each notification an identity derived from its cause: (recipient, event id, channel). Store it under a unique constraint and insert with "ignore on conflict":

INSERT INTO notifications (dedupe_key, user_id, …)
VALUES ('u88:evt991:email', 88, …)
ON CONFLICT (dedupe_key) DO NOTHING;

The first insert creates the row. Every replay hits the constraint and does nothing.

A notification is planned at one moment and sent at another. Normally the gap is seconds. Behind a large announcement, or during a provider outage, it can be an hour.

Anything decided at planning time can be out of date by sending time. The question is which decisions must be checked again just before the irreversible step.

There are good reasons to check at planning time as well. At fan-out scale, not creating unwanted deliveries saves work in every queue and sender.

When a send-time check cancels a delivery, record it as suppressed rather than deleting it. Support can then answer "why didn't I get this?" from the record.

A pipeline that crosses several services stays reliable when every hop follows the same rhythm: record durably, act, record the outcome. Each record lets the next step be retried without redoing or losing work.

When tracing a request, look for where the user's part ends. Everything after that can be slow or retried without them seeing it.

Push providers (APNs, FCM) report a device token as invalid when the app has been uninstalled or the token has rotated. Sending to it again will fail every time.

A sender that removes tokens as soon as a provider rejects them keeps each user's device list accurate without a separate cleanup job.

A queue's visibility timeout is a lease: once a sender receives a message, the queue hides it for that long. If the sender doesn't acknowledge in time, the message reappears and another sender can receive it.

So the visibility timeout has to be longer than the slowest the work can take, or the sender has to extend it while it works.

A second safeguard: record state before the side effect. Before calling the provider, the sender changes the delivery from queued to sending with a fresh attempt token, in one conditional update:

UPDATE deliveries SET status = 'sending', attempt_token = $2
WHERE id = $1 AND status = 'queued'

If another sender already claimed it, zero rows change and this one skips the delivery.

Per-message retries with backoff assume failures are independent: this message failed, the next one might not. A provider outage breaks that assumption. Every message fails for the same reason at the same time.

A circuit breaker watches failures across all calls to a dependency. When the failure rate crosses a threshold it "opens": calls stop for a while. Then it lets a few probe calls through, and closes again once they succeed.

Failures also differ in kind, and each kind deserves a different response:

Two different problems hide behind "urgent messages wait":

• Order: urgent work is behind other work in the same queue. Priorities or separate queues fix this.
• Capacity: even at the front of the queue, there is no capacity left to serve it. Only reserving capacity fixes this.

Here the capacity is the provider's 500 emails a second for the whole account.

Reserving capacity means giving each class its own token bucket that draws from the account limit, for example critical 50/s, transactional 150/s, bulk 300/s. Bulk can't take what's reserved for critical, so a critical alert waits only for other critical alerts.

Unused reserved tokens can be lent to bulk each second, so reservation costs little when there are no alerts.

A digest replaces "send when it happens" with "periodically, send what accumulated". Two new things need care:

• The digest itself is a notification, sent by a scheduled job that can run twice, so it needs an identity too, such as (user, issue, hour window).
• Membership: which notifications went into which digest. If that isn't recorded atomically with the digest, a crash can put one notification in two digests, or in none.

"Quiet hours from 22:00 to 07:00" means the user's 22:00. A server running in UTC has to convert using the user's time zone, including daylight-saving changes, which shift the offset twice a year in many regions.

A delivery moves through a small set of states. Each move is a conditional update: it only happens if the row is in the expected state, so concurrent senders can't both make it.

The order of steps inside one attempt matters:

1. Claim (queued → sending).
2. Re-check preferences: the last chance to cancel before something irreversible.
3. Take a token from the class's rate limit.
4. Call the provider.
5. Record the outcome, conditioned on the attempt token.

Explaining a guarantee to a non-engineer comes down to three plain statements:

• What we guarantee, in terms of what the person sees.
• What we can't, and the one concrete situation where it happens.
• What we chose in that situation, and why it's the better failure for them.

Avoid words like "idempotent" or "at least once". Describe the outcome.