Design a Notification System, stage 8 of 11: change it
The announcement and the security alert
Same provider, same account, same 500/s.
System so far· 10 parts
Select a component to see what it is responsible for and which state it owns.
- 1Product services → Event stream: Domain events via outbox
- 2Event stream → Notification planner: Events, at least once
- 3Notification planner → Notifications DB: Insert notifications under dedupe keys
- 4Notification planner → Delivery queues: Deliveries by priority class
- 5Channel senders → Delivery queues: Claim deliveries
- 6Channel senders → Email provider: Send within quota
- 7Channel senders → Push services: Send to each device
- 8Web and mobile apps → Inbox API: Inbox and read state
- 9Inbox API → Notifications DB: Read notifications
- Asynchronous
- Request / response
What you need to know
0 of 2 checks done
Two different problems hide behind "urgent messages wait":
- Order: urgent work is behind other work in the same queue. Priorities or separate queues fix this.
- Capacity: even at the front of the queue, there is no capacity left to serve it. Only reserving capacity fixes this.
Here the capacity is the provider's 500 emails a second for the whole account.
Check
Alerts get the highest priority in a shared queue, but bulk senders have already used this second's 500 sends. When does the alert go out?Reserving capacity means giving each class its own token bucket that draws from the account limit, for example critical 50/s, transactional 150/s, bulk 300/s. Bulk can't take what's reserved for critical, so a critical alert waits only for other critical alerts.
Unused reserved tokens can be lent to bulk each second, so reservation costs little when there are no alerts.
Work it out
With 300 of the 500 sends a second reserved for bulk, about how many hours does the 5-million-user announcement take?