Design a Product Analytics System, stage 3 of 9: break it
The database is down for twenty minutes
Today the capture API inserts each request's events straight into the database. Twenty minutes at peak is about 100 million events.
System so far· 6 parts
Select a component to see what it is responsible for and which state it owns.
- 1Customers' apps → Capture API: Batches of events
- 2Customers' dashboards → Query service: Chart request
- 3Query service → Postgres: Teams and saved charts
- 4Query service → Events (column store): Aggregate by column
What you need to know
Events arrive at a rate you don't control; the database absorbs them at a rate that varies, sometimes zero during an upgrade. A durable log (a stream like Kafka) between the two lets each run at its own pace. See Append-only logs.
The capture API's job shrinks to: check, stamp, append to the stream, acknowledge.
Work it out
The database is down for 20 minutes at a peak of about 90,000 events a second. Roughly how many million events must be held?Check
Capture returns an error during the outage and relies on the snippet to retry later. What happens to many of those events?After an outage, the workers have the backlog plus live traffic. Size them, and the database, for the catch-up, not only for the average. See Backpressure and capacity.