Skip to content
sysgeeks

Design a Product Analytics System, stage 9 of 9: defend it

Defend the pipeline

Your interviewer: "That is a stream, a fleet of workers and a separate database. Why not keep Postgres with good rollups, or send everything to a managed warehouse?"

System so far· 8 parts
12345678CLIENTCustomers' appsEDGECapture APILOG / STREAMEvent streamWORKERIngestionworkersDATABASEEvents(column store)DATABASEPostgresSERVICEQuery serviceCLIENTCustomers'dashboards

Select a component to see what it is responsible for and which state it owns.

  1. 1Customers' apps → Capture API: Batches of events
  2. 2Customers' dashboards → Query service: Chart request
  3. 3Query service → Postgres: Teams and saved charts
  4. 4Query service → Events (column store): Aggregate by column
  5. 5Capture API → Event stream: Append, then acknowledge
  6. 6Ingestion workers → Event stream: Read a partition
  7. 7Ingestion workers → Postgres: Who is this ID?
  8. 8Ingestion workers → Events (column store): Batch insert

What you need to know

0 of 1 checks done

  1. A good defence ties every component to a requirement:

    ComponentRequirement it serves
    Column storead hoc questions over huge volumes
    Streamno lost events through outages and spikes
    Copied person propertiesfast filters without joins
    Hashed sharding and quotasfairness between customers

    A component you can't tie to a requirement shouldn't be in the design.

  2. Check

    When would Postgres with rollups be the better choice?