Skip to content
sysgeeks

Shard a Live Database Without Downtime, stage 4 of 9: change it

Move billions of rows, live

Notion captured writes with an application-level audit log rather than Postgres logical replication, because replicating the initial snapshot of tables this large through logical replication was too slow for them at the time.

System so far· 5 parts
1234CLIENTUsersSERVICEApplicationserversDATABASEMonolithPostgresSERVICEConnectionpoolersDATABASEShard hosts

Select a component to see what it is responsible for and which state it owns.

  1. 1Users → Application servers: Requests
  2. 2Application servers → Monolith Postgres: Reads and writes until cutover
  3. 3Application servers → Connection poolers: Queries by shard
  4. 4Connection poolers → Shard hosts: Schema on host

What you need to know

0 of 1 checks done

  1. Moving live data has one shape, whatever the tools. See Online data migrations:

    1. Capture every new write (here, an audit log).
    2. Backfill existing rows.
    3. Catch up by replaying captured writes until the copy trails by seconds.
    4. Verify the copy against the original.
    5. Switch reads and writes, briefly.
    6. Keep the old store as a fallback for a while.

  2. Check

    Why start capturing writes before the backfill begins?