Skip to content
sysgeeks

Shard a Live Database Without Downtime, stage 6 of 9: decide

Is the copy right?

Before the shards serve a single user, the team wants evidence that they match. Notion compared randomly sampled records and ran dark reads: queries sent to both databases with results compared, while users still got the monolith's answer. Stripe used GitHub's Scientist library for the same kind of comparison.

System so far· 7 parts
12345678CLIENTUsersSERVICEApplicationserversDATABASEMonolithPostgresLOG / STREAMWrite audit logWORKERBackfilland catch-upSERVICEConnectionpoolersDATABASEShard hosts

Select a component to see what it is responsible for and which state it owns.

  1. 1Users → Application servers: Requests
  2. 2Application servers → Monolith Postgres: Reads and writes until cutover
  3. 3Application servers → Write audit log: Log every write
  4. 4Backfill and catch-up → Monolith Postgres: Copy existing rows
  5. 5Backfill and catch-up → Write audit log: Replay logged writes
  6. 6Backfill and catch-up → Shard hosts: Write rows by shard
  7. 7Application servers → Connection poolers: Queries by shard
  8. 8Connection poolers → Shard hosts: Schema on host
  • Request / response
  • Asynchronous

What you need to know

0 of 2 checks done

  1. Verification is Reconciliation between two stores: compare, explain every difference, fix its cause. Two techniques:

    • Sampled comparison: pick random rows and compare them field by field.
    • Dark reads: send real production queries to both stores, compare results, and still give users the old store's answer.

  2. Check

    Row counts match for every table. Is the copy correct?