Question 1

What problem do online data migrations solve?

Accepted Answer

Data outgrows its home: a table needs a new shape, a database needs sharding, a store needs replacing. The system cannot stop while billions of rows move, and every row keeps changing during the copy. A naive "copy, then switch" loses every write made during the copy, and a big-bang cutover has no way back if the new store is wrong.

Question 2

How do online data migrations work?

Accepted Answer

The safe pattern moves data in reversible steps, verifying at each one: Dual write. Every new write goes to the old store (still the source of truth) and the new one. Writes can be mirrored in application code, through a change log (database replication, CDC, an audit log), or by tailing the binlog. A log is harder to get wrong than scattered application writes. Backfill. Copy historical data into the new store. The backfill must not overwrite newer dual-written values (use versions, timestamps or "insert if absent"), and it must be throttled so it does not starve production traffic. Verify. Compare the stores: sample rows, count by key range, and run shadow (dark) reads that query both and alert on mismatches while users still get the old answer. Switch reads, gradually (by percentage or tenant), keeping dual writes so you can switch back. Switch writes so the new store becomes the source of truth. Often this is a brief pause, or a log-based catch-up followed by redirecting traffic. Reverse replication keeps the old store current so rollback stays possible. Clean up: stop writing the old store and delete it once nothing reads it. The order matters: until step 5, the old store is authoritative and every step can be undone.

Question 3

What do online data migrations assume?

Accepted Answer

Writes can be captured completely (in code or from a log). Rows have a version or timestamp so the backfill can avoid clobbering newer data. The data can be compared between stores cheaply enough to verify.

Question 4

How do online data migrations fail?

Accepted Answer

Lost writes during the copy: Writes that arrive between snapshot and switch never reach the new store. Backfill clobbers newer data: An old snapshot value overwrites a newer dual-written one. Partial dual writes: One store's write fails and the other's succeeds, so they diverge silently. No way back: Writes moved to the new store with nothing keeping the old one current.

Question 5

What are the alternatives to online data migrations?

Accepted Answer

Maintenance window: Brief downtime is acceptable and the dataset copies within it. Expand/contract schema changes: The change is to columns in one database: add the new shape, migrate code, remove the old. Leave old data in place: Only new data needs the new store; old data can be read from the old one until it ages out.

Online data migrations

Learn it

Quick reference

How it goes wrong

Instead, consider

In practice

It assumes

Explain it in your own words

Where you practise it

Further reading