8 quick system design interview drills

Best response: Keep the order pending, then query or retry the same payment using its original idempotency key.

A timeout reports what the caller observed; it does not prove the provider did nothing. A stable key identifies one logical payment across retries. Keep the order pending until the provider confirms the result, then reconcile any outcome that remains uncertain.

Trade-off: The customer may wait for confirmation, but a retry is less likely to charge them twice.

• Why not “Send a new charge with a new idempotency key.” A new key can create a second logical charge.
• Why not “Treat the timeout as a failed charge and let checkout start over.” The first charge may already have succeeded, so restarting can duplicate it.

Best response: Give the message a stable ID and atomically record it with the state change before acknowledging.

Assume a message can be delivered more than once. Store a deduplication record in the same transaction as the consumer's durable state change, then acknowledge. If the effect is external, pass an idempotency key or use an outbox-driven retry; a broker guarantee alone cannot make an arbitrary side effect exactly once.

Trade-off: Deduplication adds retained state and cleanup work, but makes redelivery safe across the commit/acknowledgement gap.

• Why not “Configure exactly-once delivery and assume the consumer will only run once.” The consumer can still crash between its side effect and acknowledgement.
• Why not “Acknowledge before doing the work so the queue cannot redeliver it.” A crash after acknowledgement but before the durable work can lose the message's effect.

Best response: Give each ownership attempt a newer fencing token and reject writes from an older token.

A lease lets another worker take over after a timeout, but the former owner may be paused or partitioned and later resume. A monotonically increasing fencing token lets the resource reject that stale owner's writes. Persist progress at safe boundaries and make completed steps safe to retry.

Trade-off: The destination must check ownership on writes, and each attempt needs durable progress or idempotent effects.

• Why not “Make the lease longer so it is less likely to expire during normal work.” A longer timeout reduces accidental expiry but delays recovery and does not eliminate stale owners.
• Why not “Assume the lease service will stop the old worker when its lease expires.” A lease service cannot reliably terminate a worker across a network partition.

Best response: Measure load by key, then consider splitting or replicating the hot key based on its read/write pattern.

Cluster averages hide a hot key. Measure the skew first. Replicated reads can help a read-heavy key; splitting or salting can spread writes, but may require fan-in, merge work, or per-account ordering. Move an exceptional tenant only if the workload justifies the operational cost.

Trade-off: Distributing one key's load adds coordination or read/merge complexity; the right choice depends on whether reads, writes, or ordering dominate.

• Why not “Add database nodes while keeping the same partition key and routing.” More nodes do not help when the same key still routes every request to one partition.
• Why not “Increase every service's database connection pool.” More concurrent connections can increase pressure on the already saturated partition.

Best response: Coalesce concurrent fills for the same key, stagger expirations, and bound origin work.

This is a cache stampede: many misses trigger duplicate work for one value. Let one request refresh the key while concurrent callers wait for or reuse its result. Add TTL jitter or refresh a hot key before expiry, and cap origin concurrency. Serve stale data only when the freshness requirement allows it.

Trade-off: Coalescing and stale-while-revalidate add cache coordination and freshness choices; they reduce duplicate origin work during a burst.

• Why not “Increase database capacity so it can handle the extra reads.” Capacity may absorb a burst, but it does not prevent repeated work for the same key.
• Why not “Shorten the TTL so expired values are less stale.” Shorter TTLs cause more frequent misses and can make the burst problem worse.

Best response: Track arrival rate and oldest-message age, then check whether the downstream system can take more work.

Compare arrival and completion rates and track how old the oldest job is; depth alone does not show how long users wait. Add consumers only if the bottleneck can accept the extra load. Otherwise slow or reject producers, shed optional work, or make each job cheaper, then define how the backlog drains.

Trade-off: Backpressure protects downstream systems but can delay or reject work, so the product needs an explicit overload policy.

• Why not “Watch queue depth; if it is high, keep adding consumers until it falls.” Extra consumers can overload a database or dependency and grow the backlog somewhere else.
• Why not “Retry failed messages immediately so they do not hold up the queue.” Immediate retries add load precisely when the system is already falling behind.

Best response: State the recovery-point objective: asynchronous replication can lose recent acknowledged writes; synchronous acknowledgement trades latency for a stronger durability promise.

An asynchronous replica may lag when the primary fails. Be explicit about the acceptable recovery point objective and what an acknowledgement promises. Synchronous replication can strengthen the durability of acknowledged writes, at additional latency and availability cost. Client retries also need stable operation IDs to avoid duplicate effects.

Trade-off: A stronger replication acknowledgement costs latency and can reduce write availability when the replica or network is unavailable.

• Why not “No acknowledged writes are lost because the replica is now primary.” Promotion cannot restore a write that had not reached the replica.
• Why not “Tell every client to retry all writes against the new primary.” A retry cannot know whether the first write committed; it needs an idempotency key or state check.

Best response: Capture changes during the backfill, replay them, compare critical invariants, then switch gradually with a rollback point.

A snapshot misses writes that arrive while it is copied. Capture changes before or during the backfill, replay until caught up, and verify counts plus business invariants. Switch reads gradually only after parity is good, keep the old path available for rollback, and retire it after the new path is proven.

Trade-off: Change capture and verification add temporary operational complexity, but they expose drift before the new database becomes authoritative.

• Why not “Copy a snapshot, then immediately switch all reads to the new database.” Writes made after the snapshot can be absent from the new database at cutover.
• Why not “Start dual writes and assume both databases stay in sync.” A partial failure can update one database but not the other; dual writes need retry, reconciliation, and a cutover plan.