A payment workflow that never double-charges

Design a Payment System

Checkout calls a payment provider that can be slow, can time out after it succeeded, and sends webhooks more than once and out of order. Keep every order's payment state correct through retries, crashes and uncertainty.

Advanced, about 60 minutes, 13 stages

Start the investigation Design it from a blank page See the full walkthrough

The situation

You run checkout for a marketplace that sells online courses. Buyers enter card details into the payment provider's hosted fields, so your servers only ever see a short-lived payment token, never card numbers. Your backend creates an order, asks the provider to charge it, and, once the money has moved, grants the course and emails a receipt.

The provider behaves like every real one. Most charges finish in a second or two, a few take ten, and occasionally a request hangs until something times out. Some cards require a 3-D Secure challenge that completes minutes later. The provider accepts an idempotency key on every write and remembers it for 24 hours, sends signed webhooks for every state change, and lets you look up payments by your own reference.

Buyers double-click. Mobile connections drop mid-request. Your app servers are redeployed during business hours. Finance reconciles every day against the provider's settlement report, and they have found discrepancies before.

What it has to do

Functional

A buyer pays for an order; the order shows paid and the course is granted.
Declined payments show a reason, and the buyer can try a different card.
Payments needing 3-D Secure complete asynchronously after the buyer's challenge.
Admins can refund a paid order.
Finance can reconcile every order against the provider's records daily.

Non-functional

A buyer is never charged twice for the same order.
An order is never marked paid, and a course never granted, unless the provider captured the money.
Payment state survives a process crash or deploy at any instant.
Every state change is auditable: what changed it, when, and based on which provider event.
The buyer gets either an answer or a clear 'processing' state within about 10 seconds.

Constraints and assumptions

Provider latency: p50 1.5 s, p99 10 s; occasional hangs past 30 s. Rate limit 100 requests/second.
Provider idempotency keys are retained for 24 hours.
Webhooks are signed, delivered at least once, in no guaranteed order, retried with backoff for up to 3 days; the endpoint must answer within 10 s.
Stateless app servers behind a load balancer with a 30-second request timeout; one Postgres database.
About 5,000 orders a day, peaking at 50 a minute during course launches.
Card data is tokenized by the provider's client SDK; you are out of scope for storing card numbers.
The provider is the source of truth for whether money moved.
The provider's API can look up a payment by your idempotency key or metadata reference.
Server clocks are roughly synchronized but are not used to order events.

Interview questions it prepares you for

“Design a payment system for an e-commerce checkout.”
“How do you make an API endpoint idempotent?”
“Your service called a payment API and the request timed out. What do you do?”
“Design a wallet or ledger service that must never lose or duplicate money.”
“How would you handle webhooks from Stripe reliably?”

Read and practise next

How Stripe built it · Payments that never charge twice, in their engineers' own words

Concepts to know first: Idempotency, Transactions, State machines for business state.