Skip to content
sysgeeks

Design a Payment System, stage 10 of 13: break it

Thirty-seven payments stuck in processing

Every component behaved as designed: the provider retried as documented, your handler rejected what it could not verify, and the attempts stayed in processing instead of guessing. Now the system needs a way to finish what messages could not.

System so far· 7 parts
12345678CLIENTBuyer's browserSERVICECheckout APIDATABASEPostgresEXTERNALPayment providerSERVICEWebhook receiverWORKERFulfillmentworkerEXTERNALEmail provider

Select a component to see what it is responsible for and which state it owns.

  1. 1Buyer's browser → Payment provider: Card entry and 3-D Secure in hosted fields
  2. 2Buyer's browser → Checkout API: Pay (Idempotency-Key header); poll status
  3. 3Checkout API → Payment provider: Create payment with the attempt's key
  4. 4Checkout API → Postgres: Claim attempt; conditional transitions
  5. 5Payment provider → Webhook receiver: Signed events: at least once, unordered
  6. 6Webhook receiver → Postgres: Dedupe by event id; forward-only transition
  7. 7Fulfillment worker → Postgres: Claim outbox rows; record completion
  8. 8Fulfillment worker → Email provider: Send receipt with idempotency key
  • Request / response
  • Asynchronous

What you need to know

0 of 1 checks done

  1. Messages can't guarantee resolution: a webhook endpoint broken for longer than the provider's retry window loses those events permanently. Reconciliation asks the source of truth directly and fixes what the messages missed. See Reconciliation.

    A reconciler has no special powers. It learns a fact, then submits it through the same transitions as the API and webhooks, so it can't conflict with them.

  2. Check

    The reconciler asks the provider about an attempt that has been processing for 20 minutes. The provider has no record of it. What should happen?