At 10:00 on launch day a client's flash sale opened to 30,000 waiting customers. By 10:06 the support inbox had 140 emails saying the same thing: I was charged, but I have no order.
The obvious suspects
The payment gateway's dashboard showed every callback delivered with a 200 response. Our API was under load, so we scaled from 4 to 12 pods. The failure rate stayed at roughly 8%, which told us capacity was not the problem.
The real clue was in the timestamps. For every missing order, the browser redirect and the gateway webhook had arrived within 40 milliseconds of each other.
Two writers, one row
Both handlers read the pending order, saw it unpaid, and tried to mark it paid. One of them also cleared the cart and, in a bad branch, rolled back the order it had just written. We fixed it with an idempotency key per payment intent and a row lock on the order.
await db.transaction(async (tx) => {
const order = await tx.orders
.where({ paymentIntent: key })
.forUpdate().first();
if (order.status === "paid") return order; // second writer exits
return tx.orders.markPaid(order.id);
});We have run four sales on the same system since. Zero dropped orders. Every payment handler we write now starts from the assumption that it will be called twice.