Idempotent Payment Webhooks: Why Your Payment Gateway Charges Twice

RedBite takes payments through a gateway — call it PayFlow, same shape as Razorpay or Stripe. Customer pays, PayFlow’s servers POST a webhook to RedBite’s /webhooks/payment endpoint: { event: "payment.success", orderId: "RB-4821", amount: 4999 }. RedBite’s handler marks the order paid and kicks off fulfillment.

One Tuesday, RedBite’s server takes 11 seconds to respond to that webhook — a slow query, nothing exotic. PayFlow’s webhook client times out at 10 seconds, assumes the delivery failed, and retries. RedBite’s original request was still running; it finishes and marks the order paid. Four seconds later, the retry arrives and marks it paid again — and RedBite’s fulfillment code runs a second time, shipping customer Rohan’s order twice.

This isn’t a bug in PayFlow. Retrying is the correct behavior for a webhook client that didn’t get a timely 200 back. If your handler isn’t built to be hit with the same event twice, it will be, and it will do the wrong thing when it happens.

The handler that ships Rohan’s order twice

Here’s the first version almost everyone writes:

app.post('/webhooks/payment', async (req, res) => {
  const { event, orderId, amount } = req.body;

  if (event === 'payment.success') {
    await pool.query(
      'UPDATE orders SET status = $1 WHERE id = $2',
      ['paid', orderId]
    );
    await fulfillOrder(orderId);
  }

  res.status(200).send('ok');
});

Nothing here asks “have I already seen this exact event?” Every delivery — the original, and any retry — runs fulfillOrder again. If fulfillOrder charges a loyalty-points wallet, Rohan gets double points. If it triggers a shipment, he gets two packages. If it’s a subscription renewal, he’s billed twice and gets a second invoice email.

Rohan’s order ships twice.

The first request never failed — it just answered slowly. PayFlow has no way to know that. Treating “I got this event” and “I finished handling this event” as the same thing is the actual bug.

Fix 1: an idempotency table, keyed on PayFlow’s event ID

Every PayFlow webhook carries a unique eventId — PayFlow generates one per delivery attempt of the same logical event, so the retry has the same eventId as the original. Use that as a dedupe key:

CREATE TABLE processed_webhook_events (
  event_id   TEXT PRIMARY KEY,
  processed_at TIMESTAMPTZ NOT NULL DEFAULT now()
);

Before doing any work, try to claim the event ID. If the insert succeeds, nobody’s handled this delivery yet — proceed. If it fails because the row already exists, this is a retry of something already processed — skip it and return 200 anyway, so PayFlow stops retrying:

app.post('/webhooks/payment', async (req, res) => {
  const { eventId, event, orderId } = req.body;

  const { rowCount } = await pool.query(
    `INSERT INTO processed_webhook_events (event_id)
     VALUES ($1)
     ON CONFLICT (event_id) DO NOTHING`,
    [eventId]
  );

  if (rowCount === 0) {
    // already processed this exact delivery — ack and stop
    return res.status(200).send('ok');
  }

  if (event === 'payment.success') {
    await pool.query('UPDATE orders SET status = $1 WHERE id = $2', ['paid', orderId]);
    await fulfillOrder(orderId);
  }

  res.status(200).send('ok');
});

INSERT ... ON CONFLICT DO NOTHING is atomic — it’s a single statement, so there’s no read-then-write gap for two near-simultaneous retries to race through (the same class of bug as the checkout race condition from last post, solved the same way: let the database’s own constraint do the deciding, not application logic). When PayFlow’s retry for Rohan’s payment arrives, the insert hits the existing primary key, rowCount is 0, and fulfillOrder never runs a second time.

Fix 2: the dedupe table doesn’t stop a forged webhook

Here’s what the idempotency table doesn’t protect against: someone who isn’t PayFlow sending a POST to /webhooks/payment with a made-up eventId and a fake payment.success for an unpaid order. Dedupe only catches events you’ve already legitimately seen — it does nothing for an event you’re seeing for the first time, forged or not.

This is also how a captured-and-replayed attack works: if an attacker intercepts one real webhook payload (say, from logs, a misconfigured proxy, or a leaked request) and replays that exact payload a month later with a new HTTP request, your dedupe table will correctly reject it — but only if you kept that old eventId around. If the table’s been pruned, the replay looks brand new again.

The actual defense is verifying PayFlow signed the request. PayFlow signs every webhook body with a secret only you and PayFlow know:

const crypto = require('crypto');

function verifySignature(rawBody, signatureHeader, secret) {
  const expected = crypto
    .createHmac('sha256', secret)
    .update(rawBody)
    .digest('hex');

  return crypto.timingSafeEqual(
    Buffer.from(expected),
    Buffer.from(signatureHeader)
  );
}

timingSafeEqual matters here, not just === — a plain string comparison leaks timing information about how many leading bytes matched, which is enough for an attacker to brute-force the correct signature one byte at a time over many requests.

PayFlow also stamps a timestamp into the signed payload. Check that it’s recent before doing anything else — this is what actually stops a replay of an old, legitimately-signed payload, since the signature alone doesn’t expire:

function isTimestampFresh(timestamp, toleranceSeconds = 300) {
  const ageSeconds = Math.abs(Date.now() / 1000 - timestamp);
  return ageSeconds <= toleranceSeconds;
}

Putting the full pipeline together

app.post('/webhooks/payment', express.raw({ type: 'application/json' }), async (req, res) => {
  const signature = req.headers['x-payflow-signature'];
  if (!verifySignature(req.body, signature, process.env.PAYFLOW_SECRET)) {
    return res.status(401).send('invalid signature');
  }

  const payload = JSON.parse(req.body);
  if (!isTimestampFresh(payload.timestamp)) {
    return res.status(400).send('stale timestamp');
  }

  const { rowCount } = await pool.query(
    `INSERT INTO processed_webhook_events (event_id)
     VALUES ($1)
     ON CONFLICT (event_id) DO NOTHING`,
    [payload.eventId]
  );
  if (rowCount === 0) {
    return res.status(200).send('ok'); // already processed
  }

  if (payload.event === 'payment.success') {
    await pool.query('UPDATE orders SET status = $1 WHERE id = $2', ['paid', payload.orderId]);
    await fulfillOrder(payload.orderId);
  }

  res.status(200).send('ok');
});

Signature check first, before even parsing the body as trusted JSON — an unverified request shouldn’t influence anything, including which code path runs next. Timestamp check second, to reject old signed payloads cheaply before touching the database. Dedupe third, to absorb legitimate retries of events that already passed both checks. fulfillOrder only ever runs after all three hold.

Where to store the dedupe key

For payment webhooks specifically: Postgres, in the same transaction as the order update. Losing a dedupe record for a payment event is the one failure mode worth paying a round-trip to avoid — an in-memory set that forgets on deploy is how RedBite ends up refunding someone for an order it shipped twice.

Hit a webhook retry bug that wasn’t about payments — a different provider, a different failure mode? Drop it in #system-design on Discord.