Exactly-Once Message Processing Is a Lie: The Outbox Pattern

RedBite’s checkout service does two things when an order is placed: write the order to Postgres, then publish an order.placed event to Kafka so the fulfillment service can pick it up and ship it. The obvious code:

async function placeOrder(order) {
  await pool.query(
    'INSERT INTO orders (id, user_id, total) VALUES ($1, $2, $3)',
    [order.id, order.userId, order.total]
  );
  await kafka.send({
    topic: 'order.placed',
    messages: [{ value: JSON.stringify(order) }],
  });
}

This works almost every time. “Almost” is the entire problem. Between the INSERT committing and the kafka.send call returning, the process can crash, the network to Kafka can blip, or Kafka itself can be unreachable for a few seconds during a broker failover. When that happens: the order is in Postgres — the customer was charged — but fulfillment never hears about it. Nobody notices until a support ticket shows up three days later: “where’s my order?”

Flip the order of operations — publish to Kafka first, then write to Postgres — and you get the opposite failure: fulfillment ships an order that doesn’t exist in the database, so the dashboard, refund logic, and customer order history all desync from what’s actually shipping.

There is no way to make “write to Postgres” and “publish to Kafka” a single atomic operation. They’re two different systems with two different commit protocols; nothing enforces that both happen or neither does. Any code that treats them as one step — in either order — has a window where they disagree.

The fix: write the event in the same transaction as the data

The trick is to stop treating “publish to Kafka” as something that happens at all, at order-placement time. Instead, write the event as a row in Postgres, in the exact same transaction as the order — so either both the order and the event exist, or neither does, because they’re now the same atomic write:

CREATE TABLE outbox (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  topic TEXT NOT NULL,
  payload JSONB NOT NULL,
  created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  published_at TIMESTAMPTZ
);

async function placeOrder(order) {
  const client = await pool.connect();
  try {
    await client.query('BEGIN');
    await client.query(
      'INSERT INTO orders (id, user_id, total) VALUES ($1, $2, $3)',
      [order.id, order.userId, order.total]
    );
    await client.query(
      'INSERT INTO outbox (topic, payload) VALUES ($1, $2)',
      ['order.placed', JSON.stringify(order)]
    );
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK');
    throw err;
  } finally {
    client.release();
  }
}

Now there’s no window where one exists without the other: Postgres’s own transaction guarantee — the same ACID guarantee that already protects the orders table — covers the event too, for free, because it’s just another row in the same COMMIT. If the transaction fails, neither the order nor the outbox row exists. If it succeeds, both do.

This moves the problem, it doesn’t remove it — now something has to read rows out of outbox and actually publish them to Kafka. That’s the second half of the pattern: a relay process.

The relay: polling the outbox and publishing for real

A separate background process polls outbox for unpublished rows, publishes each one to Kafka, and marks it published only after Kafka confirms:

async function relayOutbox() {
  const { rows } = await pool.query(
    `SELECT id, topic, payload FROM outbox
     WHERE published_at IS NULL
     ORDER BY created_at
     LIMIT 100`
  );

  for (const row of rows) {
    await kafka.send({
      topic: row.topic,
      messages: [{ value: JSON.stringify(row.payload) }],
    });
    await pool.query(
      'UPDATE outbox SET published_at = now() WHERE id = $1',
      [row.id]
    );
  }
}

setInterval(relayOutbox, 500);

Walk through the crash scenario again with this in place. The checkout transaction commits the order and the outbox row together — guaranteed atomic. If the process crashes right after, the outbox row just sits there with published_at = NULL. The relay process — running separately, polling on its own loop — picks it up on its next pass and publishes it. The order is never silently lost, because “has it been published” is now a durable column in Postgres, not an in-memory fact that evaporates if the checkout process dies.

“Exactly once” is still a lie — and that’s fine

Here’s the part the pattern name oversells: the outbox pattern doesn’t give you exactly-once delivery to Kafka. It gives you at-least-once. If the relay process publishes a row to Kafka successfully, but crashes before the UPDATE published_at = now() commits, the next relay pass sees published_at IS NULL again and publishes the same event a second time. That gap — publish succeeded, mark-as-published didn’t — is small, but it’s not zero, and no amount of careful code closes it, because it’s the same two-systems problem one level down: Kafka’s publish ack and Postgres’s UPDATE are still two separate commits.

The fix for that isn’t “try harder to make it atomic” — it’s the same answer every exactly-once claim eventually reduces to: make the consumer idempotent. Give every outbox row a stable ID, and have the fulfillment service dedupe on it:

async function handleOrderPlaced(event) {
  const inserted = await pool.query(
    'INSERT INTO processed_events (id) VALUES ($1) ON CONFLICT DO NOTHING',
    [event.id]
  );
  if (inserted.rowCount === 0) return; // already handled, skip

  await shipOrder(event);
}

With this, a duplicate delivery from the relay’s own double-publish is harmless — the consumer’s ON CONFLICT DO NOTHING catches it, same dedupe-key shape as the idempotent-webhook fix. The outbox pattern’s real contribution isn’t “exactly once” — it’s turning “did we even try to publish this” into a durable, transactional fact instead of something that can silently vanish on a crash. At-least-once delivery, plus an idempotent consumer, is what “exactly once” actually means in practice.

Running an outbox relay at real volume and hit a case this doesn’t cover? Drop it in #system-design on Discord.