Leader Election Without ZooKeeper: A Postgres Lease, Not a New Service

RedBite runs a worker process that generates the nightly sales report at 02:00. For reliability, it runs on 3 instances behind a process supervisor — if one crashes, the others keep the service up. This is correct for every part of the worker except the nightly report job: at 02:00, all 3 instances wake up, all 3 see “time to run the report,” and all 3 run it. RedBite gets 3 identical emails, 3x the load on the reporting query, and if the report writes anything (a summary row, a cache invalidation), it happens 3 times.

The job needs exactly one of the 3 instances to run it, and the other 2 to sit out — without hardcoding “instance #1 is special,” because any instance can be the one that’s down when 02:00 comes around. This is leader election: picking exactly one node, among several, to do a job, where any node could be that leader and the choice has to adapt if it goes away.

Why you don’t need ZooKeeper for this

Leader election has a well-known heavyweight answer: run ZooKeeper or etcd, a purpose-built consensus system, and have instances register ephemeral nodes / watch keys to detect who’s leader. That’s the right tool when you need strong, low-latency consensus across many coordinating processes — a Kafka cluster picking a controller, for instance. For “exactly one of 3 cron workers should run a nightly job,” it’s a new, operationally expensive service for a problem RedBite’s existing Postgres database can already solve, because Postgres already gives you the one primitive leader election actually needs: atomic conditional writes with a single source of truth.

The lease table

A lease is a row that says “instance X holds the leader role until time T.” Any instance can try to claim it; the claim only succeeds if the row doesn’t exist, or its lease has expired:

CREATE TABLE leader_lease (
  job_name TEXT PRIMARY KEY,
  holder TEXT NOT NULL,
  expires_at TIMESTAMPTZ NOT NULL
);

Each instance, on a short interval, tries to become or stay the leader for nightly_report:

const LEASE_DURATION_MS = 15_000;
const instanceId = `${os.hostname()}-${process.pid}`;

async function tryAcquireLease(jobName) {
  const result = await pool.query(
    `INSERT INTO leader_lease (job_name, holder, expires_at)
     VALUES ($1, $2, now() + interval '15 seconds')
     ON CONFLICT (job_name) DO UPDATE
       SET holder = $2, expires_at = now() + interval '15 seconds'
       WHERE leader_lease.expires_at < now()
          OR leader_lease.holder = $2
     RETURNING holder`,
    [jobName, instanceId]
  );
  return result.rows[0]?.holder === instanceId;
}

The WHERE clause on the DO UPDATE is the entire mechanism. A row update only happens if either the existing lease has already expired (expires_at < now()), or the row is already held by this same instance (renewing its own lease). If another instance currently holds a live lease, the WHERE condition is false, ON CONFLICT does nothing, and RETURNING gives back whatever holder is actually in the row right now — which won’t match instanceId, so tryAcquireLease correctly returns false. Postgres’s row-level write is atomic, so there’s no window where two instances both see “lease is free” and both write — the same guarantee that made SELECT ... FOR UPDATE work for the checkout race condition, applied here to a leadership claim instead of a stock count.

Using the lease: only the leader runs the job

Wire this into the actual nightly job — each instance checks lease ownership right before doing the work, not just once at startup, so a leadership change mid-run is handled too:

async function runNightlyReportIfLeader() {
  const isLeader = await tryAcquireLease('nightly_report');
  if (!isLeader) return; // some other instance has it — sit this out

  await generateAndSendReport();
}

setInterval(runNightlyReportIfLeader, 15_000);

At 02:00, whichever instance’s setInterval tick happens to run tryAcquireLease first wins the claim for the next 15 seconds and generates the report; the other two instances find the lease held and return immediately, doing nothing. If the leader crashes mid-report, its lease simply expires 15 seconds after its last renewal — the next instance to call tryAcquireLease after that point picks it up. No instance needs to detect “the leader died” directly; they just need to notice “the lease looks free now,” which a plain expires_at < now() check already does.

What this doesn’t give you — and when that matters

This lease pattern is deliberately a soft mechanism, not a strict exclusion guarantee, and that gap matters for how long-running the leased work is:

• There’s a window where two instances can briefly both believe they’re leader — if the current leader stalls (GC pause, slow I/O) long enough for its lease to expire, but it later resumes and keeps working under the assumption it’s still leader, while a new instance has already claimed the lease. For a report job that finishes in a few seconds, the 15-second lease window makes this exceedingly unlikely. For a job that runs for minutes, either lengthen the lease substantially past the job’s runtime, or make the job itself idempotent (same shape as the webhook and outbox fixes) so a double-run is harmless instead of corrupting state.
• This is not a replacement for ZooKeeper/etcd-grade consensus when you need sub-second failover or many processes coordinating on rapidly changing state — a Postgres row poll has latency bounded by your polling interval, not milliseconds. For “which of 3 cron workers runs a nightly job,” that bound is irrelevant; for a service mesh picking a live routing leader under tight SLAs, it would not be.

For a job this simple, a 30-line lease table in the database RedBite already runs is the right amount of machinery — correct, auditable with a plain SELECT * FROM leader_lease, and zero new infrastructure to operate.

Running leader election on something longer-lived than a nightly job? Drop the shape in #system-design on Discord.