Clock Skew: Why Date.now() Lies Across Servers, and Hybrid Logical Clocks

RedBite’s order service runs on 4 API instances behind a load balancer. Every event gets a timestamp with new Date() at the moment it’s recorded, and the audit log sorts by that timestamp to reconstruct “what happened in what order.” One day, support pulls up a customer’s order history and the sequence makes no sense: order.shipped shows a timestamp before order.paid for the same order — a physical impossibility, since fulfillment only fires after payment confirms.

Both events really did happen in the right order. order.paid fired on instance A, then — correctly, causally after it — order.shipped fired on instance B a few hundred milliseconds later. The bug isn’t in the event logic. It’s that instance A’s system clock was running 400ms ahead of instance B’s, so B’s genuinely-later event got a wall-clock timestamp that reads as earlier.

Why server clocks disagree at all

Every server syncs its clock against NTP (Network Time Protocol) servers periodically, but NTP sync is not instantaneous or perfectly precise — network latency to the time source varies, local clock drift happens between syncs (cheap crystal oscillators drift a few milliseconds per hour even on good hardware), and different servers may sync against slightly different upstream sources. The result: at any given instant, two servers’ system clocks can differ by anywhere from a few milliseconds to, under a bad NTP config or a VM host hiccup, hundreds of milliseconds or more. Date.now() on instance A and Date.now() on instance B, called at the exact same physical moment, can return different numbers — and there’s no way to know by how much just by looking at either one.

// instance A, 10:00:00.100 wall clock
console.log(Date.now()); // 1735700400100

// instance B, same physical instant, clock running 400ms behind
console.log(Date.now()); // 1735700399700

Across one machine, Date.now() is monotonic enough for logging — it basically always increases call to call (ignoring NTP step corrections). Across machines, it’s not a reliable ordering at all. Comparing two timestamps from two different servers tells you which server’s clock read a bigger number, not which event actually happened first.

Fix 1: use the database’s clock, not each server’s

The cheapest fix for RedBite’s specific bug: don’t timestamp events with the application server’s clock at all. Let Postgres assign the timestamp when the row is written:

INSERT INTO order_events (order_id, type, occurred_at)
VALUES ($1, $2, now());

Every event, regardless of which API instance wrote it, now gets a timestamp from the same clock — Postgres’s. There’s no cross-machine clock disagreement because there’s only one machine’s clock involved in the comparison. This fixes RedBite’s audit-log-ordering bug completely, and it’s the right fix whenever every event genuinely funnels through one database that can be the single timestamp authority.

This stops working the moment events don’t all pass through one database — multiple regions each with their own primary, or events that need ordering before they ever reach a database at all (two services exchanging messages directly). That’s a structurally different problem: no single clock exists to delegate to.

Fix 2: Hybrid Logical Clocks — ordering without a shared clock

A Hybrid Logical Clock (HLC) solves the general version: give every event a timestamp that combines wall-clock time with a logical counter, such that timestamps are comparable and causally correct across machines, even with clock skew — without needing a single shared clock authority.

The core idea: each node tracks its own HLC, which is (physicalTime, counter). On every event, update it like this:

class HybridLogicalClock {
  constructor() {
    this.lastPhysical = 0;
    this.counter = 0;
  }

  now() {
    const physical = Date.now();
    if (physical > this.lastPhysical) {
      this.lastPhysical = physical;
      this.counter = 0;
    } else {
      this.counter += 1; // clock didn't advance (or went backward) — bump counter instead
    }
    return { physical: this.lastPhysical, counter: this.counter };
  }

  // call this when receiving a timestamp from another node
  update(remote) {
    const physical = Math.max(Date.now(), remote.physical, this.lastPhysical);
    if (physical === this.lastPhysical && physical === remote.physical) {
      this.counter = Math.max(this.counter, remote.counter) + 1;
    } else if (physical === this.lastPhysical) {
      this.counter += 1;
    } else if (physical === remote.physical) {
      this.counter = remote.counter + 1;
    } else {
      this.counter = 0;
    }
    this.lastPhysical = physical;
    return { physical: this.lastPhysical, counter: this.counter };
  }
}

The rule that matters: whenever one node receives a timestamp from another node (a message, an event, a reply), it calls update() with that remote timestamp. The local clock’s physical value becomes at least as large as whatever it just learned about from the other node — so a node can never emit a timestamp earlier than one it has already observed from somewhere else. That’s the property wall-clock Date.now() doesn’t have: causality gets baked directly into the timestamp, instead of relying on two independent clocks happening to agree.

Walk through RedBite’s scenario with HLC instead of raw Date.now(). Instance A emits order.paid with HLC (T_A, 0). The message carrying that event — or the request that triggered shipping — passes (T_A, 0) to instance B. Instance B calls update((T_A, 0)) before emitting order.shipped: even if B’s raw wall clock reads behind T_A due to skew, update() forces B’s next timestamp’s physical component to be at least T_A, with the counter incremented — guaranteeing order.shipped’s HLC timestamp sorts strictly after order.paid’s, regardless of how far the two machines’ physical clocks actually disagree.

Which one RedBite should ship

For the audit log specifically, Fix 1 is the right call — it’s a one-line change and every event already writes to the same Postgres instance. Reach for HLC when ordering has to survive across systems that don’t share a database at all: cross-region replication, direct service-to-service events, or any place “which happened first” needs to stay correct even when the two clocks involved can’t be trusted to agree.

One thing HLC doesn’t fix: NTP stepping a clock backward abruptly (a bad sync correcting a large drift) can still make Date.now() go backward on a single machine between two calls. The counter bump in now() above is exactly the guard for that case — don’t skip it even on a single-node clock.

Chasing an ordering bug HLC alone doesn’t explain? Drop it in #edge-cases on Discord.