Graceful Shutdown Under Kubernetes SIGTERM: Don't Drop In-Flight Requests

RedBite deploys several times a day. Each deploy rolls pods one at a time: Kubernetes starts a new pod, waits for it to be ready, then tells an old pod to shut down. Most deploys are invisible to customers. Every few days, though, a support ticket shows up: “my payment failed with a connection error, but I wasn’t refunded, and the order never appeared” — and it always correlates with a deploy that happened in that exact minute.

What actually happens when a pod shuts down

Kubernetes doesn’t kill a pod instantly. It sends SIGTERM first, then waits up to terminationGracePeriodSeconds (default 30s) before sending SIGKILL if the process hasn’t exited on its own. That 30-second window exists specifically so a well-behaved process can finish what it’s doing before it dies. Here’s the server code RedBite was actually running:

const server = http.createServer(app);
server.listen(3000);

No SIGTERM handler at all. The default Node.js behavior for an unhandled SIGTERM is to exit immediately — the process terminates mid-request, whatever it was doing. If a customer’s checkout request is mid-flight — already past the “charge the card” step, not yet at “write the order row” — and that’s the exact pod that gets SIGTERM during a deploy, the connection drops with no response, the client sees a network error, and the order never gets written even though the card was charged. Same shape of damage as a missed webhook, triggered by a routine deploy instead of a flaky network.

Fix: handle SIGTERM, stop accepting new work, drain what’s in flight

The fix has two parts that have to happen in the right order: stop accepting new connections immediately, but let already-accepted requests finish before actually exiting.

const server = http.createServer(app);
server.listen(3000);

let shuttingDown = false;

process.on('SIGTERM', () => {
  if (shuttingDown) return;
  shuttingDown = true;
  console.log('SIGTERM received, draining in-flight requests');

  // stop accepting new connections immediately
  server.close(() => {
    console.log('all in-flight requests finished, exiting');
    process.exit(0);
  });

  // hard safety net: if draining takes longer than the grace period
  // allows, exit anyway rather than getting SIGKILLed mid-cleanup
  setTimeout(() => {
    console.error('drain timed out, forcing exit');
    process.exit(1);
  }, 25_000); // leave margin under the 30s terminationGracePeriodSeconds
});

server.close() does exactly the two things needed, in order: it stops the HTTP server from accepting any new incoming connections right away, and it waits for every connection already in progress to finish its response before its callback fires. The checkout request that was mid-flight when SIGTERM arrived gets to complete — charge the card, write the order, send the response — because it was already accepted before server.close() ran. A new checkout request arriving a moment after SIGTERM gets rejected by the load balancer before it ever reaches this pod, because Kubernetes also marks the pod NotReady and routes new traffic elsewhere the instant it starts terminating.

The setTimeout safety net matters because server.close()’s callback only fires once every open connection closes — including a client that opened a connection and is just sitting there, slow or stalled. Without the timeout, one stuck connection could hold the whole shutdown open past terminationGracePeriodSeconds, and Kubernetes would SIGKILL the process anyway, except now mid-way through whatever cleanup was still pending instead of cleanly. Setting the timeout a few seconds under the grace period means the process always exits on its own terms.

Don’t forget the Kubernetes side of the handshake

The application-level fix is necessary but not sufficient on its own — two Kubernetes-side settings interact with it:

• readinessProbe must flip to failing before traffic actually stops. Kubernetes marking a pod as terminating and the load balancer actually stopping new traffic to it aren’t the same instant — there’s a small propagation window. A preStop hook that sleeps a few seconds before the container actually receives SIGTERM closes that gap:

  lifecycle:
    preStop:
      exec:
        command: ["sh", "-c", "sleep 5"]

• terminationGracePeriodSeconds has to be comfortably longer than your slowest realistic in-flight request, not just the typical one. If RedBite’s checkout occasionally takes 20 seconds end-to-end under load, the default 30-second grace period leaves little margin once the preStop sleep is subtracted from it. Set it explicitly rather than trusting the default to happen to be enough:

  terminationGracePeriodSeconds: 45

What RedBite shipped

Added the SIGTERM handler with server.close() and a safety-net timeout, a 5-second preStop sleep, and raised terminationGracePeriodSeconds to 45. Confirmed it under load by curling the checkout endpoint in a loop during a live deploy and watching for any connection reset — zero, across several deploys, versus the occasional drop beforehand. The database-level fixes from earlier posts (idempotent webhooks, the outbox pattern) matter because distributed systems fail in ways no single process can fully prevent; graceful shutdown is the version of that discipline for the deploy process itself, which fails on a schedule RedBite controls directly — no reason to leave it unhandled.

Running something that doesn’t drain cleanly even with server.close() — a long-lived WebSocket, a streaming response? Drop it in #system-design on Discord.