Too Many Open Files: Linux File Descriptor Limits in Production

RedBite’s API server runs fine for hours, then under a traffic spike, every new request starts failing with:

Error: EMFILE: too many open files, open '/var/log/app/access.log'

Restarting the process fixes it instantly — for a few more hours. Nothing in the code looks wrong. It happens every time traffic is high, never in dev, never in the staging environment with low traffic. EMFILE is Linux telling the process, plainly: you have hit your file descriptor limit, and the kernel will not give you a new one until you close something.

What a file descriptor actually is

Every socket, every open file, every pipe a process holds is represented by a small integer — the file descriptor — that the kernel tracks per process. 0, 1, 2 are stdin/stdout/stderr; everything else your process opens gets the next free number. Each one costs the kernel a slot in a per-process table, and that table has a hard ceiling:

$ ulimit -n
1024

1024 open files, per process, is the default on most Linux distributions. Every incoming HTTP connection RedBite’s API accepts is a file descriptor. Every outbound connection to Postgres, Redis, or an external API is another one. Every log file handle is another. In dev, with one engineer poking at the API, you might have 20 descriptors open at once — nowhere near 1024. Under real production load — thousands of concurrent client connections plus a connection pool to Postgres plus Redis connections plus log file handles — that number climbs fast, and 1024 stops being “obviously enough.”

Reproducing it

A minimal server that holds connections open (say, long-polling clients waiting for an update) burns through descriptors under concurrent load:

const http = require('http');

const server = http.createServer((req, res) => {
  // simulate a slow endpoint that holds the connection open
  setTimeout(() => res.end('ok'), 30000);
});

server.listen(3000);

Hit this with enough concurrent requests and you’ll see exactly RedBite’s production symptom. Check how many file descriptors the process is actually holding, live:

$ ls /proc/$(pgrep -f "node server.js")/fd | wc -l
1019

That number climbing toward ulimit -n’s ceiling, under load, with no plateau, is the signature of either (a) genuinely needing more concurrent connections than the limit allows, or (b) leaking descriptors — opening them and never closing them. Both get the same EMFILE error; they need different fixes.

Fix 1: raise the limit, if the load is legitimately that high

If RedBite’s API genuinely needs to hold 5,000 concurrent connections open and that’s expected, intentional traffic — not a leak — the limit itself is just configured too low for the workload. Raise it at the systemd unit or process level:

# /etc/systemd/system/redbite-api.service
[Service]
LimitNOFILE=65536

Or for a process started directly, before exec:

ulimit -n 65536
node server.js

This is correct only if the number of concurrently open descriptors is actually load-proportional and bounded — connections that open, get served, and close within a reasonable time. Raising the limit on a process that’s leaking descriptors doesn’t fix anything; it just buys more time before hitting the new, higher ceiling. Check the leak case first.

Fix 2: find and close the leak

The most common real cause in a Node API isn’t “too much legitimate traffic” — it’s a response path that doesn’t close something it opened. A classic one: reading a file to stream it back, but never handling the error case that skips the close:

const fs = require('fs');

app.get('/download/:file', (req, res) => {
  const stream = fs.createReadStream(`/var/data/${req.params.file}`);
  stream.pipe(res);
  // if the client disconnects mid-stream, 'close' may never fire,
  // and this stream's fd is never explicitly released
});

Under light dev traffic, a handful of abandoned streams a day go unnoticed — the process has thousands of descriptors to spare. Under production load, with clients disconnecting mid-download constantly (mobile users losing signal, people closing tabs), the leaked descriptors pile up continuously and never get reclaimed until the process restarts. Explicitly destroy the stream on client disconnect so the descriptor is released immediately instead of waiting on a close event that may not fire:

app.get('/download/:file', (req, res) => {
  const stream = fs.createReadStream(`/var/data/${req.params.file}`);
  stream.pipe(res);

  req.on('close', () => {
    stream.destroy();
  });
});

To find leaks like this before they take down production, watch the descriptor count over time rather than waiting for EMFILE:

watch -n 2 'ls /proc/$(pgrep -f "node server.js")/fd | wc -l'

A count that climbs under steady load and never comes back down between traffic waves is a leak. A count that rises and falls with traffic, staying under the limit, is normal — and if it’s getting close to the ceiling on legitimate peaks, that’s when Fix 1 applies.

Why epoll is why this scales at all

It’s worth knowing what makes thousands of open connections on one process feasible in the first place, instead of assuming more descriptors always means more overhead. The naive way to watch N sockets for activity is to ask the kernel about each one in a loop — O(n) work every tick, whether or not anything happened. Linux’s epoll (what Node’s event loop uses under the hood via libuv) instead lets the kernel tell the process which specific descriptors have activity, without scanning all of them:

int epfd = epoll_create1(0);
epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &event);
int n = epoll_wait(epfd, events, MAX_EVENTS, timeout);
// 'events' now holds only the fds that actually have data ready

This is why a single Node process can hold thousands of concurrent connections without thousands of threads or an O(n) poll loop — the per-descriptor cost of waiting is cheap. The limit you’re actually hitting with EMFILE isn’t a performance ceiling, it’s a bookkeeping ceiling: the kernel’s per-process table of “which descriptors does this process own” running out of slots, independent of how efficiently those descriptors are being watched.

What RedBite actually shipped

1. watch’d descriptor count in production for a day — confirmed it climbed steadily under normal traffic and never dropped, even during quiet overnight hours. Leak, not legitimate load.
2. Found the unclosed stream on /download/:file, added the req.on('close') handler.
3. Raised LimitNOFILE to 65536 anyway as headroom — not as the fix, as insurance against the next leak buying more time before it pages someone.

Raising the limit alone would have delayed the next incident, not prevented it. The leak was the actual bug; the limit was just the number that made it visible.

Chasing an EMFILE that isn’t a simple unclosed stream? Drop the /proc/<pid>/fd trend in #os-internals on Discord.