File Descriptor Exhaustion: Keep an Embedded Service Diagnosable at EMFILE

At EMFILE, stop admitting work that needs new descriptors and preserve diagnostics through resources already open. Identify whether the exhausted resource is the process’s descriptor table, the system-wide open-file pool or another allocation path before changing limits. A larger descriptor limit can postpone a leak while leaving the same failure mechanism intact.

This guide uses Linux man-pages 6.12 and a single process owning a nonblocking service listener. The proposed exhaustion and recovery tests belong in a disposable process with a bounded descriptor budget. No exhaustion fixture, resource-limit change or production connection test was executed for this article.

Classify the failing operation and error

Record the exact call, errno, current lifecycle phase and existing descriptor count using a pre-established diagnostic path. open, accept, pipe, dup and descriptor receipt can fail at different ownership transitions. Preserve the original error before a logging or cleanup call changes errno.

The RLIMIT_NOFILE definition is one greater than the maximum descriptor number the process may open. Attempts beyond the limit yield EMFILE. It is not simply a memory-size budget, and a sparse descriptor table can make “number of live descriptors” differ from the descriptor-number boundary relevant to an operation.

Error class	Initial diagnostic direction
EMFILE	Process descriptor limit, descriptor allocation paths and retained references.
ENFILE	System-wide open-file resource pressure; do not assume one process’s limit is the cause.
ENOMEM or ENOBUFS	Memory or socket-related allocation pressure; changing RLIMIT_NOFILE is not a direct repair.
EAGAIN from nonblocking accept	No connection available at that instant; distinguish ordinary readiness races from exhaustion.

The accept error contract explicitly separates these cases. A readable listener does not guarantee a subsequent accept succeeds, especially with competing consumers. Keep the listener nonblocking so a diagnostic or recovery operation cannot unexpectedly wait forever.

Build a descriptor ledger before the failure

Track live references by role: listeners, active clients, files, pipes, event loops, timers, received handles and deliberately reserved diagnostics. Include descriptors retained by error paths and short-lived helpers. Measure after a complete repeated connect/disconnect or request lifecycle, not only during peak activity.

Distinguish a bounded steady plateau from a count that grows by one on every failed request. Correlate creation and final close using operation identifiers. A descriptor number can be reused, so a log keyed only by that number can accidentally pair the close of a newer object with an older leaked one.

Inherited references can keep underlying objects alive, but a child retaining a descriptor does not necessarily consume another slot in the parent’s own table. Follow both process-local descriptor usage and shared open-file-description lifetime. Otherwise a real resource leak can be diagnosed against the wrong limit.

Keep the error path usable without new opens

Arrange bounded logging or a preopened diagnostic channel before admitting clients. At exhaustion, opening a new log file, creating a helper pipe, launching a diagnostic subprocess or enumerating a directory may itself need a descriptor and fail. A recovery path that assumes those operations always work is incomplete.

Reserve memory for the small error record too. Avoid allocating an unbounded stack trace or formatting a large payload in the critical path. Record counts, roles and failure categories rather than sensitive descriptor contents. If the existing diagnostic channel is blocked, use the service’s bounded drop/coalesce policy instead of blocking the dispatcher indefinitely.

Apply backpressure to new application work and disable or defer the failing admission path according to the event-loop design. Repeatedly retrying accept on a still-readable listener can produce a tight failure loop. Define a bounded retry trigger tied to released capacity or a timer, and keep existing connections serviced where safe.

Treat an emergency descriptor as a narrow tool

A common design reserves one harmless descriptor. On EMFILE, its owner closes that reserve, uses the freed slot for a bounded nonblocking accept-and-close of one pending connection, then reopens the reserve. This can help shed one request rather than repeatedly waking on an unserviceable listener.

This is an application technique, not an accept guarantee. It needs serialized control of descriptor allocation; another thread can consume the freed slot first. The accept can still fail, the reserve reopen can fail, and one discarded connection does not cure sustained overload. Define every branch before using the technique.

Do not promote this into a universal ENFILE recovery algorithm or assume one freed descriptor is enough for a diagnostic subprocess. Keep the reserve’s purpose precise. If capacity cannot be recovered, enter the documented degraded or restart policy without closing arbitrary client descriptors selected only by numeric range.

Include descriptors in flight and received handles

Since Linux 4.5, RLIMIT_NOFILE also limits descriptors an unprivileged process has in flight over Unix sockets. The Unix-socket reference describes ETOOMANYREFS when that sending-side limit is exceeded. This is a different observation from EMFILE on a newly opened local descriptor.

Receiving SCM_RIGHTS also needs capacity and rejection cleanup. Count successfully admitted handles and close all delivered references when their request is rejected. The kernel’s automatic disposal of excess descriptors does not discharge cleanup for references actually returned to the application.

Include fork/exec inheritance policy in the audit, but do not conflate close-on-exec with normal close. A protected descriptor stays live in the current process until its owner releases it. A service that never executes a replacement program cannot rely on the flag to recover its own leaked slots.

Qualify failure and recovery as separate outcomes

In an isolated test process, propose a deliberately small soft limit established before the workload, retain a preopened result channel, and open only owned harmless resources up to a fixed cap. Require the intended error, no unbounded retry loop and complete cleanup. Do not lower limits in a shared production supervisor or try to exhaust the entire host.

Then release a known set of owned descriptors and verify that admission resumes under policy. Test reserve-reopen failure, multiple allocating threads, rejected received handles and repeated connect/disconnect cycles. Record whether the final count returns to baseline and whether already accepted requests retain correct outcomes.

The release evidence should pair a descriptor ownership ledger with an exhaustion-state machine. A limit increase may be justified by a measured bounded workload, but it should follow that evidence. The useful repair keeps the service diagnosable, prevents overload from becoming a spin loop and accounts for every retained reference.