Cgroup PID Limits: Diagnose Thread Creation Failure When RAM Is Still Available

When process or thread creation fails, inspect task-count limits and their hierarchy before blaming free-memory capacity. Linux’s cgroup PID controller counts kernel tasks, including threads. A service with only a few process IDs can still consume its allowance through worker pools and helper activity.

This guide uses upstream Linux v6.12 cgroup v2 behavior and Linux man-pages 6.12. It proposes a bounded creation/failure test; no cgroup limit was changed and no process or thread exhaustion was induced. The example counts are hypothetical.

Preserve the API’s actual failure result

Record the failing operation, caller identity, timestamp and returned error. The fork reference in the official man-pages 6.12 archive lists EAGAIN for several task-limit conditions, including pids.max. A single EAGAIN is not a unique diagnosis of the cgroup controller.

The pthread_create reference also lists resource and limit failures, but its calling convention differs: it returns an error number directly rather than using a minus-one/errno result. Preserve that return value instead of reporting a stale errno from an earlier call.

Separate task-count enforcement from memory allocation and scheduling-policy failures. Thread creation needs resources beyond an identifier, and a library can fail before the expected kernel path. Keep the complete error and relevant trace rather than translating every failure to “out of RAM.”

Count threads and descendants

The v6.12 cgroup v2 PID controller documentation explicitly says its PIDs refer to TIDs, the kernel’s task identities. Read pids.current, pids.peak and pids.max in the actual service group where available. pids.current includes descendants; it is not simply the number of lines in a process-only service list.

Inventory the main thread, fixed worker pools, per-connection workers, runtime-created threads and transient helper processes. Include restart overlap: an old generation may still be retiring while a new service instance begins. The permitted peak can be larger than the steady-state count.

Record the accounting group’s identity and lifetime with every snapshot. If a supervisor recreates the group, a lower counter or peak can belong to a new epoch. Do not infer that a leak disappeared solely because the old counters were discarded.

Walk every enforcing ancestor

Task admission is hierarchical. A leaf with max set to max or with apparent spare capacity can still fail when an ancestor is full. Inspect the ancestors and their descendant populations, including sibling services outside the leaf’s own view.

For a hypothetical parent limited to one hundred tasks, suppose the service subtree uses thirty and sibling subtrees use seventy. A new service thread can be rejected even though the service’s own limit is fifty. Raising only that leaf limit would not create room in the parent.

Compute headroom as a diagnostic snapshot, not a reservation. Other tasks can appear or exit between collection and the next creation attempt. A release design needs bounded concurrency and a failure path rather than relying on a previously observed spare slot.

Interpret events at the correct level

Collect before-and-after pids.events and pids.events.local values around the failing operation, and record whether the hierarchy uses pids_localevents. The v6.12 documentation describes the default hierarchical max event and the option that restores local fork-failure accounting.

The pinned implementation distinguishes the forking group from the group whose limit was exceeded. Under the default v2 event behavior, the local max event is charged at the limiting group and propagated upward. Therefore a leaf’s unchanged max counter does not exclude a rejection caused by an ancestor.

Use the events together with the error and hierarchy snapshot. An event increment identifies enforcement, not which application request should be retried or which task should be killed. The code prints a limited diagnostic rather than one log message for every rejected creation, so log silence is not a complete event count.

Check other limits before selecting a fix

Candidate boundary	Evidence to inspect
Cgroup task limit	Leaf and ancestor pids limits, counts and event changes
Per-real-user limit	RLIMIT_NPROC and tasks belonging to that real user
System-wide task or identifier limit	threads-max, pid_max and corresponding system state
Thread-resource failure	Library result, stack/resource configuration and allocation evidence
Scheduling setup failure	Requested policy, attributes and the actual error

The resource-limit reference distinguishes RLIMIT_NPROC from the cgroup controller and documents its real-user scope and privilege exceptions. Do not assume its value is a per-service limit or that raising pids.max changes it.

The PID controller permits organizational operations that leave pids.current greater than pids.max, such as moving tasks or lowering the limit. It then blocks creation that would violate policy. Such a snapshot does not prove that the controller malfunctioned, nor does lowering a limit automatically terminate existing tasks.

Design a bounded failure-and-recovery fixture

1. Use an approved disposable cgroup subtree with an external controller and a known cleanup route.
2. Set a deliberately small, reviewed test allowance without altering production ancestors.
3. Create a bounded set of waiting workers and record every successful identity.
4. Attempt one additional creation and preserve the API result plus leaf/ancestor event changes.
5. Release and join or reap the workers through their normal lifecycle.
6. Verify that a later permitted creation succeeds and that the application returns to service.

The controller and essential diagnostics must remain able to run when the test group is full. Avoid a fixture that needs to spawn another process inside that group to stop the test. This procedure remains proposed; no task limit was approached on the authoring host.

Keep recovery bounded

Do not retry creation in a tight loop. Define whether the service rejects work, queues a bounded amount, reuses a worker or enters a maintenance state. Keep diagnostics usable through already established resources, and avoid starting a new helper for every failure.

Investigate unexplained task growth before increasing limits. If the workload legitimately needs more concurrency, review the parent budget and the impact on sibling services as well as the leaf. A larger allowance can postpone a leak or overload without fixing it.

Deliver the task inventory, hierarchy ledger, exact error, event correlation and recovery result. The conclusion should name the enforced boundary and its owner. Available RAM is useful context, but it cannot establish that the service is permitted or able to create another task.