Linux Workqueue Stalls: Find Dependencies That Cannot Run Under max_active Limits

A work item can be blocked by a dependency that is queued correctly but cannot become active under the workqueue’s admission limit. Adding worker threads does not necessarily solve that condition. First identify the queue, effective concurrency domain and release dependency, then show whether the required producer can obtain an active slot.

This article uses upstream Linux v6.12 workqueue code. The main example is an ordinary bound workqueue, with two distinct work items targeting the same online CPU and max_active set to one. It excludes bottom-half workqueues and ordered-queue assumptions. A small userspace state model was executed; no kernel worker was blocked and no test module was loaded.

Distinguish a worker from an active work item

The workqueue guide explains shared worker pools and concurrency management. A workqueue is a domain for attributes and progress guarantees, not simply a private set of threads. A sleeping worker can allow the pool to dispatch other work, but that does not automatically release the sleeping work item’s active accounting.

In the v6.12 implementation, the bound-queue admission path compares active work against the queue’s max_active value. Work awaiting an active slot remains inactive until admission is possible. The distinction matters when an active callback waits for another callback from the same constrained domain.

Record the allocation flags, max_active, actual queue pointer, target CPU and work-item identities. Avoid grouping everything named kworker together. Two similarly named callbacks on different queues can have different limits, while different callbacks on one queue can share the same admission bottleneck.

Draw the saturated dependency

In the synthetic scenario, work A becomes active and queues work B to the same bound queue and CPU. A then waits for completion C. B is the only producer of C, but it cannot become active because A occupies the single permitted slot. A cannot finish until B runs; B cannot run until A finishes.

This is an admission dependency cycle, even if A sleeps and the worker pool has spare threads. There need not be a mutex cycle. Raising worker priority does not create another permitted slot, and moving unrelated activity away from the CPU does not resolve the logical dependency.

Use a bounded model before a kernel reproducer

The supplied pure userspace fixture represents active slots, an inactive B and the completion dependency. With capacity one it confirms that B cannot be admitted while A waits. With capacity two it admits B, supplies the completion and retires both items. These are model assertions, not observed Linux scheduling results.

capacity = 1
active = {"A"}
B_can_enter = len(active) < capacity
assert not B_can_enter
# A still owns its slot while waiting for B's completion.
# No modeled transition can release A without admitting B.

The model intentionally omits scheduling, hotplug, cancellation, memory pressure and requeue behavior. It helps expose the graph error; it cannot validate a driver fix or kernel concurrency guarantee. The executed fixture and its pass result are retained with this article package.

A real laboratory reproducer should use a private bound queue, distinct A/B objects and an explicitly fixed online CPU. A queues B and uses a bounded completion wait so the experiment can unwind. An independent controller outside that queue observes the timeout and guarantees cleanup. Keep CPU hotplug out of this controlled test, and preserve the queuing return values.

After A’s timeout releases its slot, B may run late. The fixture must keep its completion and request storage alive until both callbacks are safely retired. Otherwise, an attempted stall test can introduce a use-after-free that obscures the original problem. This kernel reproducer remains proposed and unexecuted.

Do not generalize bound limits to unbound queues

There is a documentation boundary in this release. The v6.12 prose guide still describes max_active as per-CPU even for unbound queues. However, the tagged alloc_workqueue API comment and implementation describe unbound admission using per-NUMA-node accounting and distribution of the configured limit.

The implementation distributes the limit according to eligible online CPU proportions and clamps each node’s allowance between min_active and max_active. The default minimum is the smaller of max_active and WQ_DFL_MIN_ACTIVE, which is eight in this tag. Thus the sum of node allowances can exceed the configured value, and a dependency concentrated on one node may have less concurrency than a naive global interpretation expects.

Inspect the deployed source, affinity configuration and effective node domain before diagnosing an unbound case. Do not reuse the bound two-item example as a numerical proof for every unbound topology. Likewise, use alloc_ordered_workqueue when strict ordering is required; WQ_UNBOUND combined with max_active one is not a substitute for its ordering contract.

Collect evidence at the admission boundary

Evidence	Question it answers
Queue allocation and effective attributes	Which active limit and domain apply?
A’s wait stack and owned resources	What prevents A from returning?
B’s queueing and execution evidence	Was B submitted, admitted and actually started?
Completion identity and generation	Is B the producer required by this A?
Other active work and dependencies	Can all slots be occupied by similar waiters?

An absent execution trace alone does not prove admission starvation; event loss, a failed queueing attempt or cancellation can explain it. Correlate queue state, stacks and source. Use a bounded capture with preserved loss statistics instead of treating a sparse timeline as complete.

Repair the dependency, then test saturation

Prefer a design in which a callback returns and a later completion continues the state machine, or place a genuinely independent producer in a progress domain that cannot be exhausted by its consumers. Check ordering and object lifetime when changing that design.

Increasing max_active can break this particular two-item cycle, but a workload with more simultaneous waiters can saturate the larger limit again. Establish a bound on interdependent work, including nested submissions, or remove the synchronous dependency. WQ_MEM_RECLAIM supplies an execution-context guarantee under memory pressure; it does not make every dependency cycle safe.

Validate normal traffic, maximum approved concurrency, aborts and teardown. A passing lightly loaded run is insufficient. Deliver the queue-specific graph, actual source semantics, bounded reproduction plan and a fix that preserves progress when all permitted active slots are occupied.