Hung Linux Tasks: Read D-State Stacks Before Treating the Message as a CPU Lockup
A hung-task warning calls for a wait-dependency investigation: what must happen before the blocked task can run, and who can make it happen? It is not automatically a CPU lockup. Other tasks and interrupts may continue while one thread waits indefinitely for a resource, completion or lower-layer operation.
This guide uses upstream Linux v6.12 hung-task detection and its configuration-specific scope. It does not interpret RCU stall fields or IRQ-disabled intervals. No task was deliberately hung, no timeout was changed and no target stack was collected for this article.
Preserve the first warning and its policy
Save the complete warning, task name and identifier, stack, taint/build context and nearby device errors. Record the operation active when the task blocked. Task names can be reused, so keep the process/thread identity and lifetime rather than grouping every identically named worker together.
The v6.12 sysctl reference describes the hung-task timeout, check interval, task-count limit, warning budget and panic policy. A zero timeout disables this checking. A finite warning budget can suppress later printed reports without proving that the task recovered.
The implementation checks relevant uninterruptible states and uses changes in task switch counts as progress evidence. It excludes certain states, including killable and genuinely idle waits, and has additional skip conditions. Therefore “a task appears as D” is a useful observation, not a complete statement that the detector must report it.
Read the stack from wait site toward the request
Locate the primitive where the task is sleeping, then follow its callers back to the operation that created the dependency. A generic scheduling function at the top is usually the mechanism of waiting, not the subsystem responsible for releasing it.
Classify the wait: mutex or semaphore acquisition, completion, device request, filesystem operation, workqueue synchronization or another identified resource. Verify the exact wait API and source path. Similar stack names can hide different semantics, including whether the wait has a timeout and whether a signal can interrupt it.
Use a read-only task-stack interface where the platform permits it, with matching symbols and approved diagnostic access. A missing or restricted stack is an evidence limitation; do not broadly weaken kernel inspection permissions to obtain one. Preserve raw snapshots before annotating them.
Draw the release dependency
For a hypothetical driver request, write the graph as: application thread waits for completion C; driver completion path supplies C; that path requires a worker or interrupt plus a valid request object. Then test each edge. Was the request submitted? Did hardware signal completion? Did the relevant handler run? Can the worker access the state it needs?
If the release owner is blocked too, extend the graph. For example, task A may hold lock L while waiting for worker B, while B waits for L. That is a concrete cycle. A long stack alone does not establish it; identify the actual lock instance and matching request generation.
Do not assume the waiter owns the defect. It may be correctly waiting for an event that another subsystem lost. Conversely, an absent completion can result from the waiter holding a resource needed by the completion path. Follow ownership and progress in both directions.
Compare repeated snapshots without overinterpreting them
- Choose a bounded observation interval appropriate to the expected operation, without first changing the detector timeout.
- Capture the same task’s stack and state repeatedly, along with the identified release owner’s progress.
- Correlate request identities, device status and completion or lock events using a justified time relationship.
- Record whether the wait site remains stable, changes to another wait, or completes normally.
- Stop collecting when evidence loss, workload changes or a reset makes the snapshots incomparable.
An unchanged stack supports a persistent wait hypothesis, but it does not prove continuous waiting between samples. A task can repeatedly enter the same function. A changing stack can likewise reflect motion within an unproductive loop. Include operation counters or request generations when those distinctions matter.
Use a decision worksheet
| Correlated evidence | Next investigation |
|---|---|
| Waiter remains at completion; producer never starts | Submission, queue admission and producer scheduling |
| Producer starts but waits on a lock held by the waiter | Source-backed wait cycle and ownership redesign |
| Hardware request remains outstanding | Device progress, interrupt route and bounded recovery contract |
| Completion occurs for another generation | Request identity, stale callback and lost handoff |
| Task resumes but reports continue elsewhere | Separate task lifetimes and distinct blocked resources |
The worksheet prioritizes evidence rather than assigning a universal root cause. A storage wait can reflect a lower-layer failure, and a filesystem stack can simply be the visible consumer. Do not diagnose a damaged disk from D state alone.
Test the release and recovery boundary
Use a recoverable laboratory setup to repeat the smallest valid operation that exposes the wait. If a failure must be modeled, prefer a reviewed test fixture that owns its completion source and can guarantee recovery. Do not disconnect production storage or a live control device merely to force a warning.
Test the intended timeout or abort path separately. A timeout return is insufficient if DMA, callbacks or references remain active after the caller frees its request. The correct recovery must explain who stops outstanding work, who supplies final completion and when the object can be reclaimed.
Sending a signal does not necessarily release a genuinely uninterruptible kernel wait immediately. Likewise, raising the hung-task timeout only changes observation policy. Neither action explains why the required producer failed to make progress.
Distinguish detection from causation
Hung-task checking is not a guarantee of reporting every stalled thread. Its configured scan limit, skip states, warning policy and the checker’s own scheduling affect coverage. A quiet log can coexist with an untested or unobserved wait condition.
After a repair, repeat normal operation, the original failing sequence and relevant abort cases. Require the dependency graph to terminate in a producer that can run, or in a bounded error path that safely retires the operation. Validate application behavior rather than merely suppressing the warning.
Deliver repeated raw stacks, the resource-owner graph, exact detector settings and the source-backed correction. The final explanation should identify what progress was missing and how it was restored. That is more useful than labeling every blocked task a frozen CPU.
