Suspend Failure Halfway Through: Restore Every Device Already Quiesced

A failed suspend attempt needs two kinds of recovery: the failing callback must undo its own partial changes, and the power-management core must unwind phases that other devices completed successfully. Do not assume that returning an error automatically schedules the matching resume callback for the failed phase.

Applicability: this article pins Linux v6.12 system suspend with ordinary device PM callbacks. Its example excludes syscore devices, direct completion, and smart-suspend skip paths unless explicitly discussed. Hibernation and runtime-PM error handling are separate contracts. The fault sequence is hypothetical; no kernel modification or target suspend test was executed.

1. Record completed transitions, not callback entry alone

Give each callback a precondition, a success postcondition, and an error postcondition. A callback can stop a queue, mask an interrupt, and then fail to save a register. Its entry marker proves none of those operations completed, while its error return does not identify which changes remain active.

Build a local ledger of irreversible or externally visible steps. Record whether work admission stopped, in-flight operations drained, device interrupts changed, wake configuration changed, and resources were released. Derive undo actions from completed steps, not from the intended final state.

Keep these local transitions distinct from the PM core’s phase tracking. The core knows a callback’s return value; it cannot infer the private hardware changes made before that return. A single “suspended” flag in the driver is inadequate if it cannot represent a partially completed sequence.

2. Check the exact core success boundary

In Linux v6.12 drivers/base/power/main.c, the ordinary suspend path sets is_suspended only when its selected callback returns without error. The ordinary resume path checks that flag before invoking the corresponding callback.

The late and noirq paths use analogous is_late_suspended and is_noirq_suspended flags. Their failure branches leave the respective success flag unset, and the matching early or noirq resume path skips that phase when its flag is unset. These are implementation observations for the pinned release, not a promise that every subsystem invokes every driver callback directly.

The practical consequence is local rollback: if suspend_late partially changes hardware and returns an error, do not rely on resume_early to undo those partial changes. Earlier successfully completed phases can still be unwound. Inspect the effective PM-domain, type, class, bus, and driver layers to establish which callback owns each change.

3. Work through a late-phase failure

Consider hypothetical devices A and B with ordinary suspend already successful for both. A completes its late callback. B then makes one reversible late-phase change, but its next operation fails. B must undo that first late change before returning the failure, restoring the state expected after its successful ordinary suspend.

The core can then run A’s early resume to reverse A’s successful late phase. B’s early callback is skipped for the failed late phase. Ordinary resume can subsequently run for both because their ordinary suspend phases succeeded. Preparation cleanup follows the applicable completion path.

This does not mean B should reopen normal I/O inside its failing late callback. Its rollback target is the preceding valid phase, not necessarily fully running service. Reopening too much early can violate dependencies or race the still-suspended devices around it.

If B’s ordinary suspend itself fails, the target is different: undo the partial ordinary-suspend changes locally. If B’s noirq callback fails, restore its valid post-late state locally, allowing previously successful late and ordinary phases to unwind later. Write these targets down before adding error labels.

4. Separate framework unwind from guaranteed recovery

The device-PM guide describes abandoning the requested sleep state and resuming devices when a suspend callback fails. The system suspend implementation connects the failure path back to device resume and completion.

That is a control-flow guarantee, not proof of successful hardware restoration. A resume callback can itself fail. Preserve both the original suspend failure and subsequent recovery errors rather than allowing the final log line to hide the first cause.

The dev_pm_ops contract documents completion cleanup and explains that specified resume-method errors do not abort the resume transition. A driver must not advertise usable hardware merely because the wider system continued recovering. Define a safe unavailable state and report failed operations honestly when restoration is impossible.

5. Make cleanup conditional and repeatable

Pair each acquired reference, enabled wake facility, stopped queue, and altered device setting with one owner. Cleanup should act only on changes that actually occurred. Avoid a broad “restore everything” helper that disables an unacquired resource or releases the same reference twice.

A rollback helper and a normal resume helper may share lower-level operations, but their incoming states differ. Either pass a precise completed-step state or provide separate entry points with explicit assumptions. Do not call the public PM callback recursively to bypass framework state tracking.

Review concurrency during recovery. A timeout does not establish that a worker or transfer engine stopped using its resources. Before releasing them, establish the consumer’s completion or cancellation contract. Ensure an asynchronous completion cannot race a second cleanup path.

6. Inject failures at meaningful boundaries

On an authorized development fixture, propose a reviewed fault hook after each reversible step. First test the failing device alone where practical, then include a peer that successfully reaches the same phase. Keep loads independently safe; do not simulate electrical faults by violating component limits.

Capture device identity, phase, callback entry and exit, local completed-step state, injected failure, local undo result, and framework resume callbacks. With asynchronous PM, independent devices may already be progressing when failure becomes visible. Evaluate the actual completed set rather than expecting one fixed total ordering.

Include failures in ordinary, late, and noirq callbacks separately. Respect each phase’s execution context when designing the hook and cleanup. A recovery operation that requires ordinary interrupt completion cannot simply be moved into noirq because it works during normal operation.

7. Require useful service after an aborted attempt

After the system returns, perform a representative transaction on both the failed device and previously quiesced peers. Check IRQ delivery, queue admission, resource accounting, and the next supported runtime-PM cycle. Then remove the injected fault and attempt another complete system transition.

The acceptance record should show each phase reached, each local rollback target, the core’s actual unwind, and subsequent service results. An expected suspend error is a passing fault-injection observation only when recovery also satisfies the contract. A responsive shell alone does not demonstrate that every affected device was restored.