Zephyr Workqueues: Cancel, Reschedule and Drain Without Racing the Handler

Cancellation is a lifetime question

A driver cancels delayed work, frees its context, and then crashes when the handler accesses that context. The work item may have moved from delayed to queued or running while shutdown was making its decision. The correct question is not merely “did cancel return?” but “what prevents every producer and handler from referencing this object after reclamation?” This guide uses Zephyr 4.2.0; older examples, especially those predating the workqueue changes in 2.6, need careful translation.

Workqueue status is not a single mutually exclusive enum. An item can be running and queued again, with additional delayable activity. The 4.2.0 lifecycle documentation warns against using busy or pending snapshots to decide whether an operation is safe. The state can change immediately after the query. Use the operation’s documented synchronization contract together with an application-level shutdown protocol.

Distinguish asynchronous and synchronous cancellation

k_work_cancel() attempts cancellation without waiting for a running handler to finish. For delayable work, use the corresponding delayable API rather than treating its embedded ordinary work member as the entire object. When reclamation requires handler completion, a thread can use k_work_cancel_sync() or k_work_cancel_delayable_sync() with a struct k_work_sync object. These calls are synchronization boundaries, not a way to terminate arbitrary handler instructions.

Review the version-pinned API contracts for context restrictions, return values and synchronization-object storage. Synchronous waiting is not appropriate in an ISR or when invoked in a way that makes a workqueue wait for itself. Do not hold a mutex needed by the handler while waiting for that handler to finish. Architectures with kernel-coherence requirements also constrain where the synchronization object can live; copying a stack-based example without checking that configuration is unsafe.

Choose schedule or reschedule from the deadline policy

k_work_schedule() preserves an existing pending delay, while k_work_reschedule() replaces the delay deadline. That distinction maps naturally to different application requirements. “Process one batch a fixed interval after its first event” and “process after the input has been quiet for an interval” are different policies. A stream of new input can keep postponing a quiet-period policy indefinitely, so consider a separate maximum-age bound if the product requires one.

Do not interpret either API as a promise that the handler runs at the exact requested instant. Delay expiry makes work eligible for queue submission; workqueue load and thread scheduling still affect execution. Nor does rescheduling undo side effects of a handler that has already run. State whether the application wants coalescing, one operation per event, or the latest state only. If every event must be preserved, store those events separately rather than assuming repeated submission creates a history of work items.

Close the producer gate before waiting

A robust shutdown sequence begins by moving the owning object into a CLOSING state under the synchronization used by every producer. Prevent new timer, ISR, and thread submissions; synchronize producers already inside their submission path. Then cancel the relevant delayable and ordinary work synchronously from a suitable controller thread. Only after those boundaries may shutdown reclaim the containing object and its payload buffers.

A lone atomic “closing” flag is not automatically sufficient. A producer could read false, pause, and submit after cancellation has finished. Solve the check-and-submit race with an appropriate admission protocol, producer reference tracking, or driver-specific interrupt synchronization. The handler must also stop self-resubmission once closing begins. Do not keep an admission lock held across synchronous cancellation if the handler needs that same lock to exit. The important invariant is that no new reference can appear after the final completion boundary.

Drain a queue only when you own the queue policy

k_work_queue_drain() waits for the queue to empty, and its plug option can keep external submissions blocked afterward. Work resubmitted by the queue’s own thread has special treatment while draining. More importantly, delayed work whose timeout has not expired is not simply erased by draining. The documentation states that draining does not itself cancel delayable scheduling; a delay expiring while the queue is plugged can fail to submit.

Therefore queue drain is not a substitute for cancelling each owned delayable item. It is also a poor default shutdown operation on the shared system workqueue, because unrelated services depend on it. Use a private queue when its blocking behavior or shutdown ownership genuinely requires one, accounting for its thread and stack cost. Record what happens to rejected submissions and who is responsible for retrying or abandoning them after unplugging.

Test transitions around the handler boundary

Build a proposed test handler with controlled gates before it reads its context, while it holds an application resource, and immediately before it attempts resubmission. Start shutdown at each gate. Separately trigger a producer that passed its admission check just before shutdown closed the gate. Assert that shutdown does not free the object until all permitted access has ended, and that no producer can revive it afterward.

Test delayed-only, queued, running, running-plus-resubmitted, and already-idle cases. Exercise a plugged queue with a delay expiring during the plug, then verify the documented recovery policy. Capture API return values, object generation, gate state, handler start/end and final reclamation. A successful no-load run is not evidence that these races are covered. This article performs source review only and reports no executed Zephyr or hardware tests.

Acceptance checklist

• The implementation is tied to Zephyr 4.2.0 or rechecked against the selected release.
• Shutdown closes and synchronizes every submission source before reclamation.
• Synchronous cancellation runs in a context that cannot deadlock the handler.
• Delayable work is managed through delayable APIs and has an explicit deadline policy.
• Drain, plug and rejected submissions have defined ownership and recovery behavior.
• The final free occurs after completion and after future references have been prevented.