When Should an RTOS Feed the Watchdog? Task Health Checks and Fault-Injection Tests
Feed a watchdog only when the evidence required by the product’s health policy is still valid. A task reaching its loop body, a timer interrupt firing, or a network stack answering a ping may leave the critical operation stalled. The design must say what progress means, who may authorize feeding, and what safe recovery looks like. The architecture below is a proposal for validation, with Zephyr 3.7.0 used as a versioned reference rather than a claim of tested firmware.
Define the Failures the Watchdog Must Detect
List the failures the mechanism is intended to detect: a critical worker stops completing work, a peripheral operation exceeds its bound, the health supervisor stalls, or scheduling stops. Also list what it cannot establish. Software can remain active while computing the wrong result; a watchdog based on progress alone is not a correctness proof. Additional plausibility checks or independent protection may be needed.
For each target failure, describe the consequence of late detection, an unnecessary reset, and a reset that fails to restore service. Define output behavior during reset and early boot. Rebooting is not automatically a safe state for a heater, valve, or motor controller. The acceptable detection and recovery budget must come from the product’s operating constraints, not a timeout copied from a demonstration application.
Define Health Evidence for Each Task Phase
Make health checkpoints correspond to completed phases: a sample passes validation, a processing cycle finishes, or an output update is committed. A task reports a monotonically advancing sequence and the phase it has completed. The monitor compares that evidence with the phase-specific deadline. Repeatedly publishing the same sequence or reporting entry into a permanently blocked operation should not refresh the proof of progress.
| Task phase | Health evidence | Conditional acceptance rule |
|---|---|---|
| Periodic acquisition | A new valid sample and sequence | Next required sample completes before its acquisition deadline |
| Event-driven idle | Explicit idle state plus bounded wake path | No work is pending and the idle policy remains valid |
| Peripheral transaction | Completion or bounded recovery | Neither success nor recovery may exceed its approved budget |
| Firmware update or startup | Stage progress and finite stage deadline | Special budget is entered deliberately and cannot extend forever |
A quiet event-driven task can be healthy without producing new samples. Define that legitimate state explicitly instead of granting an unlimited exemption. Sleep, flash operations, startup, and update stages need their own reviewed budgets and exit conditions. Verify counter wrap, clock behavior in low power, and consistency when progress fields are read during an update.
Control Health Aggregation and Watchdog Feeding
Choose and document the feed authority. In an application-managed design, workers publish progress and one supervisor evaluates all currently required tasks before feeding the hardware watchdog. A stale critical report prevents authorization even when other workers remain active. The monitor must not accept a producer’s heartbeat as evidence for an unrelated consumer.
Review every path that can feed, disable, reconfigure, or restart the watchdog. An unconditional timer or interrupt feed may keep a device alive indefinitely while its main control task is dead. A manual service mode also needs a bounded entry and an explicit exit. Protect snapshot consistency without giving the supervisor an indefinitely blocking dependency on the same mutexes it monitors. If the supervisor itself stops, the hardware fallback should still reach its declared recovery action.
Separate Task Monitoring from Hardware Fallback
Software task monitoring can identify which supervised channel stopped progressing while the kernel still runs. Hardware timeout provides another layer when software monitoring cannot execute. Their coverage depends on independent clocks, reset routing, low-power behavior, and whether the hardware watchdog is actually enabled. A fallback using the same failed resource may provide less independence than the block diagram suggests.
Zephyr 3.7.0 task watchdog uses kernel timers, supports multiple channels, and can use a hardware watchdog as an optional fallback. This is a concrete library architecture, not proof that every board supports every timeout or callback. Check initialization results and the target driver’s capabilities against the versioned watchdog API. The application-managed supervisor in the diagram is conceptual and is not a diagram of Zephyr’s internal implementation.
Choose Isolation and Reset Responses
Define responses by fault class. A noncritical sensor fault may permit a bounded driver reset and degraded operation, if the product explicitly allows it. Loss of a required control function may demand an immediate safe-output action followed by reset. Continuing to feed while a critical task remains unhealthy requires an explicitly approved degraded mode with its own limits; it should not happen merely because a recovery loop is running.
Attempt a minimal diagnostic record only when that path is safe and time-bounded. Do not block recovery waiting for a full log flush. Some failures will prevent any last-minute record, so retain useful progress before the fault. Define a boot-attempt policy as well: repeated resets should not endlessly re-enable hazardous outputs or replay an uncertain command. A recovery or restricted mode needs entry criteria and an authorized route back to normal service.
Map Each Injected Fault to an Expected Response
Use controlled test hooks on an isolated target with safe loads. For each injected failure, specify the trigger, expected detection, permitted degraded interval, expected isolation or reset, and required evidence. One particularly useful case leaves a low-priority health or logging task running while a critical worker stops. It exposes feed policies that confuse general activity with completion of required work.
| Injected condition | Expected result to define | Failure indicated by |
|---|---|---|
| Critical sequence stops advancing | Feed denied after its permitted deadline | Unrelated activity keeps hardware fed |
| Peripheral never completes | Bounded recovery, then escalation if required | Unbounded waiting or endless recovery feeds |
| Monitor stops executing | Hardware fallback action occurs | System stays hung without the required reset |
| Legitimate longest operation | Progress remains valid under its special budget | Spurious reset within the approved operating envelope |
| Repeated boot failure | Restricted recovery policy is entered | Unbounded restart loop with unsafe outputs |
Lock waits do not automatically mean priority inversion; require scheduler and ownership evidence for that diagnosis. Scheduler-stop or interrupt-masking experiments must follow platform-specific limits. Remove or disable test hooks in the release build and record that check.
Test Detection, False Resets and Post-Reset Recovery
Run both negative and positive cases. Fault injections establish whether missing progress is detected; peak legitimate workloads establish whether valid operation is incorrectly reset. Test the longest permitted startup, update, sleep, and peripheral-recovery paths. Account for clock tolerance, scheduling delay, and hardware watchdog windows where applicable. Report the measured interval from fault to detection, safe-state action, reset, and first correct recovered operation separately.
Preserve hardware and build versions, configuration, test duration, repetitions, and uncovered conditions. Passing selected tests does not establish universal fault coverage or a safety certification. Use reset evidence preservation to make failures diagnosable and put these criteria into firmware acceptance requirements. For a real review, provide Obeita the task deadlines, watchdog hardware capabilities, safe-state requirements, and permitted fault-injection setup.
