Watchdog Resets in RTOS Devices: Preserve Evidence and Find the Stalled Task
A watchdog reset tells you that a reset mechanism fired; it does not identify the stalled task or explain why progress stopped. Useful diagnosis starts before the next failure, with evidence that survives the relevant reset and a boot path that preserves it. This guide uses Zephyr 3.7.0 documentation for concrete software boundaries, while retention domains and reset registers must be checked against the actual MCU and board.
Verify the Reset Source
Capture raw reset status as early as the boot chain permits, before application initialization clears flags or overwrites evidence. Record the MCU revision, bootloader version, firmware build, and the component responsible for reading and clearing each register. A bootloader may inspect the status before the application starts; agree on a handoff record rather than assuming the application sees untouched hardware state.
Multiple flags can be present, and some devices accumulate causes until software clears them. Zephyr 3.7.0’s hardware-information API documents supported-cause discovery and this accumulation behavior. Preserve the raw value and decoded interpretation separately, and record any API failure. Compare watchdog, power-on, brownout, software reset, and reset-pin evidence. Do not select the most convenient flag and discard the rest; reset-source evidence and the earlier business failure are different layers.
Choose Evidence That Can Survive the Failure
Choose the smallest useful record for each failure boundary. A non-initialized RAM section only prevents ordinary startup clearing; it does not create a retention power domain. Check whether the selected RAM survives the exact watchdog reset, whether the bootloader touches it, and what happens during power loss or low-power transitions. Backup registers and nonvolatile storage have their own capacity, clock, access, and endurance constraints.
| Evidence location | Condition to verify | Conclusion allowed after a valid record |
|---|---|---|
| Reserved retained RAM | Target reset preserves the domain and boot path | State was recorded before this restart; power-loss survival is not implied |
| Backup registers or battery domain | Supply and reset behavior verified on this board | Only the tested reset/power conditions are covered |
| Nonvolatile record | Commit completed; corruption and endurance handled | A durable earlier checkpoint exists, not necessarily the final instruction |
| Ordinary log buffer | Buffer recovered before initialization overwrites it | Available events are usable; missing events do not prove inactivity |
Use a format version, build ID, generation, length, and integrity check. Design for interrupted writes: validate a committed slot and reject torn or incompatible records. A checksum detects some corruption but does not prove the causal story. Zephyr’s retention system offers header/checksum support over suitable retained memory; its documentation explicitly separates that use from power-failure persistence.
Record Task Progress Instead of Only the Last Feed
Record completed work at meaningful checkpoints: sample accepted, computation completed, output committed, or a bounded recovery finished. Keep a monotonically advancing sequence, phase, timestamp, and current wait object for each critical task. Include enough context to distinguish a legitimate wait from overdue progress. A loop counter that advances before the operation succeeds can make a stalled task look healthy.
The last task that fed the hardware watchdog may be a supervisor summarizing many workers. Its final message is not evidence that it caused the stall. Preserve the last accepted progress for each worker and the supervisor’s reason for refusing the next feed. Where records can be updated concurrently, use a coherent snapshot protocol and account for counter wrap. Avoid high-rate flash writes simply to obtain a breadcrumb; choose a bounded logging design compatible with storage lifetime.
Separate Common Stall Mechanisms
Construct hypotheses from multiple observations. A task blocked on a mutex calls for its owner and scheduler trace. A task waiting on a peripheral calls for driver state, interrupt evidence, and the operation’s deadline. A full queue calls for producer and consumer progress. A ready task with no execution suggests checking CPU availability, priorities, interrupt load, and scheduling state.
Priority inversion needs an ownership-and-scheduling chain; it is not a synonym for every lock wait. Use the mutex timeline method when that chain is plausible. Likewise, silence in the log does not prove the scheduler stopped: the logging path itself may have blocked or overflowed. Label each statement as observed, inferred, or not yet measured. That discipline prevents an attractive reset explanation from outrunning the evidence.
Know When a Core Dump Is Available
A hardware watchdog can reset the MCU without executing a software fault handler. A pretimeout interrupt or callback is useful only if the hardware supports it, the CPU can service it, and enough safe time remains. A damaged stack, masked interrupts, a hung storage driver, or loss of power may prevent the emergency capture. Do not make the recovery mechanism depend on finishing a large dump.
In Zephyr 3.7.0, configured core-dump support captures registers and selected memory through enabled backends, normally on a fatal-error path; it can also be invoked on demand. This does not mean every watchdog reset automatically traverses that path. Match dump configuration and architecture support to the target, validate the complete output, and preserve the exact ELF and symbols. A dump decoded against a different build can produce misleading addresses.
Turn Field Evidence into a Controlled Reproduction
Turn each hypothesis into a controlled bench experiment. Block one critical worker, delay one peripheral completion, fill a queue, or introduce a bounded competing workload, changing one mechanism at a time. Keep outputs connected to a safe load or simulator and establish a recovery route independent of the injected fault. Do not inject uncontrolled hangs into production machinery.
Compare signatures, not merely the fact that both runs reset. Did the same task stop at the same phase? Was the same wait object involved? Did the reset flags and retained-record generations agree? Also test evidence handling itself: reset during a record update, present an invalid checksum, and boot after a firmware change. The reader should reject unusable data clearly instead of silently interpreting it as current. Record mismatches between field and bench behavior because they constrain the hypothesis.
Hand Off a Reproducible Diagnostic Package
Package the original reset flags, retained bytes, decoded records, relevant log window, exact build artifacts, RTOS configuration, board revision, and reproduction steps. Include the meaning and units of every timestamp and counter. Preserve originals before producing filtered traces. Remove credentials and unrelated user payloads from shared diagnostics, while keeping the technical evidence needed for analysis.
The handoff should state what is established, what has been excluded, what remains uncertain, and the next observation that could distinguish candidates. Use the existing crash logging and reproduction guide for the broader workflow, then verify the repair with watchdog fault-injection acceptance. For a diagnostic review, send Obeita the MCU/RTOS versions, raw reset status, last task-progress record, and matching build symbols.
