Embedded Linux Recovery: Separate Service Watchdogs from the Hardware Reset Path

A watchdog setting does not define a complete recovery architecture. A service watchdog asks whether an application is notifying its supervisor. A hardware watchdog asks whether the system continues servicing a hardware timer. Either can remain satisfied while a different part of the product has stopped doing useful work.

This guide uses systemd v257 and distinguishes service recovery from whole-board reset. Actual watchdog availability, supported timeouts and safe output behavior depend on the board. No watchdog was enabled, no process was deliberately stalled and no reset test was executed.

Identify who observes each failure

Start with the product's failure classes. An acquisition process may stop processing samples while the operating system remains responsive. PID1 may continue running while another service is blocked. A broader system failure may prevent PID1 from servicing a hardware timer. Shutdown can stall after ordinary services have already terminated.

These conditions need different observers. Do not infer coverage from a single healthy indication. A board that periodically refreshes its watchdog can still deliver stale application data, while an application restart may restore service without requiring a disruptive hardware reset.

Give service notifications a meaningful contract

The tagged service reference defines WatchdogSec as a timeout activated after service startup completes. The service must send WATCHDOG=1 notifications. Missing the deadline places it in a failed state and terminates it with the configured watchdog signal, normally SIGABRT.

Automatic restart is a separate policy choice. Appropriate Restart settings can restart a watchdog-failed service; enabling the timeout alone does not promise that recovery action. Notification access also matters: the manager must accept the sender under the unit's NotifyAccess policy.

Design the notification around the progress the product actually needs. A dedicated thread that sends pings regardless of a stuck acquisition pipeline can conceal that failure. This is an application design concern, not a claim that systemd knows whether a sensor reading is fresh. Define which work must advance before the application is willing to report liveness.

Understand the hardware observer

The system-manager reference describes RuntimeWatchdogSec separately. PID1 programs and services the available hardware watchdog. When enabled, it contacts the device at least once within half the configured interval; hardware capabilities can constrain the timeout actually selected.

This protects a different boundary from the application's notifications. PID1 continuing to feed the hardware does not establish that every product service is functioning. Conversely, a failed application notification can trigger local service recovery while PID1 and the board remain healthy.

Confirm that a supported hardware watchdog is present and identify its real reset behavior. Do not choose a timeout from an example configuration without accounting for the platform's capabilities and intended operating conditions. These settings have no effect when the required hardware watchdog is unavailable.

Include shutdown in the recovery design

RebootWatchdogSec covers the second shutdown phase, after regular services have terminated and PID1 has been replaced by systemd-shutdown. During the earlier phase, the normal system manager is still running and RuntimeWatchdogSec remains relevant.

Therefore, a reboot timeout cannot be interpreted without locating the stalled phase. Preserve enough independent observation to tell whether the system was stopping services, performing final shutdown or running normally. Do not report a reboot watchdog as protection for every earlier service-stop operation.

The appropriate recovery action also depends on the physical product. Restarting software or resetting a board can interrupt outputs and communications. Define safe behavior and an independent recovery route before testing failure handling on an isolated target.

Test the boundaries separately

Use a controlled fixture to distinguish a service that stops meaningful progress, a service that stops notifying and a failure affecting the hardware-refresh path. These are proposed test categories, not instructions to stall a production system. Keep reset evidence independent of the component being tested.

For each case, state which observer detected the failure, which recovery action occurred and whether the product function actually recovered. A successful restart is not itself proof of restored acquisition or communication. The resulting coverage map is more useful than a single checkbox saying that the image has a watchdog.