systemd Service Shutdown: Wait for Workers Before ExecStop Returns

An acquisition service can acknowledge a stop request while its file-writing worker is still finishing. If the program named by ExecStop= returns at that acknowledgement, systemd may start terminating the remaining processes before the application’s own shutdown protocol has finished.

Define what the stop helper waits for. This review uses the tagged systemd v257 manuals and a fictitious main process with one writer. The timelines are design fixtures, not observed service stops or measurements of a storage device.

Make helper completion mean service completion

The v257 service manual says remaining service processes are handled by the configured kill policy after the stop commands finish. It specifically warns that a command which requests termination without waiting can prevent a clean stop.

For the fictional collector, distinguish four milestones: the main process accepts the request, admission of new samples stops, the writer completes the declared application commit, and the relevant processes exit. A reply saying “request accepted” establishes only the first milestone. Even a successful process exit does not define what the application promised to store.

A synchronous helper needs an application protocol that can report the later milestones or wait for the appropriate service termination. It should preserve a failure result if the writer cannot complete the commit. Sleeping for an assumed flush duration does not establish either completion or failure.

Compare two stop traces

In the first synthetic trace, the helper sends a shutdown request and immediately returns success. The main process is coordinating the writer when the helper exits. That is the point at which systemd’s configured termination behavior becomes relevant; the earlier acknowledgement has not reserved extra cleanup time.

In the second trace, the same request stops admission, the writer finishes or reports an explicit failure, and the main process coordinates termination. The helper returns only after the agreed terminal condition. This design makes its completion meaningful to the service manager, provided all waits are bounded and the error path is implemented.

Write down who owns each wait. The helper may communicate over an application control channel without being the worker’s parent. Do not assume it can reap an arbitrary worker simply because it knows a PID. The main process remains responsible for the process relationships and completion information its protocol claims to manage.

Review signal targets with the process tree

The v257 kill manual distinguishes these ordinary stop policies:

• KillMode=control-group, the default, applies termination to remaining processes in the unit’s control group.
• KillMode=mixed initially targets the main process, then targets remaining control-group processes at the final-kill stage.

The default initial and final signals are SIGTERM and SIGKILL, subject to the documented signal options. In mixed mode, main-process exit is itself a documented condition for escalation against remaining processes. A writer cannot assume it receives the entire stop timeout after the main process exits.

For the fictional collector, that means the main process must not exit early while relying on its child to finish independently. Review the actual process tree and main-process identification alongside the helper protocol. Changing to KillMode=process or KillMode=none to let workers survive is not a cleanup solution: the manual warns that those modes undermine lifecycle and resource management.

Treat the timeout as a failure boundary

TimeoutStopSec= covers waiting for each stop command and waiting for the service to stop. It is not a single unconditional grace period granted separately to every worker. The configured timeout-failure mode and optional notification-based extensions also affect behavior; this fixture assumes ordinary defaults without extensions.

Give the application a bounded drain policy and a terminal result that can explain unfinished work. If its deadline expires, record which generation was being committed and which completion evidence is absent. A later forced termination must not be relabeled as an orderly application commit merely because the unit eventually becomes inactive.

Do not set an infinite stop timeout as a substitute for that policy. A permanently blocked writer would then leave the shutdown contract unresolved rather than make its data safe.

Cover failed startup separately

ExecStop= is used after successful startup. If startup fails partway through, ExecStopPost= provides the documented post-stop cleanup path, including cases where normal stop commands were never called. Such cleanup must tolerate partially initialized state and should not depend on communicating with processes that have already exited.

A proposed review fixture should include normal drain, writer failure, helper timeout, main-process early exit, and failed startup after creating temporary state. Record request acceptance, application completion, process exits and manager escalation separately. The acceptance artifact is a timeline that proves what the helper’s return means, rather than a log containing only “shutdown requested.”