Pause or Roll Back an OTA Rollout? Separate Detection from the Recovery Decision

Pause new rollout admissions when the evidence no longer justifies expanding exposure. Decide what to do with devices already updating or already running the release through a separate, device-state-aware recovery policy. A fleet pause does not undo an installation, and a rollback command is not a safe universal response to missing telemetry.

This guide defines an original campaign decision procedure. It uses RFC 9019’s role separation and RAUC 1.13’s boot-confirmation boundary as references. Failure thresholds, safety constraints and recovery authority must come from the product owner. No campaign or device recovery was executed.

1. Give the two decisions separate owners

RFC 9019 distinguishes the status tracker, firmware consumer and bootloader roles. The tracker coordinates update availability and progress; installation and subsequent invocation involve other components. Use that separation to avoid treating one dashboard button as control over every update stage.

Decision	Question answered	Required evidence
Pause expansion.	May additional eligible devices start this release?	Campaign revision, admission policy and enforcement coverage.
Allow an in-flight operation to finish.	Is its qualified completion path safer than interruption?	Actual stage, cancellation boundary and recovery behavior.
Recover an affected device.	Which approved state can this device safely reach?	Running image, data compatibility, trust policy and recovery route.
Resume expansion.	Has the reason for the hold been resolved with sufficient evidence?	Correction, bounded requalification and explicit approval.

Record the decision maker, reason, affected artifact identity and policy revision. A local boot health check may independently trigger fallback even while the campaign remains paused; neither action should silently overwrite the other’s audit record.

2. Define what a pause actually stops

Choose a precise admission boundary, such as authorization to begin installation. Devices may already possess metadata or downloaded bundles when the server pauses the campaign. A client that acts on an old cached assignment can continue unless its policy requires a sufficiently current admission decision.

Inventory devices as not admitted, admitted but not started, downloading, installing, awaiting reboot, trial-running, confirmed or unknown. These are operational categories for the pause decision, not an instruction to redesign the updater’s internal state machine.

Document whether an admission is revocable, when it expires and which component checks it. Offline devices cannot instantly receive a new server decision. Report that residual exposure rather than claiming the entire fleet stopped at the instant a server flag changed.

3. Classify the first credible failure

Preserve the failure stage and evidence before counting incidents. A download authorization failure, installer rejection and repeated trial-boot failure point to different problems. Missing status may mean delayed reporting, loss of connectivity or a failed device; it does not identify the cause by itself.

Observed pattern	Expansion decision to consider	Device recovery question
Artifact access fails before installation.	Hold affected delivery paths while access is investigated.	Can the unchanged device remain in normal operation?
Wrong hardware is offered an artifact.	Pause the affected eligibility rule immediately under the incident policy.	Was installation rejected, or has any incompatible content been applied?
Repeatable local health regression on the new release.	Stop wider exposure while identifying the affected population.	Which validated recovery image can read the current persistent data?
Status disappears without a confirmed failure.	Hold expansion if uncertainty exceeds the approved evidence budget.	Obtain state evidence before issuing a potentially harmful recovery action.
Healthy devices are confirmed on the new image.	They still count toward exposure while the campaign is paused.	Is there a specific reason and compatible route to move them elsewhere?

These are decision prompts, not universal automatic triggers. A safety-critical symptom may require immediate product-specific protective action regardless of sample count. Conversely, a temporary server outage does not establish that healthy firmware needs to be replaced.

4. Check recovery feasibility before ordering rollback

RAUC 1.13’s boot-confirmation guidance separates completed writes from successful operation and leaves fallback behavior to the boot integration. Therefore, a successful installer result cannot establish either application health or a guaranteed route back.

For each affected state, identify the permitted recovery artifact, supported data schema, security-version restrictions, required keys already trusted by the device, and physical or network access needed. Do not assume that an older image remains acceptable merely because its signature verifies.

If the new application made an irreversible data change, returning to an older root filesystem may worsen the fault. The decision might instead be a forward repair, a restricted operating mode or an authorized service intervention. Those choices require the product’s qualification evidence and safety policy.

Do not interrupt a flash write just because the campaign has been paused. Cancellation must use the updater’s supported boundary. If the exact in-flight state is unknown, retain the uncertainty and obtain evidence rather than inventing a safe cancellation point.

5. Write a pause receipt with remaining exposure

Include the campaign and artifact identifiers, decision time, enforcement revision, affected eligibility strata and known admissions still outstanding. Separate server acknowledgement from device acknowledgement. List clients that may still act on a valid cached admission and the rule that eventually closes that window.

Preserve evidence arriving after the pause. An already-started device can finish successfully, fail or remain unknown. Do not rewrite those events as “canceled” simply to make the campaign look stopped. Reconcile them against the original admission and installation attempt.

Keep incident data minimal and access-controlled. Diagnostic evidence should identify the build, stage and bounded failure class without copying credentials or private application payloads into campaign logs.

6. Qualify pause and resume separately

1. Use recoverable test devices and a harmless development artifact.
2. Pause while devices occupy each documented admission and installation stage.
3. Verify which new operations stop and which admitted operations can still proceed.
4. Reconnect an offline client carrying a stale campaign decision.
5. Deliver delayed and duplicate status reports without changing their original attempt identity.
6. Resume under a new policy revision and confirm the intended population, not an accidental replay of the old decision.

These tests are proposed, not executed. Resume only after the hold reason has a reviewed disposition and a bounded requalification plan. Do not reset the evidence history when opening a new rollout phase.

Hypothetical proxy outage

A campaign sees failed downloads through one factory proxy, while devices already running the release remain healthy. Operators pause admissions for that path, preserve outstanding device states and repair access. They do not send a fleet-wide rollback merely because the dashboard’s failure count rose.

If later evidence identifies a genuine runtime regression, recovery becomes a separate decision with its own compatibility checks. The release record should explain both choices: why exposure stopped and why each affected device was left running, repaired or recovered.