Boot Verification Fails Only on Production Images: Compare Signing Inputs Byte for Byte
Freeze one known-good signed artifact and compare its descendants at every production transformation boundary. The aim is to locate the first unexplained change in verifier-consumed bytes, not to regenerate signatures until a manufacturing image boots.
This procedure uses MCUboot v2.2.0’s ordinary unencrypted, uncompressed hash-then-sign image as a concrete baseline. Other formats require their own coverage and decoding rules. No signing key was accessed, no production image was obtained, and no target readback or verification test was executed.
1. Preserve the failing evidence before rebuilding
Retain the exact failing delivery file, its delivery manifest, the bootloader binary/configuration identity and the earliest available verification log. Preserve a known-good signed control from the same intended release path.
Do not overwrite the failing file with a fresh build. Rebuilding can change inputs, metadata or packaging enough to remove the evidence that identifies the faulty step. Give each retained object an immutable local identifier and a whole-file digest.
Record whether the failure occurs while parsing the container, locating the image, checking its hash, selecting a trusted key or validating its signature. A broad “boot failed” result is too coarse to establish post-signing mutation.
2. Freeze the signing checkpoint
The v2.2.0 image-tool guide describes preparing binary or Intel HEX inputs with the header and metadata expected by MCUboot. Header reservation, padding and address handling depend on the selected input format and options.
Record the pre-signing input identity, tool revision, non-secret option set and final signed output identity. Include version, header size, security-counter setting, dependencies and signing-mode identifiers where applicable. Keep private keys, credentials and signing-service tokens out of the report.
Use the same frozen signed output as the baseline for every later checkpoint. A separately signed output belongs to a different lineage, even when someone gives it the same release label. Comparing two independently generated containers does not isolate a later production transformation.
3. Build a transformation ledger
| Checkpoint | Object to preserve | Comparison question |
|---|---|---|
| Signed output | Exact signed image bytes | Which bytes and metadata were approved? |
| Packaged delivery | Archive, transport wrapper or manufacturing container | What reversible transformation was intended? |
| Decoded image | Image extracted with the production decoder | Does it reproduce the frozen signed image? |
| Placement plan | Image start, length, fill rules and target region identity | Will the verifier read the same byte sequence? |
| Controlled readback | Approved non-secret test-device image range | Did programming preserve the intended bytes at the intended location? |
Store the transforming tool/version and its relevant options on every edge. An archive name alone is not an edge description. If a step intentionally adds transport framing, compare the decoded result rather than demanding that the outer file hash equal the inner image hash.
The tagged image-tool implementation provides the concrete load/save and image-construction behavior to inspect. Do not assume a text-format conversion preserves addresses, gap filling or the image start merely because its output parses.
4. Compare at three distinct levels
Whole delivery object: compare the archived file to the file actually delivered. This catches wrong versions, transfer errors and accidental replacement. A difference at this level may be harmless repackaging, so continue with the defined extraction.
Verifier input image: compare the extracted bytes at the declared image origin. Resolve prepended headers, address offsets, erased-fill assumptions and length truncation. Two identical payload fragments at different assumed origins can still produce different verifier inputs.
Authenticated span: for the scoped MCUboot profile, compare the header span, payload and protected TLVs covered by the hash. The tagged verifier computes over that span and separately validates digest/signature records. A changed byte outside the span is not automatically irrelevant; its parser, signature or trailer semantics still need review.
5. Find the first unexplained difference
Walk forward from the frozen signed output. At each edge, apply only the documented decoding or mapping, compare expected bytes, and stop at the first mismatch. Preserve both sides and report the first differing relative offset, affected region and transforming stage.
Common candidates include a post-signing version stamp, header padding applied twice, an image-start offset interpreted twice, altered gap-fill bytes, a truncated final record or accidental selection of an unsigned intermediate file. These are hypotheses to test, not diagnoses established by the symptom alone.
If all verifier-consumed bytes match, stop blaming packaging. Compare the actual bootloader’s trusted public-key identity, signature mode, algorithm support and policy with the control environment. An artifact can be unchanged while the deployed verifier differs.
6. Keep verification evidence policy-equivalent
A host utility validating a mathematical signature against a supplied public key does not automatically reproduce the device’s trusted-key selection or required-signature policy. Record which part of the decision the host check establishes.
Use the unchanged valid control under the same device verifier before testing the failing descendant. If the control fails too, investigate the baseline and loader state. If only a transformed descendant fails and its covered bytes differ, the transformation becomes a concrete engineering lead.
Any readback should use an approved read-only procedure on a recoverable development unit and the minimum relevant non-secret range. Do not dump full production storage, keys or unrelated customer data merely to compare an application image.
Hypothetical manufacturing stamp
A build pipeline signs an image successfully. A later manufacturing step writes a release label into reserved bytes that lie inside the authenticated header span. The package still has the same filename and the application payload is unchanged, but the verifier’s covered input is different.
The ledger locates the first difference between signed output and manufacturing output. The fix is to place the authorized label in the correct pre-signing input or a separately designed metadata location through the normal release process. Re-signing an unexplained final file would hide the process error rather than define the intended boundary.
7. Add a release gate at the faulty edge
Retain a regression fixture for the identified transformation and assert the decoded verifier input matches its approved parent. Keep intentional mutable metadata explicitly outside that equality assertion only when its own validation and ownership rules are documented.
Close with the checkpoint ledger, exact first-difference report, corrected transformation contract and repeatable control. The outcome is a traceable explanation of which bytes changed, where and why that change affected verification, without exposing signing material.
