U-Boot FIT Verification in Sandbox CI: Build a Source-Derived Coverage Ledger
Translate a green sandbox job into a ledger of collected nodes, reached branches and checked assertions before claiming verifier coverage. A selected algorithm label does not identify every path exercised, and a skipped case contributes no successful execution evidence.
This guide pins U-Boot v2024.10’s FIT verification tests. It provides an original CI review method based on source inspection, not a completed sandbox run. No U-Boot build, key generation, cryptographic test or board boot was performed.
1. Freeze the evidence-producing environment
Record the source revision, downstream patch set, sandbox binary identity, resolved configuration, host-tool identities and Python environment. The tagged requirements file pins pytest 6.2.5. A different environment requires an explicit compatibility record rather than an assumption that the test names imply identical behavior.
The sandbox guide explains the host testing role and setup dependencies. Preserve generated fixture identities and tool output with the job receipt; a command line alone cannot reconstruct every input produced during a test.
Use isolated test material and authorized CI resources. Never substitute production signing secrets into a regression fixture to make its results look more representative.
2. Map parameter labels to actual dispatch
The tagged test source has 15 main parameter rows; all except sha1-basic are marked slow. Dispatch chooses global-sign first, required-key second, then the ordinary branch. Only sha1-basic reaches the ordinary full-test extras.
| Source label or function | Reached coverage family | Ledger distinction |
|---|---|---|
| sha256-basic | Ordinary, without full-test extras | Do not credit omitted paths |
| sha1-basic | Ordinary, including forged-tree, fake-root and kernel@ cases | Historical regression route only |
| sha256-pss-pad-required | Required-key branch despite full_test being true | Dispatch overrides flag-based inference |
| sha256-global-sign | Pre-load signature branch | Separate family |
| test_fdt_add_pubkey | Separately parameterized key-tool function | Separate collected nodes |
The SHA1 label describes a legacy test path, not deployment advice. Do not enable a weaker product policy to pursue a coverage count.
3. Preserve collection and selection as evidence
Capture the complete collected node IDs for the actual environment, then the selected subset. According to the pytest 6.2 selection documentation, parameter values belong to node identities, while -k can also match parent names. Thus a broad filename-related filter is not a reliable substitute for inspecting the selected function list.
In particular, do not assume that a filter containing “vboot” necessarily excludes the separate key-tool function in that file. Use the collection result to establish what was selected. Retain exclusions and selection expressions so another reviewer can explain absent coverage.
Keep source-derived expected reachability separate from runtime evidence. Collection establishes that a node exists under those conditions; it does not prove that setup succeeded or the body ran to completion.
4. Read the actual assertion boundary
In run_bootm(), assertions check an expected diagnostic and sandbox continuation presence or absence. Earlier unsigned-positive stages precede required-key enforcement. These are not real-board Linux boots.
For each claimed property, name the assertion that supports it and the fixture state at that point. Avoid converting a test function’s overall pass into a claim that every intermediate artifact was subject to the same trust requirement.
If the assertion checks refusal, record the expected refusal stage and whether the test independently establishes its cause. If it checks a message, retain the exact observed output in the execution receipt. A summary dashboard should link to that receipt rather than replace it.
5. Classify every missing result
The tagged required-tool setup skips a test when a marked tool is unavailable. Other setup and applicability conditions also need review. Having the four tool markers satisfied does not prove every branch-specific dependency is present.
| Recorded state | Coverage treatment |
|---|---|
| Passed with retained output and identity | Credit the executed assertions only |
| Skipped during applicability or setup checks | No executed coverage for that node; preserve reason |
| Deselected | Outside the run, even if source contains useful tests |
| Collection or setup error | Execution evidence incomplete; investigate environment |
| Job terminated before result | Unresolved, never silently counted as passed |
Do not collapse these outcomes into one success percentage. A job that passes a narrow subset while skipping required coverage may be healthy infrastructure yet insufficient release evidence.
Hypothetical green SHA256 job
A fictional team selects one ordinary SHA256 case and sees a green result. Its release note initially claims the extra forged-tree family is covered. The dispatch ledger exposes that unsupported claim without requiring speculation about the test framework.
The team corrects the evidence statement and separately decides how to obtain the missing regression coverage in its approved test environment. Changing the wording does not create coverage; another identified execution or a properly scoped alternative test is still needed.
6. Keep board trust as a separate evidence column
List the real product’s ROM-to-loader trust entry, protection of verifier code and trusted configuration, actual storage/image route, recovery route and final application handoff. Attach target evidence for each claimed property. Leave them unverified where only the sandbox receipt exists.
This does not diminish host regression testing. It makes the result usable: reviewers can see exactly which software assertions have evidence and which integration properties remain open.
Close the ledger with one row per collected node: binary/source identity, parameter values, dispatch family, selected status, runtime result, assertion evidence and unmodeled product boundary. Re-evaluate it when upgrading tests or changing filters. A reproducible, bounded coverage statement is more valuable than an unexplained green badge.
