Yocto 5.0.8 Test Exports for Hardware Labs: Keep Runner, Image and Metadata Together
A hardware lab can run Yocto runtime tests without keeping the complete build environment on its test host. The handover must preserve the relationship between the exported runner, selected tests, image metadata and the image actually installed on the board. Otherwise a valid-looking result can describe the wrong release.
This guide audits Yocto Project 5.0.8, using OpenEmbedded-Core revision cd2b6080a4c0f2ed2c9939ec0b87763aef595048. It proposes a release-bundle qualification process, not a completed hardware test or a recommendation to deploy this historical maintenance level.
Inspect the exported entry point before writing lab instructions
The 5.0.8 runtime-testing manual describes exporting selected tests with the testexport task. Its example then invokes runexported.py. However, the pinned export class copies scripts/oe-test into the export root and places metadata under data/. Base the lab’s entry point on those actual exported files rather than copying the documentation example unchanged.
The corresponding runtime executor defaults to data/testdata.json and data/manifest. It accepts explicit target and server addresses. Keep both addresses in the lab configuration: one identifies the device, while the other describes the host as reached from that device. Do not assume their defaults match the lab network.
From the root of a trusted extracted bundle, begin by inspecting ./oe-test runtime --help. This is an instruction to validate during qualification, not a command executed for this article. Preserve the resulting help output with the tested runner revision.
Package the image and tests as one identified release
Create a product-owned release manifest containing the image digest, test-export archive digest, machine configuration, layer revisions, selected test names and expected board variants. Include the package manifest and test-data file as separately identifiable members. A filename such as product-image is insufficient when several build jobs reuse it.
The export class reads deployed image metadata and the package manifest to construct the test context. Those files describe a build; they do not interrogate the board to prove that this build is running. Add an admission check that compares the installed release identity with the manifest before granting the run release-qualification credit.
Define how that installed identity is obtained and trusted. A mutable text label alone is weak evidence. Tie the lab’s deployment record to the approved image digest and use the product’s available runtime identity mechanism. Document the remaining limitation if the hardware cannot establish a stronger binding.
Qualify the bundle on a clean test host
The pinned class copies a fixed set of core runner libraries, then selects case files from TEST_SUITES. A custom case can still import another helper, open a fixture file or expect a host program outside that copied set. Review these dependencies explicitly; successful export does not prove that every custom dependency was included.
Use a fresh, authorized lab host or disposable environment with only the documented prerequisites. Keep the original build checkout and developer-specific Python paths unavailable. Verify case discovery and the intended selected-case list before connecting to hardware. Then exercise the actual runtime path, because import success cannot establish that a helper invoked later exists.
Preserve Python and host-tool versions. If the suite requires exported packages or the optional test SDK, deliver and identify those companion artifacts as well. Do not silently borrow them from another machine or an older release after the first failure.
Separate target access from board qualification
The simpleremote target expects an already running system. Treat deployment, cold power cycling and fixture control as explicitly owned lab steps rather than assuming that an exported SSH suite performs them.
Review the pinned SSH backend before using it: its options disable strict host-key checking and discard the known-hosts file. This is not a strong device-authentication boundary. Confine the qualified setup to an appropriately isolated, authorized test network; do not carry those assumptions into production administration.
For a physical I/O requirement, retain the fixture revision, applied stimulus and actual observation with the result. An emulator run may cover application parsing while leaving electrical behavior untested. Keep those as different evidence rows, even when they execute the same Python method.
Challenge the admission and packaging checks
| Controlled qualification case | Expected decision |
|---|---|
| Correct bundle, wrong installed image | Hold before crediting functional results. |
| Required custom helper absent | Report an incomplete bundle; retain the failure. |
| Required case missing or skipped | Leave its requirement uncovered. |
| Correct image, unavailable physical fixture | Run only applicable checks; defer hardware claims. |
| Matching bundle and qualified target | Execute the named suite and retain actual outcomes. |
These are proposed negative controls, not observed results. Use disposable copies when removing a helper or substituting metadata. Never alter the archived original to make a problematic run appear reproducible.
Return a result package that can be audited
Return the release manifest, bundle identities, target-admission evidence, actual case list, named results and raw failure logs. Separate packaging errors, connection errors, skips and functional failures. Preserve the first failed attempt when a justified retry occurs.
Version and testing boundary: Documentation and the exact tagged-release source were inspected. No BitBake export, runner execution, image deployment, SSH session or hardware measurement was performed. The proposed bundle and admission gates require qualification on the selected BSP, host and fixture before they can support a release decision.
