A Test Is Skipped: Keep Unsupported Hardware Separate from Missing Equipment
A skipped or filtered test needs a reason before it can be removed from a coverage claim. Unsupported product hardware, missing bench equipment, unavailable host tools and deliberate selection filters are different conditions. Preserve them separately and reconcile every required configuration against actual execution evidence.
The implementation examples below are pinned to Zephyr v4.0.0 Twister. A seven-case original policy model ran on CPython 3.12.14, but Twister was not executed and no Zephyr build or device test was performed. The model is a reporting example, not an implementation of Twister’s complete selection logic.
Start with the required configuration set
List requirements against the hardware variants, firmware configurations and operating modes that must satisfy them. Decide applicability from the product definition and reviewed capability evidence before looking at test outcomes.
If a shipping variant lacks a feature by design, a feature-specific functional test may be outside its scope. A required refusal or unsupported-feature response can still need a separate test. Conversely, a capable board without the required cable or external sensor has a bench gap, not an unsupported product.
Preserve the matrix before selection filters run. A report listing only selected instances cannot reveal required cases that disappeared during discovery or filtering.
Do not flatten Twister’s statuses
The v4.0.0 status definitions distinguish filtered, not run, skipped, blocked, failed, error and passed. The existence of those labels does not tell you why a particular case received one. Retain its reason and the phase that produced it.
In testplan.py, missing declared hardware capabilities can add a No hardware support filter. Other branches add command-line, toolchain, memory and test-scope filters. These represent different decisions even when a dashboard gives them the same color.
The integration-platform handling can also promote some filtered conditions to errors, with explicit excluded filter categories in the tagged implementation. Do not assume that one status mapping applies to every invocation or downstream version. Keep options, test metadata and the actual report together.
Distinguish fixture metadata from physical availability
The Twister fixture guide describes a test’s dependency on external equipment or special wiring. Hardware-map entries declare which fixtures boards provide. The command-line fixture option assigns its listed fixtures to all boards selected by that invocation, so an overly broad declaration can overstate the available setups.
In v4.0.0 testinstance.py, testsuite_runnable checks supported harness types and matches a requested fixture name against supplied fixture names. Configuration suffixes after a colon are excluded from that name comparison. check_runnable can also consider matching hardware-map entries.
That is a metadata match, not a continuity test, calibration check or proof that a sensor is currently connected. Qualify the physical fixture separately. Do not add a fixture name merely to make a test appear runnable.
Read build-only evidence honestly
The same check_runnable path can return false for build-only tests, disabled slow tests or unavailable simulator executables under its stated conditions. A missing executable does not establish that the product lacks the hardware feature.
In the tagged runner’s successful build path, an instance that is not marked runnable receives NOTRUN, with missing test cases marked as built only. This is useful build evidence, but it is not runtime success.
Other branches classify certain memory-overflow build failures as skipped when the relevant error option is not enabled, subject to integration handling. Therefore even the word skipped cannot be interpreted universally as an intentional harmless exclusion. Inspect the reason and original build log.
Apply a coverage-reconciliation taxonomy
| Evidence condition | Campaign classification | Required action |
|---|---|---|
| Reviewed product scope excludes the capability. | Documented non-applicability. | Retain rationale and any required refusal test. |
| Capability required; fixture absent. | Blocked by equipment. | Provide and qualify the setup; keep the gap open. |
| Capability required; host tool absent. | Blocked by tooling. | Repair the runner through its controlled process. |
| Build succeeds without execution. | Build covered; runtime missing. | Run the required behavior on a suitable target. |
| Runtime skip or deliberate exclusion. | Unexecuted requirement pending review. | Explain the condition and approve or close the gap. |
| Result cannot be associated with the required configuration. | Scope unknown. | Recover trustworthy association or rerun appropriately. |
These campaign categories are original reporting policy, not replacement Twister status strings. Preserve the original status and reason in separate fields so the interpretation remains auditable.
Use controls that expose disappearing coverage
The executed host model took already-reviewed applicability, fixture availability, tool availability and a simplified result class as inputs. Seven cases produced documented exclusion, fixture block, tooling block, execution missing, runtime-skip gap, covered and scope-unknown outcomes respectively.
The scope-unknown case stayed unresolved even when its supplied runtime result said pass. This checks that a passing result without a valid applicability association cannot silently complete a requirement. The model did not discover hardware capabilities, inspect tools or invoke Twister.
For an actual campaign, compare the required configuration set with discovered, selected, built and executed sets. Inspect the differences by reason. Include a deliberately unavailable fixture in an isolated validation of the reporting pipeline, and verify that its requirement remains visibly unexecuted.
Close or retain every gap
Deliver the pinned Twister version, exact invocation, test metadata, hardware map, original reports and requirement reconciliation. Mark completed runtime evidence separately from build checks and approved non-applicability.
A reduced pass denominator should never make coverage appear better merely because more required tests were skipped. Report the unresolved configurations and the owner of each next action. The useful decision is which requirements have credible evidence and which still need a suitable environment, not whether the dashboard contains fewer red cells.
