Supplier Test Reports on a Different Board: Decide Whether Evidence by Similarity Is Defensible
Accept a supplier’s earlier test report only for claims whose relevant differences are identified and justified. “Same processor” and “same firmware family” are not complete similarity arguments. A report can be useful evidence for one requirement while leaving another requirement unverified on the delivered board.
This article proposes a review method for engineering evidence. It does not approve a component substitution, establish certification or replace a product-specific verification plan. No supplier report or real board is assessed here. All examples are hypothetical and contain no device-specific electrical limits.
1. Start with one claim, not a whole-board verdict
Write the exact requirement the report is supposed to support. Identify the measured quantity, stimulus, conditions, acceptance rule and configuration. “The board passed” is too broad to evaluate.
NASA’s verification guidance, section 5.3.1.2.1, recognizes similarity as an analytical approach. That recognition does not make every older test transferable. The original method below requires a separate argument connecting the tested configuration to the delivered one.
Separate direct test evidence from the reasoning that extends its applicability. Labeling a conclusion “reused by justified similarity” makes its dependence on the argument visible; calling it a new target test would misrepresent what happened.
2. Establish both configurations
Request the tested board revision, populated options, component identities relevant to the claim, firmware/build configuration, external devices, fixture, supply conditions and environmental settings. Record the corresponding delivered configuration beside them.
A missing field is an unknown difference, not an unchanged field. If a report identifies only a product marketing name, ask for enough underlying configuration evidence to resolve the claim. Do not fill gaps from a catalog photograph or a current datasheet describing a later revision.
| Difference class | Evidence to request | Why the claim may change |
|---|---|---|
| Signal path | Schematic/layout revision and relevant component delta | Stimulus or observation may reach different circuitry |
| Execution path | Build identity, options, driver and application changes | Different code or scheduling may be exercised |
| Operating conditions | Load, supply, environment and peripheral population | The old test envelope may not cover delivery |
| Measurement method | Procedure, instrumentation and processing revisions | Results may not represent the same quantity |
3. Trace each difference to the requirement path
For every difference, describe the mechanism by which it could affect the requirement. Then state the evidence supporting “relevant,” “irrelevant within this scope” or “unknown.” This is stronger than assigning an unexplained low-risk label.
For example, a changed display connector may appear unrelated to a parser’s response to an artificial byte sequence. But if the claimed result is end-to-end response time under display activity, the same hardware change can affect the relevant workload or driver behavior. The requirement boundary determines relevance.
Do not rely on absence of a known problem as proof of independence. Where the architecture, source or implementation details needed for the argument are unavailable, retain the uncertainty and choose a bounded repeat test or another justified evidence source.
4. Apply explicit reuse disqualifiers
- The report cannot identify what was tested or which requirement revision governed the verdict.
- The measured quantity differs from the delivered requirement, and retained observations cannot reconstruct it.
- The delivered operating conditions extend beyond those justified by the report and analysis.
- A relevant hardware or software difference lacks an adequate impact argument.
- The report omits failures, exclusions or anomalies needed to interpret the conclusion.
These disqualify the proposed reuse claim; they do not automatically prove the delivered board is defective. Keep “insufficient evidence” distinct from a demonstrated failure. That distinction helps the supplier respond with missing information rather than defending a conclusion nobody can yet evaluate.
5. Use a claim-by-claim disposition table
| Review outcome | Disposition | Record to retain |
|---|---|---|
| No relevant difference, with evidence | Reuse for the bounded requirement if authorized | Configuration comparison and rationale |
| Relevant difference with a validated analytical bridge | Reuse with stated assumptions and limitations | Bridge method, inputs and review |
| Limited unresolved difference | Repeat the affected observation on the delivery configuration | Targeted test scope and remaining claims |
| Unknown identity or broad unsupported differences | Hold reuse and obtain adequate evidence | Missing information and decision owner |
A targeted repeat should cover the mechanism that broke the argument. It should not become a minimal happy-path test chosen merely because it is easy to run. Determine coverage and acceptance limits through the actual verification plan.
6. Consider a fictional two-claim report
A supplier reports parser behavior and input-to-network latency on board revision A. Revision B uses the same parser source and test vectors but changes the network device and driver. No actual compatibility is implied by this example.
The parser claim may be reusable after checking the exact build options, test seam and relevant dependencies. The end-to-end latency claim requires a separate argument for the changed network path. A matching application commit alone cannot supply that argument.
The receiver therefore records one proposed reuse decision and one targeted repeat requirement. It does not mark the entire supplier report accepted or rejected. If later evidence shows the parser build also changed, that first decision returns to review.
7. Preserve the argument with the released scope
NASA’s reporting guidance in section 5.3.1.2.4 associates verification records with the requirement, product and tools used. Extend that record with the tested-to-delivered difference argument and the authority that accepted its use.
Give each reused claim a review trigger, such as a relevant component, firmware option or operating-envelope change. A defensible similarity assessment is a configuration-bound explanation that another engineer can challenge and reproduce. It is not a permanent equivalence label attached to a product name.
