A Requirement Has Two Verification Methods: Resolve Conflicting Analysis and Test Results
When analysis and target testing disagree, align their exact claims before choosing which evidence to rely on. Check the requirement revision, measured quantity, configuration, workload, assumptions and decision rule. Preserve both results and design the next observation to distinguish the competing explanations.
This article proposes an engineering reconciliation method. It does not assert that models are inherently weaker than tests or that a passing test overrides a failed analysis. The example is fictional; no model or target experiment was executed, and no release decision is made.
1. Record the disagreement without averaging it away
Create a discrepancy record linking the analysis artifact and test result to the governing requirement. Write each conclusion in its own terms: predicted maximum, calculated bound, observed sample, statistical estimate or simple functional outcome.
NASA’s verification guidance, sections 5.3.1.2.2–3, treats discrepancies as matters to investigate in the product, procedure or conditions. Apply the project’s response process rather than continuing to present a disputed requirement as unconditionally verified.
Averaging a predicted value with a measured value does not resolve why they differ. Nor does counting more green reports make an unexplained red result disappear.
2. Check whether the two methods answer the same question
| Comparison dimension | Analysis record | Test record |
|---|---|---|
| Quantity | Defined variable and boundary events | Actual stimulus and observation events |
| Configuration | Modeled hardware/software/options | Identified unit/build/settings |
| Conditions | Input ranges, load and environmental assumptions | Conditions actually exercised |
| Statistic | Bound, average or distribution claim | Sample count, exclusions and reported statistic |
| Decision rule | Criterion and treatment of uncertainty | Criterion and treatment of measurement uncertainty |
If one method measures application execution time and the other measures request-to-response time including queueing, the numeric comparison is not yet a direct contradiction. Both may be valid for different quantities, while only one answers the actual requirement.
3. Inspect assumptions as inputs, not footnotes
List assumptions that could explain the difference: no concurrent traffic, bounded interrupt interference, a particular cache state, a fixed device response or an excluded startup interval. Determine which assumptions are justified for the claimed operating envelope.
A model can be internally correct but inapplicable because a premise is false on the target. A test can be correctly recorded but outside the specified conditions. Neither situation should be disguised as a mysterious disagreement between theory and reality.
Keep assumption corrections separate from changing the requirement. If the product needs to operate outside the original model’s envelope, expanding the analysis is a new evidence task rather than a wording fix.
4. Check the measurement path independently
Review trigger selection, timestamps, synchronization, scaling, instrument settings, data exclusions and postprocessing. Confirm that the recorded sample belongs to the claimed build and event.
The NIST uncertainty approach ties uncertainty to the measurement result and configuration. Do not assume a displayed digit resolves a threshold decision without considering the method’s actual uncertainty and approved decision rule.
Keep measurement uncertainty distinct from model inadequacy. A missing execution path is not automatically covered by a small numerical tolerance attached to an otherwise incomplete model. Conversely, an instrument artifact should not be accepted as a product failure merely because it looks pessimistic.
5. Choose an observation that separates explanations
| Candidate explanation | Discriminating evidence | Interpretation limit |
|---|---|---|
| Different timing boundaries | Capture both event pairs in the same controlled run | Does not validate every workload |
| Unmodeled interference | Compare approved workloads with relevant interference observed | Finite tests do not establish a universal bound |
| Wrong target configuration | Verify identities and repeat on the intended configuration | Does not retroactively change the old result |
| Processing error | Independently inspect retained raw observations and calculation | Missing raw data may require another experiment |
Write the predicted observations for competing explanations before the next run. Change a controlled factor where practical, preserve the original artifacts and use safe authorized test conditions. A run that cannot distinguish the hypotheses may add volume without resolving the discrepancy.
6. Work through a fictional timing conflict
A hypothetical analysis predicts 8 ms for processing after dispatch. A target report records 12 ms from request submission to completion, against a 10 ms end-to-end requirement. The first review discovers that the model omitted queue waiting.
The next proposed experiment captures submission, dispatch and completion separately. If the observed interval includes 4 ms before dispatch and 8 ms afterward, that would explain the arithmetic difference for that run. It would not by itself prove a queueing bound or compliance under all permitted loads.
The analysis must then address the end-to-end requirement with justified queue assumptions, or the design must change. The earlier processing-only calculation remains useful evidence with a narrower scope; it should not be relabeled as an end-to-end pass.
7. Close the discrepancy with a bounded conclusion
Record whether the conflict came from incompatible claim scopes, a model/input defect, a measurement/procedure problem, an actual product nonconformance or unresolved evidence. Attach corrected artifacts without deleting the originals.
State which requirement claims are now supported and which remain open. If a change was required, identify its configuration and the verification still needed. A reconciled explanation is not automatically proof that the product meets the requirement.
The outcome should let another engineer trace both original conclusions, understand why they differed and reproduce the reasoning behind the current disposition. That is stronger than choosing whichever method produced the preferred release status.
