A Customer Changes Acceptance Limits After Testing: Preserve the Original Verdict and Reassess the Data
Keep the original report intact, identify the revised requirement separately, and reuse earlier measurements only when they contain everything needed to evaluate the new criterion. A changed threshold does not change what the instrument observed. It can change the engineering verdict, or reveal that the old experiment never measured the required property.
This procedure concerns technical evidence for an embedded product. It does not determine whether a customer may change a contract, whether payment is due or whether a release is legally accepted. Those decisions belong to the applicable agreement and authorized parties. All examples below are hypothetical; no measurements or reevaluations were executed.
1. Separate the requirement from the observation
Create four linked identities: original requirement revision, original test record, proposed requirement revision and new assessment record. Preserve the original pass/fail statement with the criterion that produced it.
NASA’s requirements and configuration-management guidance supports controlled changes and historical status records. The practical method here is to append a new assessment instead of editing an old result until it appears to have always used the new limit.
Also distinguish a request from an approved baseline. An engineer can estimate the effect of a proposed limit without claiming that the proposal is already the governing requirement. Mark that analysis provisional and name the authority needed to activate it.
2. Describe exactly what changed
| Changed element | Question for the retained evidence | Likely route |
|---|---|---|
| Numeric threshold only | Same quantity, units, statistic and conditions? | Reevaluate if complete and valid |
| Maximum becomes percentile | Are individual observations and the statistical rule retained? | Recompute if sufficient; otherwise repeat |
| Timing reference changes | Were both required events captured on a usable time base? | Reconstruct only if supported |
| Environmental range expands | Were the newly required conditions actually exercised? | Test the uncovered conditions |
| Decision rule changes | Are uncertainty and classification inputs available? | Reassess under the approved rule |
Do not classify every wording edit as a threshold change. “Response within 100 ms” can become a different measurement if its start event moves from transmission to application submission. A retained duration cannot reconstruct an unrecorded queue delay.
3. Check the data before calculating a new verdict
Confirm the tested firmware, board, configuration, stimulus, load, environment, procedure and observation method. Identify the population represented by the result: one transaction, a defined sample, a particular unit or a stated set of units.
Then verify data completeness. A chart may omit failed attempts, an average may conceal outliers, and a rounded value may lose the precision needed near a tighter threshold. Preserve any original exclusion rules and identify whether the new requirement allows them.
The evidence gate is not “a spreadsheet exists.” It is “the retained observations and provenance answer this specific revised question.” If the original run was invalid, a new threshold cannot repair its validity.
4. Keep uncertainty and the decision rule visible
The NIST uncertainty discussion associates uncertainty with a measurement result and its measurement configuration. Tightening an acceptance limit does not automatically improve the instrument, sampling method or uncertainty estimate.
Record the applicable decision rule explicitly. A project may require a particular treatment of uncertainty or an indeterminate region, but there is no universal guardband prescribed by this article. Do not introduce one after seeing the data simply to produce a preferred outcome.
If only a rounded display value survives and the new limit falls within the lost precision, report that the retained evidence cannot resolve the decision. Request the original observation or repeat the measurement with an adequate method.
5. Work through a bounded example
Imagine a bench report for a noncritical status query. The original requirement permits an observed maximum of 100 ms under a named workload. The retained result is 97 ms, and the proposed revision lowers the same limit to 95 ms. Under the explicitly assumed simple comparison rule, the old verdict remains pass and the revised assessment is fail.
Neither statement implies that the firmware changed. The assessment should say which requirement each verdict answers, identify the original evidence and preserve its actual conditions and limitations.
Now suppose the proposal instead asks for a percentile under twice the original traffic. A single retained maximum from the old workload cannot establish that claim. The correct output is a data gap and a targeted test plan, not a guessed percentile or an extrapolated pass.
6. Issue a two-verdict assessment record
| Record field | Required content |
|---|---|
| Original conclusion | Requirement revision, result and immutable report reference |
| Proposed or approved change | Exact criterion delta and decision authority |
| Reuse argument | Why quantity, conditions and retained data remain applicable |
| New conclusion | Pass, fail or insufficient evidence under the named rule |
| Next action | Targeted measurement, implementation change or authorized disposition |
Where only part of the dataset qualifies, name that subset and its selection rule. Do not silently merge observations from different product configurations or discard inconvenient results. A reviewer should be able to reproduce the selection and calculation from the retained record.
7. Close the evidence question before changing release language
A useful review ends with one of three outcomes: existing evidence supports the revised criterion; existing evidence demonstrates nonconformance to it; or more evidence is needed. Keep technical classification separate from permission to ship or a negotiated exception.
Update downstream verification matrices and release statements only through the project’s approved process. Retain links to both assessments so maintenance engineers can explain why an unchanged product once passed one baseline and later failed another. That history is essential context, not an inconsistency to erase.
