A Test Passes on the Second Run: Preserve the First Failure Instead of Averaging It Away
A second-run pass is a new observation, not a replacement for the first failure. Keep both attempts, identify what changed between them and require an explicit disposition of the conflicting evidence. Repeatedly pressing “run” until a green result appears creates an undocumented selection rule rather than a qualified acceptance process.
This article defines an original first-failure/rerun/rework record model informed by NASA SP-2016-6105 Rev2 and NIST remedial-measurement guidance. Six synthetic disposition cases were executed with Python 3.12.14. No production unit, factory database or physical retest was involved, and no acceptable flake rate is proposed.
Preserve the first attempt before investigating
Store the unit identity, attempt identity, test/configuration revision, raw observations, expected criterion, result and completion status. Include the relevant station and fixture context. Preserve a timeout or incomplete acquisition as such instead of translating every non-pass into the same product-failure code.
Make the original observation immutable. Corrections should append a disposition with its authorizing evidence and link to the original record. If a parser later misread a number, retain the original raw bytes and original interpretation alongside the corrected interpretation.
The NASA handbook’s verification process records configuration, discrepancies, corrective actions and repeated verification. Its configuration-management guidance retains historical status. The attempt structure here is a manufacturing adaptation of those principles, not a NASA release rule.
Name the intervention between attempts
Distinguish a rerun with no intended change, a reinitialization, a fixture intervention and product rework. These are different experiments. A second run can pass because the product warmed up, a connection moved, persistent state changed or a real defect was repaired.
Write the intervention before starting the next attempt when practical. “No intended change” should still record elapsed time, power/reset history and relevant environmental state. It cannot guarantee that the physical conditions were identical.
Do not call a test repeated after firmware replacement the same-configuration rerun. Likewise, cleaning a contact or adjusting a fixture is a change even when the product was untouched. The record should make the comparison possible without relying on an operator’s memory.
Separate observation, validity and disposition
| Record layer | Example meaning | Must not silently imply |
|---|---|---|
| Observation | The first attempt returned a failing measured value. | That the measurement path was valid. |
| Validity assessment | Evidence establishes an invalid acquisition or a valid product failure. | That a replacement measurement has already passed. |
| Corrective action | A documented fixture repair or product repair was completed. | That the required verification is complete. |
| Release disposition | An authorized process accepts, rejects or holds the unit based on all required evidence. | That the first observation should be erased. |
A valid first failure followed by an unexplained pass remains conflicting evidence. An invalid first acquisition can be superseded for acceptance purposes only through a justified validity decision and the required new observation. Neither situation is resolved by averaging two Boolean results.
Use bounded reruns to investigate a hypothesis
Specify what the next run is intended to distinguish. Repeating without handling can test persistence; a controlled reset can test state dependence; a qualified alternative station can help separate unit and station effects. Keep each intervention limited enough to interpret the outcome.
NIST’s remedial-action guidance uses repeat measurements to investigate whether an out-of-control signal reflects a transient problem or a lasting change. It also considers artifact, instrument and operating causes. This does not authorize unlimited production retries or imply that the latest value is automatically the correct one.
Define a stop condition and escalation owner before running. If the planned diagnostic observations remain inconsistent, hold the case. More repetitions can be useful evidence, but selectively retaining only successful ones destroys information about the tested behavior.
Read the executed record-model cases
The synthetic model appended a passing second attempt to a failing first attempt while leaving the predecessor unchanged. Its result was “hold unresolved first failure.” This tests preservation and classification logic, not whether any particular physical failure was genuine.
| Synthetic history | Additional supplied evidence | Model outcome |
|---|---|---|
| First attempt passes. | No preceding failure in this limited model. | Eligible for authorized review. |
| Failure, then pass. | No explanation or disposition. | Hold unresolved first failure. |
| Failure, then failure. | No current pass. | Hold. |
| Failure, then pass. | Approved evidence invalidated the first acquisition. | Eligible for authorized review. |
| Failure, then pass. | Required corrective-action closure supplied. | Eligible for authorized review. |
| No result. | No acceptance evidence. | Hold. |
All six expected outcomes passed. The model receives approval and corrective-closure flags as assumptions. It does not establish root cause, check every required test, validate unit identity or grant production release. A real implementation must bind those decisions to evidence and permissions rather than accepting an arbitrary Boolean.
Report first-pass and eventual outcomes separately
Keep the unit population and denominator explicit. First-attempt outcomes, diagnostic attempts, post-rework outcomes and final dispositions answer different questions. Counting every successful retry as a new good unit can distort both the apparent yield and the unresolved workload.
A useful report links each unit to its first attempt, all later attempts, interventions, validity decisions and final status. It should expose a unit with a passing latest attempt but unresolved earlier failure. Avoid a dashboard that shows only the latest row.
If statistical reliability or sampling conclusions are needed, design that study separately. The number of retries selected after seeing failures is not automatically an unbiased reliability experiment, and this record model supplies no universal acceptance rate.
Close with evidence that explains the change
For a demonstrated fixture fault, qualify the correction and assess other results that may have used the affected path. For product rework, verify the changed and potentially affected functions. For an unexplained intermittent result, retain the hold or approved residual-risk disposition required by the product process.
The engineering deliverable is an auditable chain from first observation to final decision. A second pass is valuable evidence when its conditions and meaning are preserved. It becomes misleading when it is used to make the first failure disappear.
