Known-Bad Fixtures for Test Harnesses: Prove the Suite Can Fail for the Right Reason

A test suite needs known-fault challenges that require the correct diagnostic result, not merely a red icon. Include a valid control, deliberately invalid product data and infrastructure/identity failures, then verify that an always-pass or wrong-reason interpreter is detected. A harness that fails because its fixture file is missing has not demonstrated detection of the intended product defect.

This article uses Python 3.12.14 unittest and five synthetic result mappings. Three host campaigns were executed: the correct interpreter, an always-pass mutation and a wrong-reason mutation. No saleable product was damaged, physical fault injected or real manufacturing result classified.

Write an independent expected outcome

For each challenge, state what is deliberately wrong, where that condition enters the test chain and which diagnostic category should result. Keep the expectation independently specified rather than generating it by calling the same interpretation function under test.

A known-bad artifact should have a stable identity and a clear purpose. It might be a recorded invalid response, a fake instrument that times out or an approved test load designed to expose a measurement path. These exercise different boundaries and must not share an exaggerated claim of complete fixture validation.

The NASA verification handbook links procedures to criteria, configurations and discrepancy evidence. For harness qualification, apply that discipline to the challenge itself: the expected failure reason is part of the criterion.

Separate product failure from invalid evidence

Synthetic challenge	Expected category	Why the distinction matters
U1 reports integer value 5 within the example interval 0–6.	Pass.	A harness that rejects everything is not qualified either.
U1 reports integer value 9.	Product fail under the example rule.	Validly interpreted data violates the supplied criterion.
U1 reports a malformed value string.	Invalid result.	Missing numeric evidence is not a measured product failure.
Fake transport reports a timeout.	Infrastructure error.	The result path did not establish product behavior.
U2 reports an otherwise acceptable value.	Identity mismatch.	A good result for another unit cannot satisfy U1’s test.

These records are already-decoded Python mappings. The campaign is not a full serial, JSON or instrument parser test. Encoding, framing, duplicate keys, numeric edge cases and authentication need their own fixtures at the boundary that handles them.

Assert the exact category and supporting context

Python 3.12.14’s unittest reference provides equality assertions and checks for specific exceptions. Use a precise expected category or exception contract rather than accepting any exception as successful rejection.

For a real harness, also assert the attempt identity, affected test and absence of prohibited downstream actions. A test may correctly report an invalid value yet still write a passing unit record before raising the error. The observation boundary should include the effect that matters.

Keep failures in acquisition distinct from failures in interpretation. If a known-bad input never reaches the interpreter because setup failed, mark the challenge incomplete or failed at setup. Do not credit it with detecting the intended data defect.

Read the executed mutation controls

The correct interpreter classified all five supplied fixtures as expected. The unittest campaign ran five checks with zero failures and zero errors. This confirms the narrow result-category contract for those inputs.

The first mutation returned pass for every input. The same five-check campaign then produced four assertion failures: out-of-range, malformed, timeout and wrong-unit challenges were all incorrectly accepted. The valid control still passed.

The second mutation classified only the malformed value as product_fail instead of invalid_result. That campaign produced one assertion failure and no runner errors. This shows that “it failed somehow” would have missed the wrong diagnostic owner, while the exact-category oracle detected it.

The four and one assertion failures are expected mutation detections, not failures on real products. They are useful evidence that the challenge suite can reject two deliberately broken interpreters. They do not establish that every possible harness defect is detectable.

Prevent the challenge from becoming a privileged shortcut

Route test fixtures through the same interpretation and reporting path used by ordinary results where practical. A special “known bad” flag that directly forces the desired failure category can make the self-check pass without exercising the suspected logic.

At the same time, keep test-only injection mechanisms isolated from production use. Define how they are enabled, identified in reports and prevented from masquerading as real unit evidence. Do not store synthetic results in a production acceptance table without an explicitly designed and authorized separation.

Use harmless substitutes first. A software fake can test timeout classification without unplugging a live instrument. A qualified reference load can test a physical path without damaging a saleable assembly. The substitute’s limits must remain visible.

Qualify the full chain in layers

Begin with pure interpretation fixtures, then exercise transport/framing, instrument simulation and controlled physical challenges as separate layers. Each layer should have its own expected failure owner and evidence that the challenge actually occurred.

For a physical known-fault load, document ratings, connection sequence, required protection and the specific fault it represents. An arbitrary short or damaged customer product is not an acceptable generic fixture. Hazardous equipment needs its qualified safety procedure and personnel.

Keep positive controls alongside negative ones. If an instrument or parser stops working entirely, every negative challenge may appear to “fail correctly” unless the valid case proves the chain can still complete.

Maintain challenges when the harness changes

Version the fixture inputs, expected outcomes and interpreter/reporting implementation. When requirements change, review the oracle explicitly instead of updating expected values merely to make a new run green.

Record which deliberate mutations the suite detects and which gaps remain. A new diagnostic category, missing-data representation or release-state change can require additional challenges even if the core numeric limit is unchanged.

The deliverable is a known-fault set with independent expected categories and demonstrated false-pass detection. It builds confidence in the harness’s ability to fail for the intended reason, without claiming that a small software fixture set certifies the physical product or the whole production line.