Fault Injection Through a Simulator: State Which Hardware Failures It Does Not Represent
A simulator returning the same error code as a failed device does not automatically reproduce the device failure. Review the causal path, outstanding work, side effects and recovery state before using that simulated result as evidence for a hardware fault requirement.
This article defines an original fault-equivalence worksheet. Its only executed example is a small CPython 3.12.14 state model; it is not a QEMU, peripheral-emulator or target campaign. NASA SP-2016-6105 Rev2 supplies the verification-planning context. Every physical test described below remains proposed and requires an approved, recoverable bench setup.
Name the requirement and the injected boundary
Begin with the required behavior: reject a corrupted reading, bound a retry, preserve a committed record, recover a link or move an output to a defined state. Then identify where the simulator inserts the fault: application call, driver return, register read, interrupt delivery or modeled physical interface.
Those boundaries expose different software. Returning an application error may bypass driver timeout detection entirely. A register-level model may exercise the driver but omit electrical settling, bus contention or a device’s undocumented state transitions.
Record the simulator version, model revision, build configuration and injection mechanism. If a substitute is an ordinary host stub, call it that. A broad label such as “hardware simulated” should not hide which parts of the target are absent.
Compare observable behavior, not just the label
| Simulated condition | Possible physical cause | Equivalence gap to investigate |
|---|---|---|
| A read immediately returns timeout. | Peripheral response is delayed beyond the deadline. | A real response or interrupt may still arrive later. |
| A write call returns failure before changing state. | Link fails after the peripheral accepted the write. | The external action may have happened despite uncertain completion. |
| One stored byte is changed. | Storage suffers interrupted programming or media damage. | Granularity, adjacent data, status reporting and persistence can differ. |
| An input value becomes constant. | Sensor, wiring or conversion path stops updating. | Status flags, timestamps and electrical diagnostics may differ. |
| A callback is omitted. | Interrupt path, DMA engine or peripheral stalls. | Resources may remain owned and later activity may still be possible. |
These are hypothetical relationships, not claims that every listed physical cause has that exact behavior. Select the actual device specification and schematic before making a product-specific mapping. One simulator case can represent a narrow observable symptom without representing the full cause.
Build a causal-equivalence worksheet
Give each row a fault identity, physical hypothesis, injection location, time of injection, pre-existing operations, modeled side effects, observations and required recovery. Add a column for excluded behavior and a target-test owner.
Separate what is established by the model from what is assumed about the hardware. For example, the model might prove that an application rejects a stale sequence number once delivered. Whether the peripheral actually supplies that identifier, and whether a stale completion reaches the same code path, needs separate evidence.
The NASA verification guidance distinguishes methods and requires suitable configuration and result records. Use analysis, simulation and physical testing for the claims they can support; do not rename one method as another to fill an empty verification row.
Exercise both sides of an uncertain completion
A timeout is especially easy to model too narrowly. Define whether it means the command was never accepted, accepted but not completed, completed without acknowledgement, or completed with a late response. If the interface cannot distinguish those cases, application recovery must account for that uncertainty.
For a proposed write test, examine at least the no-effect branch and the effect-with-lost-completion branch. Check whether retrying can duplicate an action and whether querying state can safely resolve the uncertainty. Use the product’s actual command semantics; some operations are idempotent and others are not.
Also model delayed evidence crossing a new session. A late callback from an old operation must not satisfy the new operation merely because both used the same buffer or channel. That is a useful software invariant, but the model still needs a credible representation of how callbacks are generated and associated.
An executed counterexample with the same error
The retained Python fixture contains two deliberately simple models. The first returns the string timeout without scheduling an effect. The second schedules a write_commit event, returns the identical string, and processes that pending event afterward.
The assertions confirmed equal visible return values and unequal effect lists: the first remained empty, while the second contained write_commit. No device wrote anything. The example only demonstrates why return-code equality cannot establish side-effect equivalence.
A real simulator qualification would compare richer traces: accepted command, resource ownership, output state, completion event and cleanup. It must also include a fault-free control. A harness that always suppresses an output can appear to contain every injected fault while failing the ordinary requirement.
Assign the missing physical evidence
For each model gap, choose an observation or target experiment that can address it. Examples include capturing actual completion timing, checking peripheral state after a lost acknowledgement, or inspecting affected storage boundaries after an approved interruption.
Do not improvise dangerous electrical faults to make the simulator table look complete. Fault application needs qualified limits, suitable equipment, protected outputs and a restoration plan. Some hardware failure modes require specialist qualification and cannot be safely approximated by disconnecting an arbitrary cable.
Record an uncovered row when the required physical evidence is unavailable. A simulator result can still support the software branch it exercised, but the release scope must retain the unresolved hardware assumption.
Close with a bounded claim
The final report should state the injected boundary, model identity, exercised invariant, observed result, assumptions and remaining target evidence. Preserve the first failure and the fault-free control as well as the corrected regression.
Accept a model-to-hardware equivalence claim only for the behaviors justified by those records. The useful outcome is not a large count of synthetic faults; it is a clear account of which failure responses were tested and which physical behaviors still need evidence.
