Randomized HIL Tests: Save the Seed and the External Event Schedule

A saved seed can reproduce a generated command list under controlled software conditions. It cannot reproduce a USB delivery delay, an independently running peer, an interrupt order or the electrical state left by a previous case. Preserve the generated schedule and the observed external events alongside the seed when a randomized hardware-in-the-loop test fails.

This procedure uses CPython 3.12.14 for a small executed host fixture. No hardware-in-the-loop campaign, serial transmission or timing replay was performed. The goal is to make a failure replayable at a stated level without calling a second random run an exact physical reproduction.

Separate three different sequences

Record the generator’s decisions, the runner’s submitted operations and the observed device events as separate streams. A generated delay is an intention. The host may submit later, and the device may observe a different interval again.

For each generated event, retain its sequence number, action, parameters, intended trigger and intended delay units. For each submission, add the actual connection generation and host timestamp. Link observations through a request identifier, marker or qualified correlation method rather than assuming that the next log line belongs to the next command.

When observations use different clocks, retain their original timestamps and the mapping evidence. Record uncertainty or unknown ordering where appropriate. Two overlapping uncertainty intervals do not establish which physical event occurred first.

Pin the generator contract

The tagged Python random reference documents independent Random instances, generator-state capture and restoration. Its reproducibility notes qualify repeated-seed behavior and explain that most algorithms can change across Python versions; the compatibility guarantee for random() does not promise identical output from every higher-level helper forever.

Save the language version, runner revision, generator implementation, seed type and value, operation list and call pattern. An integer seed and a textual representation are not interchangeable metadata. Do not use a cryptographic generator when deterministic test generation is required, and do not reuse a deterministic test generator for secrets.

Prefer one explicit generator owner when generating a schedule. Even thread-safe random draws do not make the assignment of draws to racing workers repeatable. Independent per-worker generators help only when worker identities, stream derivation and input order are also stable.

Save decisions before callbacks can alter them

Where feasible, generate and serialize a bounded schedule before starting the physical test. This makes the intended experiment inspectable even when the runner crashes halfway through. Retain the actual file and its content hash, not just the seed that supposedly regenerates it.

If later decisions depend on responses, record each decision together with the triggering observation and generator-state checkpoint or draw position. A diagnostic callback that consumes one extra random value can change every later choice while the seed remains unchanged.

Keep untrusted replay data as validated plain records. Do not execute arbitrary serialized objects or commands from an external failure report. The replay runner should accept a narrow action vocabulary, enforce bounded parameters and retain the same hardware safety restrictions as an ordinary campaign.

Capture the initial physical state

Manifest section	Evidence to retain	Why the seed cannot replace it
Software	Runner, generator, DUT and peer artifact identities.	The same choices can drive different implementations.
Fixture	Routing, supply condition, connected devices and qualified starting state.	External state is not generated by the host PRNG.
History	Relevant prior operations, persistent data and boot/session identity.	Starting from a warm state may differ from a cold start.
Execution	Intended schedule, actual submissions, observations and dropped-event indicators.	The runner cannot infer unobserved physical timing.
Failure	First violated invariant and evidence before cleanup.	A final error message can hide the causal sequence.

The NASA verification guidance supports recording test configuration, conditions and results. The manifest here is an original application of that principle, not a NASA-prescribed HIL file format.

Use an honest replay ladder

First reproduce schedule generation on the pinned host implementation. Next replay the saved schedule against a deterministic software model. Then replay its controlled actions on an authorized, recoverable bench while observing the real external events again.

Label the result accordingly: identical generated decisions, equivalent model trace, repeated target symptom, or physical timing reproduced within specified bounds. These are different claims. A target failure occurring again does not establish that every intermediate event was identical.

For a timing-sensitive defect, define the critical relationship explicitly, such as completion arriving after cancellation but before buffer reuse. The bench experiment needs evidence of that relationship. Repeating a delay value in a script is insufficient when adapter buffering or target scheduling can alter the relationship.

Executed host controls

A CPython 3.12.14 fixture generated eight action/delay records using a private Random instance seeded with integer 421. Repeating the same calls produced the same records. One additional random() call before generation changed the resulting schedule. Saving and loading the records through JSON preserved them exactly.

A separate state-capture control restored four random() draws after setstate(). These checks establish local generator and record behavior only. They did not test concurrency, hardware timing, electrical state or cross-version compatibility. The complete synthetic schedule and assertions are retained with the article’s evidence.

Minimize without erasing the trigger

After preserving the original failure, reduce the schedule one change at a time. Remove unrelated actions, shorten data and simplify initial conditions while retaining the violated invariant. Preserve both the original and reduced manifests.

If reduction stops reproducing the failure, identify which relationship changed before concluding that the device is nondeterministic. Missing initial state, an unrecorded peer action or a callback-dependent draw can explain the difference. A useful final regression includes its required preconditions and unresolved timing limits as well as its compact action list.