Intermittent Failures Only in Parallel Tests: Identify Shared State Before Adding Sleeps
When tests fail only in parallel, first inventory shared names and mutable state. Two otherwise correct cases can overwrite one result file, change a process-global setting or reuse a cached connection. Adding a sleep may change the interleaving without repairing ownership.
This guide focuses on runtime test-host interference, not shared electrical supplies or parallel build dependencies. A bounded two-thread CPython 3.12.14 fixture was executed using temporary files owned by the campaign. No production test runner, device or real instrument was accessed.
Establish which dimension changes the result
Compare isolated execution, serial execution in both orders and deliberate concurrent execution. Preserve the same inputs and expected results. An order-dependent serial failure suggests state left behind; a concurrent-only failure suggests overlapping access, although both can coexist.
Record worker identity, process identity, working directory, environment snapshot, fixture instance, temporary paths, port assignments and connection ownership. The label “isolated test” is insufficient if two workers resolve the same default filename.
Keep the first failure evidence before retrying. A rerun can succeed because the second worker finished, a shared cache became initialized or a stale file happened to contain the expected value. That success narrows timing but does not explain the original result.
Inventory the shared state explicitly
| Shared object | Typical collision | Ownership question |
|---|---|---|
| Temporary or output path | One case reads another case’s file. | Is the namespace unique to the run and worker? |
| Environment or working directory | A helper resolves different configuration after another thread changes it. | Can the helper receive explicit immutable inputs? |
| Global cache or singleton | Initialization from one device affects another. | Which configuration and identity key the cache? |
| Mock or patched function | A test observes another test’s replacement. | Does the patch scope match the execution scope? |
| Connection or port | Responses or server ownership cross cases. | Who has the lease and how is association checked? |
Include cleanup paths. One worker deleting a shared directory while another still uses it is an ownership error even if the setup paths were initially correct. Cleanup must only affect objects owned by the current case or an explicitly coordinated shared fixture.
Replace timing guesses with a forced interleaving
The executed fixture makes worker A write the text A, then signals an event. Worker B waits for that signal, writes B to the same path, reads it and signals a second event. Only then does A read the file.
The result is deterministic under the enforced order: B reads B and A also reads B. The assertions confirmed that A’s expected content was replaced. This is an intentionally created shared-path defect, not an observed failure rate from a real test suite.
A corrected control repeats the same order using separate A.txt and B.txt paths in an owned temporary directory. Each worker then reads its own value. The schedule remains concurrent; only the namespace ownership changes.
The Python 3.12.14 threading reference defines Event as a signaling flag and documents bounded waiting. The fixture uses events for ordering, wait timeouts and bounded joins for hang detection. It does not use sleep to manufacture an assumed ordering.
Choose isolation or serialization for the actual resource
Give independent artifacts independent names. Prefer a run-owned directory and explicit paths passed into helpers over changing the process-wide working directory. Include the case identity in the result record as a separate association check, since a unique filename alone does not prove the content belongs to that case.
For genuinely shared resources, define an owner and serialize the complete critical operation. A lock around only file creation does not protect a later read from an intervening overwrite. The protected interval must match the invariant.
Do not assume one thread lock coordinates separate worker processes or different hosts. Select a coordination mechanism with the required scope, and qualify failure/recovery behavior. If a fixture cannot support parallel use, declare that constraint explicitly instead of depending on favorable scheduling.
Keep configuration inputs stable
Build each worker’s configuration before launch and pass it explicitly. For subprocesses, provide the intended environment mapping and working directory through the subprocess interface. Avoid temporarily mutating shared process state while another thread may read it.
Review cached objects for identity and lifetime. A cache keyed only by test name can return a connection configured for a different device. Resetting the cache between every test may hide the defect rather than establish a correct key or ownership model.
Some state cannot be safely reset inside a running process. A fresh process can provide a useful isolation experiment, but report that changed boundary. Passing only in separate processes identifies a clue; it does not automatically prove the in-process implementation is repaired.
Verify the repair under adversarial order
Retain the forced failing interleaving as a regression. Add serial A-then-B and B-then-A controls, repeated concurrent runs and any relevant cancellation or cleanup boundary. If order is randomized, save the actual order as well as the seed.
Assert output identity and content, not merely absence of exceptions. In the synthetic defect both threads completed successfully; a runner checking only thread termination would have missed the wrong result.
Observe cleanup separately after failure. Ensure a failed worker cannot remove another worker’s artifacts or leave a resource advertised as available while it remains occupied. Broader runner recovery requires tests beyond the two-file example.
Close with the shared-state explanation
Record the object that was shared, the invariant it violated, the interleaving that exposed it and the ownership change that repaired it. The NASA verification framework supports retaining configuration and discrepancy evidence; the concurrency experiment here is an original engineering application.
Keep unrelated limitations visible. Correct per-worker paths do not prove instrument concurrency, supply capacity or backend transaction safety. A useful closure repairs the identified shared state and preserves a test that would detect its return, rather than declaring all parallel failures solved.
