Hardware Test Reservations: Recover a Board After the Worker Dies Without Double Booking It
When a hardware-test worker disappears, move its board into recovery quarantine before assigning it again. Establish that the old owner can no longer issue commands, reconcile any operation already in flight and verify the required hardware state. Process death, reservation expiry and safe physical cleanup are separate facts.
This design uses upstream labgrid 24.0.2 as a concrete reference and an original Python 3.12.14 state model. Five synthetic recovery controls passed. No labgrid service, board, power controller or distributed failure experiment was run; the proposed fencing and cleanup mechanisms require qualification in the real test infrastructure.
Inventory the whole resource group
A board reservation often needs more than the board: serial console, debug probe, power channel, USB multiplexer, instrument route and any shared load. Identify which resources must have one owner and which can safely be shared under a qualified policy.
If two logical board entries reference the same physical power channel, independent reservations can still interfere. Resolve that aliasing in the resource inventory. The scheduler cannot protect a shared physical path that its model treats as two unrelated resources.
The labgrid 24.0.2 project description provides a client/exporter/coordinator remote-control architecture and board-control abstractions. Those components provide useful infrastructure, but the test application still needs a concrete recovery contract for its hardware and operations.
Check reservation semantics at the deployed version
Do not assume that a missed worker heartbeat automatically expires an acquired board. In the 24.0.2 coordinator’s schedule_reservations implementation, reservations in the acquired state are refreshed and do not expire through the ordinary reservation-timeout path. Waiting or allocated reservation behavior is not the same as acquired-resource lifetime.
The reservation object tracks timeout and refresh state. These are software allocation mechanisms, not measurements of board power or evidence that an external tool stopped. Review the exact deployment’s supervisor, client behavior and recovery policy instead of extending a timeout field into an unstated cleanup guarantee.
Also inspect error paths. The coordinator’s resource-release helper attempts exporter release calls and catches failures; place release then updates acquisition bookkeeping. Consequently, a cleared allocation field alone cannot establish that every desired physical cleanup action succeeded.
Distinguish liveness from authority
A heartbeat reports recent communication. Its absence can mean process death, a stalled worker, network partition or a delayed supervisor. A worker that is merely disconnected may continue controlling a device through an existing local handle.
Give each ownership instance a generation and require the component that performs hardware actions to reject stale generations. This is a proposed additional control, not a claim that labgrid 24.0.2 implements this application’s fencing protocol automatically.
Enforce the check at the action boundary. A scheduler-side comparison is insufficient if an old process can still write directly to the serial port or power-controller endpoint. Review every supported access path and prevent bypasses within the authorized test setup.
Use recovery quarantine as a real state
| State | Admission rule | Evidence for transition |
|---|---|---|
| Available. | A new owner may acquire the complete resource group. | Known qualified baseline and no active predecessor. |
| Owned. | Only the current owner/generation may issue permitted actions. | Bounded liveness and valid ownership. |
| Owner uncertain. | No new owner is admitted. | Loss, timeout or inconsistent ownership evidence. |
| Recovery quarantine. | A designated recovery authority performs reviewed reconciliation. | Old authority fenced, in-flight operations resolved and cleanup checked. |
| Available again or held. | Reassign only after every required recovery condition passes. | Versioned recovery result; unresolved conditions remain held. |
Recovery is not necessarily power-off. A board may be programming flash, preserving evidence or controlling a load for which immediate power removal is inappropriate. Define safe actions from the hardware and test stage, with a bounded escalation path when the state cannot be established.
Reconcile commands already accepted
Fencing prevents future stale commands; it does not undo a command the controller already accepted. Retain operation identity, last acknowledged phase and any observable completion state. A lost response can leave the action’s outcome uncertain.
Before reuse, stop or reconcile retained tool processes through their supported lifecycle. Close the prior session, inspect the required device state and establish a fresh baseline. Do not treat killing one test-runner PID as proof that all descendants, remote tools or instrument operations ended.
Serialize recovery with acquisition so two supervisors cannot both declare the resource available. If the coordinator itself restarts, reconstruct ownership from trustworthy records and current enforcement state; an empty in-memory table is not evidence that every device is idle.
Read the executed state-model controls
The synthetic model began with owner A at generation 1. Losing that owner advanced the generation to 2 and entered quarantine. It rejected a new owner before recovery and rejected A’s stale command.
Two additional controls supplied only one recovery condition at a time: fencing without confirmed cleanup, and cleanup without fencing. Both kept acquisition blocked. Only when both supplied conditions were true did owner B acquire generation 2, while A remained rejected. All five controls passed.
This model is deliberately limited. Its Boolean inputs do not implement a physical interlock, authenticated exporter, reliable distributed lease or consensus protocol. They demonstrate why the two evidence requirements must remain separate, not that a deployed laboratory has satisfied them.
Qualify worker-loss recovery without damaging hardware
Begin with fake resources and recorded command traces. Test worker exit, communication loss, late old commands, failed cleanup, duplicate recovery requests and coordinator restart. Require every rejected action and unresolved condition to remain visible.
Then use recoverable development hardware under an approved procedure. Choose interruption points whose consequences are understood and preserve required evidence. Measure actual recovery behavior and confirm that the next owner sees the qualified baseline rather than inherited state from the failed job.
The deliverable is a resource-group inventory, version-specific allocation semantics, enforced ownership boundary and recovery record. A board is ready for another worker when authority and physical state are both established, not merely when its name disappears from a booking list.
