failslab on a Test Kernel: Verify Driver Cleanup at Each Allocation Failure
Use a controlled allocation-failure sweep to connect each observed failed allocation with the resources already acquired and the cleanup that follows. The pass condition is not merely an error return. The driver must preserve its contract, release partial state exactly once, and allow the next valid operation to succeed.
This guide is pinned to upstream Linux v6.12. failslab injects slab-allocation failures; it does not cover every page allocation, hardware fault or source-level allocation call automatically. Fault injection can destabilize the whole target. The procedure below is a proposed review and test plan for an isolated, recoverable system with no valuable workloads; no injection was performed here.
Inventory resources before enabling a fault
Choose one bounded operation, such as allocating a request context through a driver interface or performing a controlled probe. List each acquisition in order: context storage, buffers, references, registration, interrupt ownership and externally visible state. For every step write the matching release and the condition that says ownership has transferred.
Use a ledger even for managed resources. Automatic release at a later device boundary may be correct, but it does not necessarily satisfy an interface that must recover immediately after a failed request. Identify whether the operation returns to an initial state, retains a reusable partial state, or deliberately marks the device unavailable.
A hypothetical request setup obtains a context, a descriptor array and a completion buffer before publication. Failure of the third allocation should release the first two and leave no published request. That statement is the acceptance contract; the actual injection must demonstrate which allocation failed before the test can claim to have exercised it.
Verify failslab eligibility in the exact build
The v6.12 configuration exposes CONFIG_FAULT_INJECTION, CONFIG_FAILSLAB and the debugfs-control option. Check the final build and available interfaces rather than enabling a random set of boot parameters on a production image.
The tagged failslab implementation excludes its bootstrap cache and __GFP_NOFAIL requests. It also defaults to ignoring allocations with __GFP_DIRECT_RECLAIM; the debugfs control is named ignore-gfp-wait. This naming difference matters when a test appears to miss ordinary sleepable allocations.
If the selected allocations require changing that filter, make the change only within the isolated campaign and record its previous value. Cache filtering can further narrow eligible allocations using the cache’s failslab mark. Confirm the actual cache and its aliases; filtering by an assumed name is not evidence that the intended request was covered.
Understand what the Nth control counts
The fault-injection reference documents per-task fail-nth files, including the thread-specific path under /proc/self/task/<tid>/. An armed value selects the Nth eligible fault-injection opportunity in that task, not the Nth allocation line in the driver source.
Crucially, fail-nth is shared across enabled fault-injection capabilities. It takes precedence over generic probability, interval and times controls, while capability-specific eligibility still matters. The generic implementation also applies its stack filtering on the Nth path. Setting another capability’s probability to zero is therefore not sufficient to exclude it from a per-task Nth campaign.
Prefer a narrowly configured diagnostic kernel and confirm the actual failure stack. If other capabilities are present, classify what consumed each index rather than labeling every failure a slab failure. Setup allocations, unrelated syscalls and helper-library activity can shift the index, so prepare descriptors, buffers and logging before arming the test.
Run one verified operation per index
- Begin from a known device state with unrelated injection disabled and the selected filters recorded. Verify a fault-free operation and cleanup.
- In a minimal test thread, arm one Nth value immediately before the intended operation. Avoid unrelated work in the armed interval.
- Capture the operation result and promptly read back the Nth state using a reviewed harness, then disarm it before reporting, cleanup or the next case. Record whether the fault was consumed and its actual stack.
- Reconcile every acquired resource with its release or documented retained owner. Check that no callback, registration or published pointer outlives the failed operation.
- With injection off, perform the same valid operation again. Confirm functional success and clean teardown before moving to the next index.
The upstream example demonstrates a systematic single-syscall sweep; adapting it to a driver requires the same isolation of the armed interval. Review even the harness’s status collection, since surrounding work can consume opportunities. A zero readback indicates that the armed count was consumed, but the failure stack establishes where and by which capability.
A positive remaining count means that index was not reached during the observed interval. It can end a deterministic sequence’s sweep only after the selected input path and filters are understood. It does not prove coverage of different input branches, retries, asynchronous work or later teardown allocations.
Use an explicit result matrix
| Observed case | Required evidence | Decision |
|---|---|---|
| Context allocation fails | No later resource acquisition or publication; defined error | Accept this entry boundary if the next valid request works |
| Descriptor allocation fails after context creation | Context released once; no descriptor use; no callback remains | Accept only with balanced ledger |
| Completion buffer fails after two acquisitions | Both prior resources unwound; visible request count unchanged | Investigate any retained or double-released resource |
| Nth value is consumed outside the intended allocation | Actual stack and capability classification | Do not count the target site as tested |
The rows describe the hypothetical setup, not measured outcomes. Error codes must follow the real interface contract; do not hard-code one universal errno for every resource failure or assume every caller exposes the original allocator error unchanged.
Keep asynchronous probe and worker paths separate
A task-local injection does not automatically follow work delegated to another kernel thread. If probe runs asynchronously or a request schedules a worker, identify the context that reaches the allocation. A quiet submitting task says little about the worker’s error paths.
Plan a separately scoped test for those paths, with reviewed filtering and observation. Do not broaden probability globally merely because the initial per-task test missed them. Likewise, repeated unbind/rebind is appropriate only for a disposable test device whose recovery and surrounding dependencies are understood.
Stop and restore with evidence
Stop a case at an unexplained warning, lost resource, unexpected external effect or failed recovery. Preserve the first failure rather than continuing a sweep on corrupted state. Restore per-task settings and every changed capability filter, then verify the normal operation with injection disabled.
Deliver the index-to-actual-site map, resource ledger, error results, recovery tests and untested branches. The strongest result is a driver whose partial construction has explicit ownership at every step. A large number of injected errors without that mapping is activity, not demonstrated cleanup coverage.
