Long-Run Tests with a Quiet Device: Ensure the Workload Exercises the Intended Resource
A device that remains powered for several days may have exercised the intended storage, radio or peripheral path for only a few minutes. Qualify a long-run test by completed work and resource-specific activity as well as elapsed time. A quiet log and a live heartbeat do not demonstrate sustained exposure.
This article provides an original workload-history schema informed by NASA SP-2016-6105 Rev2. Four small counter-classification controls were executed on CPython 3.12.14. No endurance campaign, leak measurement or target benchmark was performed, and the example counters are synthetic.
Define the resource and the required exposure
Name the resource under investigation and the activity expected to stress it. Storage testing might require writes, synchronization and reopening under defined data sizes. A radio campaign might require meaningful exchanges with the intended peer rather than an idle connection.
State the product-specific duration and workload rationale before running. Include operation mix, payload sizes, concurrency, transitions and any environmental conditions. There is no universal number of hours that establishes endurance independently of those conditions.
Keep the requirement separate from the campaign’s implementation. If the workload generator crashes, that is a test-infrastructure failure. If the generator runs but the device rejects every request, the intended successful-operation exposure may still be missing even though failure handling received activity.
Count the stages that can diverge
| Counter or state | What it establishes | What it does not establish |
|---|---|---|
| Planned operations | Work requested by the campaign definition. | Actual submission or target activity. |
| Submitted attempts | Runner dispatched work through its selected path. | Acceptance by the intended resource. |
| Terminal operations | Each operation ended in a classified result. | Successful useful work. |
| Accepted results | The fixed result predicate was satisfied. | Exercise of every internal resource. |
| Resource-specific progress | The selected path reached its defined observation point. | Unobserved physical wear or durability guarantees. |
A general accepted-result counter can continue increasing while an optional storage path is bypassed. Add observations close enough to the resource to distinguish that case, while documenting what those observations actually mean.
For example, a queued-write counter records admission rather than persistence. An application-level completion may be the right exposure marker for one requirement and insufficient for another. Define the marker from the claim being tested.
Use epochs before subtracting counters
Record a boot or process epoch with each sample. Retain workload revision, resource configuration and the cumulative counters within that epoch. A reset can return a counter to zero without undoing earlier activity.
Do not subtract values across an unexplained restart, wrap or configuration change. Start a new interval and classify the discontinuity. If the counter width and wrap behavior are known, a deliberate wrap calculation may be valid, but it must not conceal an unknown reset.
Keep the raw sample sequence. A final total alone cannot show that the device stalled halfway through, restarted repeatedly or spent most of the observation window idle. Preserve start and end times with units and clock identity rather than reconstructing them from filenames.
Detect a quiet workload without redefining a watchdog
Choose a progress observation window from the workload contract. For each window, compare expected activity, admitted operations, terminal results and resource-specific progress. A legitimate idle phase needs an explicit state and planned duration; otherwise silence remains unexplained.
If the generator is active and storage progress is unchanged, pause the exposure claim and investigate the path. The device may be blocked, the test may be sending the wrong commands, or instrumentation may be broken. Do not immediately reset the device and count the entire interval as valid stress.
This is campaign validity accounting, not a replacement for the product watchdog. Product recovery and bench evidence collection have different responsibilities. The campaign should preserve the first missing-progress interval even when the product’s own recovery later succeeds.
Relate resource history to actual work
Sample the resource quantities relevant to the hypothesis, such as allocated buffers, open handles, queue depth, used storage or retries. Record measurement method, resolution and observation overhead. Do not assume that one memory total identifies a leak.
Compare those histories with operation counts and lifecycle events. Growth during startup followed by a plateau differs from growth associated with every completed cycle. A trend in bytes per operation may help form a hypothesis, but it does not establish causality or justify extrapolating beyond the observed regime.
Retain intervals with failed or incomplete work as separate classes. Dividing a changing resource total by all attempted operations can hide that only a small subset reached the suspected allocation or cleanup path.
Executed counter controls
The host fixture evaluated four synthetic intervals. In the first, the heartbeat advanced while storage_done remained at 20; it reported no storage progress. In the second, storage_done advanced from 20 to 23 within the same epoch; it reported observed progress.
A change from epoch A to B required a new epoch rather than subtraction. A decreasing storage count within unchanged epoch A reported a counter discontinuity. These assertions checked classification only: three completed operations do not meet an unspecified throughput requirement or prove an endurance result.
Finish with an exposure ledger
The NASA verification guidance calls for configuration, conditions, criteria and results in the verification record. For a long-run campaign, include valid active exposure, planned idle exposure, invalid or unknown intervals, operation mix, failures, restarts and gaps in instrumentation.
Report both wall-clock duration and the evidence that the intended resource was exercised. Keep interrupted campaigns and unexplained stalls visible. A defensible conclusion says what ran, how often, under which conditions and with which limits; it does not turn an uneventful calendar interval into unearned coverage.
