Kretprobe Misses: Check Probe Capacity Before Interpreting Missing Function Returns
Check return-probe instance capacity and missed-entry accounting before interpreting a missing return as a stuck function. A kretprobe needs state for each admitted in-flight invocation. If that state is exhausted, the observation can omit an invocation even though the function itself later returns normally.
The baseline is upstream Linux v6.12 and the kernel kretprobe API. Interfaces built on top of it may expose different controls, so verify their mapping rather than copying names blindly. No probe module was loaded and no overlapping-call fixture was executed for this article.
Establish the probe’s actual scope
Record the kernel and module build, target symbol or address, registration result, probe configuration and handler implementation. The v6.12 Kprobes guide describes unsupported and blacklisted locations. A rejected registration is a setup failure, not a trace showing that a function never ran.
Check whether the compiler inlined the function at important call sites. Probing one out-of-line symbol does not automatically instrument all inline instances. Also examine unusual return behavior and architecture restrictions before choosing a function merely because its name resembles the operation of interest.
Keep handlers observational. They execute under context restrictions and must not sleep, wait for I/O or acquire a blocking semaphore. Use bounded, reviewed recording storage. Heavy printk calls can perturb timing, recurse through probed code or introduce unrelated lost observations.
Separate in-flight calls from CPUs
maxactive sizes the pool of simultaneous return-probe instances. A function that sleeps can have many overlapping invocations even on one CPU. Recursion can also increase outstanding instances. Therefore CPU count alone is not a complete capacity model.
The v6.12 implementation supplies a default when the requested maxactive value is nonpositive, using the greater of ten and twice the number of possible CPUs. That implementation default is not a product-specific guarantee that every invocation will be observed.
Estimate the maximum admitted overlap from the caller model: concurrent file descriptors, worker count, blocking points and recursion. Then verify the observed in-flight high-water mark under the intended workload. Keep a margin justified by that model rather than claiming a universal multiplier.
Read missed and filtered cases separately
The kretprobe’s nmissed increments when entry cannot obtain an instance. In that situation the user entry handler is skipped too. Counting only your entry-handler records therefore understates the number of real calls during capacity exhaustion.
A user entry handler can also intentionally reject an invocation by returning nonzero. That invocation does not receive the corresponding return handler, but it is a deliberate selection decision rather than pool exhaustion. Count accepted, deliberately filtered and capacity-missed cases separately.
Do not confuse the return probe’s instance-miss counter with the embedded kprobe’s missed-handler behavior under recursion or other probe interactions. Preserve the exact counter and registration epoch. Reloading or registering a new probe resets parts of the accounting and can make an apparent improvement merely a new baseline.
Build an independent operation ledger
For every external test request, record an operation identity and its expected target-function path. For admitted probe instances, record entry, return and per-instance data that binds those observations. A task identifier alone may be ambiguous when a task calls the function recursively or repeatedly.
Use the probe’s per-instance storage for correlation where the reviewed implementation supports it. Do not share one global timestamp across concurrent calls. Capture the target’s return value using architecture-appropriate conventions only after verifying the handler and function signature.
At the end of a bounded test, distinguish completed instances from those still legitimately in flight. Stopping capture immediately after the last submission can leave normal delayed returns outside the window. Quiesce the owned workload before deciding that accepted entry/return pairs are unbalanced.
Propose a controlled overlap fixture
- Use a disposable test function with an explicit, legal blocking point in the function body, not in the probe handler.
- Register a return probe with capacity for two simultaneous instances and verify registration success.
- Hold two calls at the function’s gate, then enter a third while the first two remain outstanding.
- Inspect the return-probe miss counter and independent call ledger, then release all calls and record completion.
- Repeat with adequate reviewed capacity, unchanged function behavior and a fresh accounting baseline.
This is an expressly synthetic test specification. The anticipated result is a capacity-missed observation in the constrained case, not a claim that a kernel test was run. A real fixture must ensure all three calls truly overlap; scheduling three tasks without a controlled gate does not establish that condition.
The gate must not hold a resource needed by the controller or other callers. Avoid targeting a live driver path with a fabricated wait just to validate accounting. Use the fixture to understand the observer, then apply a minimally intrusive probe to the real workload.
Interpret the missing-return evidence
| Evidence | Next interpretation |
|---|---|
| nmissed increased | Some real invocations may lack both entry and return records; capacity affected coverage |
| Entry handler deliberately filtered the instance | No return record is expected for that selection |
| Accepted instance remains in flight at capture stop | Extend the owned completion window or investigate its actual wait |
| Independent call exists but no probe evidence | Check registration, inlining, site coverage and other missed-handler paths |
Only after those cases are excluded should a missing return become strong evidence about the target function’s progress. Correlate with its wait condition and other observations rather than treating probe silence as a deadlock diagnosis.
Report coverage alongside latency
If using return probes for durations, report eligible requests, accepted instances, misses, filtering and incomplete windows with the duration distribution. Capacity misses can bias results toward shorter or less-concurrent calls. A percentile computed from admitted samples is not automatically representative of all invocations.
Deliver the concurrency model, configuration, actual counter deltas, handler constraints and correlation method. A successful investigation establishes which calls were observed and why others were absent. It does not claim that increasing maxactive repairs the driver; it repairs one possible measurement limitation.
