KASAN or KFENCE for an Embedded Linux Memory Bug? Choose the Detection Tradeoff
Start with KASAN for a reproducible kernel memory-access bug when the test image can tolerate its chosen mode; consider KFENCE for longer-running sampling when production-like behavior matters. This is a test-planning decision, not a universal ranking. The right detector must cover the suspected allocation, run on the actual architecture, and preserve enough workload exposure to make the result meaningful.
The technical baseline here is Linux 6.12. Vendor branches may change dependencies, supported configurations, reporting, or defaults. Verify the exact source tree, compiler, architecture, allocator, and boot parameters. Neither tool diagnoses ordinary userspace heap errors by itself, and neither turns an uneventful test into proof that the kernel is memory-safe.
1. State the hypothesis before selecting a detector
Write a short fault hypothesis: which kernel subsystem owns the object, who allocates and frees it, which concurrent path may retain a reference, and what action triggers the suspected access. Mark each statement as observed or inferred. “Occasional reboot” is not yet a memory-lifetime hypothesis.
Identify the memory category. A small heap object, a stack buffer, and a DMA buffer corrupted by a device demand different coverage questions. Do not assume that observing a later CPU fault proves the detector could observe the original writer. Where the object or writer is unknown, make that uncertainty part of the test record rather than promising complete coverage.
2. Apply the applicability gates
The Linux 6.12 KASAN documentation distinguishes Generic, Software Tag-Based, and Hardware Tag-Based modes. Generic KASAN uses compiler instrumentation and supports more memory categories, including stack and globals, than KFENCE. The tag-based modes are arm64-specific; hardware tagging additionally requires MTE-capable hardware. Their memory coverage and compiler requirements differ. Select the actual mode, not merely the word “KASAN.”
The KFENCE reference describes sampled heap allocations protected through a fixed pool and guard pages. Supported allocation sizes extend up to PAGE_SIZE. KFENCE targets heap out-of-bounds, use-after-free, and invalid-free errors; it is not general stack or global instrumentation. A suspected object outside that model should not receive a reassuring verdict from a clean KFENCE run.
Before building, ask the kernel configuration system whether the selected detector is available with the target’s settings. Save the resulting configuration, not just the requested fragment. If a dependency removes an option, fix the qualification plan before spending a long test window on an image that never enabled the detector.
3. Use a decision matrix tied to the investigation
| Investigation condition | Initial choice | Evidence needed before trusting the run |
|---|---|---|
| Short, repeatable kernel heap failure on a recoverable bench | Evaluate an applicable KASAN mode | The reproducer still reaches the suspect path; the image boots within the test resource budget |
| Rare eligible heap failure during long representative workloads | Evaluate KFENCE sampling | Guarded allocations occur during the exposure period; the workload remains representative |
| Suspected stack or global overwrite | Check Generic KASAN applicability | The suspect memory and access path are covered; architecture and compiler support are confirmed |
| Instrumentation makes the original timing pattern disappear | Compare a lighter diagnostic build and a baseline | Comparable workload exposure and a clear account of what changed |
| Suspected device-originated corruption or unknown writer | Broaden the investigation | A verified explanation of which writes the selected detector can observe |
The matrix is an engineering recommendation. It does not imply that KFENCE always preserves timing or that KASAN always catches a repeatable bug. Start with the cheapest experiment that tests the hypothesis while preserving a recoverable device.
4. Budget the experiment using product measurements
Measure the baseline image first: boot completion, available memory under the required workload, response deadlines, throughput, power-sensitive idle behavior, and recovery time. Define acceptable limits before enabling instrumentation. For a watchdog-supervised product, record whether diagnostic slowdown changes the watchdog outcome; otherwise, the detector can replace the original failure with a new one.
Compare each diagnostic image against that baseline using the same input sequence and completed-operation counts. Separate capacity changes from timing changes. A test that processes half as many detach cycles has not supplied the same bug exposure simply because it ran for the same number of hours.
For KFENCE, the sample interval affects guarded allocation opportunities, and a full pool prevents further allocations from being guarded. The documented debugfs statistics and object list help establish whether sampling is active. On power-constrained systems, its deferrable timer option trades idle wakeups against less predictable sampling intervals. Measure the device’s actual behavior instead of assigning a generic overhead percentage.
Keep detector builds separate initially. Changing the compiler, scheduler configuration, drivers, and detector simultaneously makes an improved outcome difficult to explain. If multiple changes are unavoidable, record them as limitations rather than attributing the result exclusively to instrumentation.
5. Validate the observation path, not just the build flag
Use the detector’s documented self-tests on a dedicated development image where deliberate faults are acceptable. The KUnit getting-started guide describes kernel test execution and result handling. Match the detector’s tests to the source revision and inspect individual outcomes. A build success is not a self-test pass, and a simulator result does not qualify board-specific behavior.
Confirm that the intended report reaches the collection destination with the exact build identity. Decide ahead of time whether a diagnostic report should stop the run or permit additional collection. Test that decision safely; do not discover during a field investigation that an unexpected panic policy discards the useful context.
This article supplies a qualification procedure, not executed target commands. No detector image, self-test, timing benchmark, or product workload was run for this article.
6. Interpret both positive and negative results
For a positive report, first correlate the access with object ownership and lifetime. KASAN’s bug classification is documented as best-effort, and hardware-tag asynchronous reporting can be delayed. Keep that boundary in mind when selecting the first source line to investigate. Record the access, allocation, and free evidence available in the chosen mode.
For a negative report, ask three questions: was the relevant workload exercised, was the object eligible and actually covered, and was the report channel working? A negative sampled run is especially weak when a long-lived suspect object was allocated outside the guarded opportunities. Increase relevant exposure or change the detector; do not simply multiply elapsed uptime and call the problem closed.
7. A hypothetical reconnect defect
Suppose a network driver appears to reuse a freed session object during repeated disconnect/reconnect cycles. The fault can be reproduced on a bench image. Begin by evaluating applicable KASAN coverage, preserving the cycle sequence, and checking whether the diagnostic build still reaches teardown concurrently with callbacks.
If instrumentation eliminates the original timing pattern, retain that finding without declaring a fix. A separate KFENCE campaign may provide a lower-intrusion sampling opportunity for eligible heap objects during representative operation. A useful outcome would be an ownership violation tied to a particular session lifetime; an uneventful run would remain bounded by sampling and exposure.
After correcting the ownership defect, rerun the original reproducer, the detector-specific test, and the normal product workload. Release evidence should state which image, coverage model, workload count, and resource limits were tested. Detection, report preservation, and source-level diagnosis are separate responsibilities; a good plan makes all three explicit.
