Linux Dynamic Debug: Enable the Smallest Useful Set of Driver Call Sites

Start from the running kernel’s dynamic-debug catalog, select the few call sites that answer one question, and save their original flags before enabling them. A source file containing debug statements is not proof that those statements are available in the deployed image. A successful control-file write is not proof that the intended path executed.

This procedure uses upstream Linux v6.12. It concerns supported dynamic-debug call sites, not every printk statement and not arbitrary source instrumentation. All module and function names in the example are fictional. No live control file, module or console policy was changed for this article.

Inventory what the running image can emit

The v6.12 dynamic-debug guide describes /proc/dynamic_debug/control as a catalog of source location, module, function, flags and format text. With debugfs available, a corresponding control path can also exist there. Verify the path actually exposed by the chosen image.

Read the catalog through approved diagnostic access and select candidate rows locally. Record the kernel and module build identities. Search using the real module name shown in the catalog, not the driver directory, device-tree node or userspace device name. Preserve exact file names and line ranges for the current build.

If the expected row is absent, investigate configuration, compilation, module loading and source revision before widening the query. A nearby release may have moved or removed the call site. Dynamic debug cannot enable a message that the image does not expose through this mechanism.

Turn one hypothesis into a small selection

Suppose a hypothetical driver sometimes retries a transfer without completing recovery. The first question is whether its retry function reaches the recovery-return branch. Identify the available messages at entry, retry decision and recovery result, then choose only the rows needed to distinguish those states.

A query combines selectors with AND, not OR. For example, module example_driver func example_retry +p enables print behavior only for matching rows in that module and function. Adding a file or line constraint narrows the set further. It does not create an alternative selection if the added constraint matches nothing.

The language supports file, function, module, format, class and line selectors. Most string selectors support wildcard matching, while the documented class selector has its own restrictions and does not accept wildcard class names. Line ranges have no internal spaces. Quote wildcard and format strings so the shell does not reinterpret them before the kernel sees the query.

Record the intended change and the inverse

The flag p controls printing. The plus and minus operators add and remove selected flags; the equals operator sets the specified flags. Prefix decorators can identify module, function, source line and context, but adding every decorator is unnecessary if it obscures a small bounded trace.

Before enabling anything, partition the selected rows into those already printing and those currently disabled. A later blanket -p would incorrectly disable pre-existing diagnostics. Build a restoration list for only the changes made in this session, preserving unrelated flags and call sites.

Catalog output is evidence, not an executable restoration script. Do not write the whole saved catalog back into the control file. Construct valid queries using exact selectors and the intended flag changes. If concurrent diagnostic users may modify the same rows, coordinate ownership or recheck before restoration.

Use a bounded enable, verify, capture and restore sequence

1. Save the baseline catalog rows, chosen selector and expected matches. Record why each message is necessary.
2. During the authorized diagnostic window, write the reviewed query to the control file. Treat write errors and parser errors as failed setup.
3. Read back the selected rows and verify that their flags changed as expected. Check that unrelated rows were not broadened into the selection.
4. Run one bounded reproduction and collect the kernel log through the existing approved channel.
5. Restore only the flags changed by the session, read back again and archive both the capture and restoration evidence.

For the fictional function, the control payload might be module example_driver func example_retry +p; after a session where those rows were initially disabled, the corresponding narrow removal payload is module example_driver func example_retry -p. These are unexecuted query examples, not a request to alter any running host.

Check message delivery separately from selection

The catalog reports enabled call sites, while console policy affects what a console displays. Do not infer an unexecuted branch solely from an empty serial capture. Confirm the collection channel, severity handling, buffering and relevant rate limits without globally increasing log output by default.

The v6.12 implementation applies parsed queries to registered descriptors. A valid query can still select no useful site, and an enabled site emits nothing if execution never reaches it. Therefore keep three distinct statuses: available, enabled and observed during the test.

A log message also proves only the location and values that its actual format captures. Review the source around it before treating it as a completed operation. A “starting recovery” message cannot establish that recovery returned successfully, and a pointer value is not an object-lifetime guarantee.

Measure and limit the observer effect

Record message count over the bounded workload, capture overflow and the application’s completion behavior with diagnostics off and on. A retry loop can generate much more output than its apparently harmless call site suggests. Slow serial output may change scheduling or extend the very timeout being investigated.

If the capture saturates, narrow the selected branch or shorten the window. Do not solve the problem by discarding all repeated records without keeping an occurrence count and loss indication. For event timing finer than the print path can support, choose a dedicated tracing mechanism rather than claiming printk timestamps measure the underlying operation precisely.

Module unload/reload or a new build changes the catalog’s lifetime. Revalidate the inventory and original flags after those transitions. A restoration query based on obsolete line numbers can match no rows while appearing superficially reasonable.

Handle boot-time questions explicitly

v6.12 also supports boot and module-load query forms, but their processing time limits which early messages can be enabled. Use them only when the suspected operation occurs before the runtime control is available. Record bootloader quoting, final command line and the exact module-load path.

For a completed investigation, deliver the hypothesis, selected rows, original and final flags, exact query payloads, observed messages and output-rate comparison. The result should explain which branch was reached and what remains unobserved. Reversible, narrow call-site selection makes the evidence easier to trust than a console filled with every debug line.