Early-Boot Panics Before pstore Is Ready: Build an Evidence Coverage Timeline

ByLIU, ZHUOHUI

If the kernel panics before a persistent-storage backend is ready, changing the userspace pstore mount cannot recover the missing capture opportunity. The right question is when each evidence path becomes usable. This guide uses upstream Linux v6.12 to build a coverage timeline for early-boot failures. It assumes the project already owns an appropriate retained-memory configuration; the focus is initialization order, not redesigning the reserved-memory map.

Draw the timeline around observable milestones

Begin with firmware output, kernel entry, early console availability, ramoops driver initialization, successful backend registration, later driver probing, and userspace collection. Mark the suspected panic between milestones rather than assigning an unmeasured time such as “the first second.” Two boards with the same kernel can reach those stages through different firmware and device-discovery paths.

For every milestone, record its evidence source. An external serial transcript can show emitted messages; a source review identifies where code intends to register a backend; a saved configuration shows whether the component was built in or modular. None alone proves all the others. Keep unknown intervals visible instead of filling them with estimated timestamps.

Boot timeline shows a capture gap before backend registration and a separate later userspace mount milestone
Conceptual ordering, not measured timing. A mount exposes records; it does not move capture readiness backward.

Locate the actual registration boundary

In v6.12, ramoops uses postcore_initcall for its initialization function. However, that function registers a platform driver; the probe must still establish the region and call pstore_register successfully. A declared initcall level is therefore not proof that the backend was usable at the earlier failure point. v6.12 ramoops initialization and probe.

The pstore registration path installs the kmsg dumper when the backend advertises dmesg capability. The success message is useful as a timeline anchor, subject to whether the chosen console actually captured it. v6.12 pstore backend registration.

Check CONFIG_PSTORE_RAM in the shipped configuration against v6.12 Kconfig. A module-loaded backend adds the time until loading and successful probing to the uncovered interval. Building it in may reduce that interval, but cannot make dependencies available before they exist. Review actual registration evidence after any change rather than announcing that a built-in driver covers all early panics.

Separate capture readiness from filesystem visibility

The pstore filesystem creates the user-visible record files when it reads the backend. This is a retrieval stage, distinct from registering a writer for future crashes. Mounting later does not mean the previous kernel waited until that mount to start recording. Equally, mounting earlier in userspace cannot rescue a panic that occurred before the writer existed. v6.12 filesystem record retrieval.

Use separate worksheet fields for “backend ready during failed boot” and “record visible during recovery boot.” A blank result in the second field is not evidence about the first. This separation prevents a team from repeatedly editing an init script while the failure occurs long before any init script can execute.

# Read-only clues from a successful comparison boot
uname -r
cat /proc/cmdline
dmesg | grep -Ei 'ramoops|pstore|persistent store|initcall'
cat /proc/mounts | grep pstore

A successful comparison boot supplies landmarks, not proof of the failed boot’s exact sequence. Preserve the original external transcript if one exists. If only the comparison boot has the registration message, label the failed boot’s registration state unknown until the failure location is established.

Choose an earlier independent evidence path

The v6.12 kernel parameter documentation describes earlycon and initcall_debug. An argument-free earlycon can use supported device-tree stdout-path or ACPI information; platform-specific forms need the correct UART and mapping. initcall_debug traces initialization calls. Neither is a universal guarantee that every instruction before normal console setup will be observable. Linux 6.12 kernel parameters.

Ask the board owner for the documented console connection, electrical interface, boot arguments, and capture settings. Start external collection before power-on or reset, and confirm a known firmware marker as well as a kernel marker. Missing firmware output may be a collection setup problem, not evidence that firmware never ran. Do not guess memory-mapped UART addresses or connect an incompatible electrical interface.

Keep the diagnostic configuration separate from the release configuration. Extra printing changes timing and can alter a race. If a failure disappears with tracing enabled, preserve that observation and compare the earliest confirmed milestones; do not infer that adding logging fixed the product.

Validate coverage in stages on a development target

Prepare a proposed matrix with checkpoints before backend registration, after successful registration but before userspace, and after normal collection becomes available. For each checkpoint, state which channel is expected to contain a unique marker and which is expected to be silent. Include a no-failure baseline to verify that the external collector is operating.

Any intentional panic or reset needs an isolated target, an approved recovery image, and permission to interrupt its workload. A controlled failure after registration demonstrates only that later checkpoint. It does not validate earlier coverage. Use source instrumentation reviewed for the selected build if precise stage placement is necessary, and remove it from the production configuration afterward.

Close with a coverage statement, not a recovery promise

  • The failing build and its built-in or modular backend choice are identified.
  • Every timeline landmark has an observation or an explicit unknown.
  • Writer registration and recovery-file visibility are tracked separately.
  • An earlier channel covers the suspected interval, or the gap is disclosed.
  • Controlled-stage tests have expected channels and unique incident markers.

A historical panic that left no surviving evidence cannot be reconstructed by mounting pstore today. The engineering outcome is to narrow where it occurred and ensure the next reproduction has an appropriate capture path. State the earliest demonstrated checkpoint and the remaining gap; do not turn an initialization-order diagram into a claim of measured board coverage.

Similar Posts