Collecting Kernel Crash Evidence from Field Devices: pstore Archives, Build Matching and Redaction

Define the minimum package and its access boundary

A field crash package should let an engineer identify the device build, understand the reset and locate the relevant evidence without collecting the entire device. Start with raw pstore files, a reset-reason observation, hardware revision, firmware identity, boot-chain versions and a concise account of the workload before failure. Record which items were unavailable instead of quietly substituting guesses.

Define who may read the raw package, who may receive a shared copy and how long each is retained. Prefer a pseudonymous device reference when a serial number is unnecessary. Kernel logs can contain command-line parameters, addresses, identifiers and fragments of application data. Local diagnostic access does not automatically authorize uploading those contents to a support portal.

The kernel reference in this guide is Linux v6.6. The workflow is a design proposal that requires product-specific integration and testing; no customer deployment or completed collection experiment is asserted.

Collect before the next overwrite or cleanup

Map the next boot from backend initialization through pstore availability, archive processing and normal application startup. Identify the single component responsible for taking the first durable copy. Two collectors that independently remove source files can race and leave an incomplete package even when each works in isolation.

On systemd systems, first inspect the existing systemd-pstore behavior. The v254 service reference describes its archival role, while v254 configuration controls destination and source removal. Verify the installed version and overrides; Buildroot or BusyBox devices may instead require an explicitly ordered local script. Do not assume that every product has /var/lib/systemd/pstore.

For a custom collector, copy without deleting the source, verify the destination and mark the package complete before authorized cleanup. A storage or copy failure must remain visible and leave the source intact where possible. The ramoops record lifecycle makes early collection important: further events can reuse finite storage.

Make archives verifiable and bounded

Give each event a unique package identifier, file list, byte sizes and cryptographic hashes. Keep the suspected failure time separate from collection time and note whether the wall clock was synchronized. A boot identifier helps correlate collection activity, but it must not be mistaken for the identity of the previous boot that crashed.

Stage files in the destination filesystem, verify them, then commit a completion marker or rename the completed directory according to that filesystem’s guarantees. Design durability checks for the actual storage stack. A checksum can detect changed bytes; it does not prove who produced the package or that every expected record was captured.

Illustrative collector logic; not an executed script:
  inventory the source records without removing them
  create a private staging directory on approved storage
  copy each record; record size and SHA-256
  write build, boot and observation metadata
  verify destination bytes and required files
  commit a completed package on the same filesystem
  report success or a specific collection failure
  only then apply the approved source-cleanup policy

Set a capacity ceiling, reserve space for essential operation, bound retry frequency and choose a documented rotation policy. Handle unavailable persistent storage and a read-only root filesystem explicitly. Avoid silent fallbacks to volatile /tmp or unlimited accumulation during an upload outage.

Match the evidence to the exact build

Preserve the kernel release string together with the vendor commit, configuration identity, DTB hash, module set and release manifest. A release string alone may identify several internally different builds. Keep a secure index to the matching unstripped vmlinux and module debug symbols in the engineering archive; the device does not need to carry all symbols.

Address resolution requires the corresponding build and any relevant relocation or module information. The Linux v6.6 bug-hunting guide explains the value of useful logs and decoding context. Treat a partially captured call trace as a lead, especially when earlier warnings or memory corruption may have changed the later stack.

A pstore text fragment cannot be used as if it were a complete kernel memory dump or a userspace core file. Keep these artifact types separate in the manifest. Link build and symbol retention to the reproducible release kit so a field report remains diagnosable after another firmware release ships.

Create a controlled redacted copy for sharing

Retain the original in approved restricted storage and make a separate shareable copy. Review command-line arguments, hostnames, network addresses, device identifiers, user data and any embedded memory snippets. Remove secrets entirely; pseudonymize repeated identifiers consistently when relationships between events remain diagnostically useful.

Record which transformations were applied and maintain the relationship between original and redacted packages without exposing the mapping to every recipient. Agree on the upload destination, access scope and retention before transfer. A broad support request is not a reason to publish raw customer logs in an issue tracker or public article.

Give engineering enough context to reproduce the conditions

Add the input sequence, connected peripherals, load, uptime before failure, reproduction frequency and last known working release. State actions already taken, including power cycling, firmware replacement and log cleanup, because these can change the evidence. Record what the field operator actually saw in their own terms before translating it into a suspected technical cause.

Separate observations, hypotheses and proposed tests. “Watchdog reset was reported by the bootloader” is an observation; “the network task deadlocked” is a hypothesis requiring more support. A call trace pointing into a driver is not by itself proof that the driver caused the original corruption.

If the package is insufficient, define the smallest next observation: earlier serial capture, one additional bounded marker or a reproducible laboratory workload. Include the cost and retention implications. An absent record should lead back to the missing-evidence stages, rather than being silently replaced with a fresh unrelated reboot log.

Validate the entire collection chain

Test the workflow on development equipment with a full archive partition, unavailable storage, an interrupted copy, upload failure, two closely spaced resets and deliberately mismatched symbols. The expected result includes a clear failure state, preserved source where possible and bounded resource use. Merely seeing a file in a directory is too weak an acceptance criterion.

Review permissions, incomplete-package handling, retry rules, rotation and cleanup together. Verify that an upload retry does not duplicate an event or discard the only good local copy. Confirm that a device can continue its essential function under the agreed storage-failure policy. None of these tests requires an unplanned production panic.

Establish ramoops retention capability first and account for read-only rootfs persistence. For a field diagnostics design review, prepare fleet size, connectivity limits, sensitive log fields, firmware versioning and storage budget. The deliverable should be an auditable collection contract, with any remote sharing limited to what the device owner authorized.

Technical references and verification scope

Versioned references: ramoops · bug · systemd · pstoreconf