Grouping pstore Crash Reports: Stable Fingerprints Without Hiding Different Failures

A growing pstore archive is useful only if engineers can tell repeated evidence from repeated failures, and repeated failures from different bugs. Hashing whole files creates too many groups when timestamps change; grouping everything containing “panic” creates too few. This guide proposes a versioned fingerprint workflow for Linux v6.12 crash evidence. The grouping policy is an engineering design, not a kernel feature or a demonstrated classifier accuracy claim.

Decide what you are counting

Maintain three identifiers: an artifact identity for exact collected bytes, an incident identity for a particular failure on a device, and a fingerprint for similar failure evidence. These answer different questions. Retransmitting one file can add an artifact observation without adding an incident. Two crashes with the same fingerprint remain two incidents.

Ramoops record numbers are not durable fleet incident identifiers: the documented counter resets after restart, allowing later records to replace earlier ones. Use a collector-generated incident envelope that preserves provenance rather than treating dmesg-ramoops-0 as a globally unique event. Linux 6.12 ramoops record behavior.

When reliable incident correlation is unavailable, report artifact count and a bounded or unknown incident count. Do not silently present upload count as crash frequency. A proposed minimum envelope includes a controlled device identifier, collection identifier, failed-build identity, original file name, byte hash, and a relationship to other records from the same recovery session.

Partition by the failed build first

Create the first grouping boundary from the actual failing kernel and relevant module build identities. Keep architecture and product variant available as facets. A release string alone may be reused across different binaries, so prefer an immutable release-manifest reference or artifact hashes from the build system. A recovery-image identifier belongs in a separate field.

This guide assumes stack addresses have already been interpreted against matching artifacts. The kernel’s bug-hunting documentation describes using matching debug information and stack traces for diagnosis; grouping cannot repair a wrong symbolization step. Quarantine unresolved or mismatched-build reports rather than merging them with confident clusters. Linux 6.12 bug-hunting guide.

Choose discriminating features deliberately

A reasonable initial key combines failure class, a normalized primary diagnostic, the faulting symbol or location, a short ordered stack signature, module identity, and the rule version. Keep exception subtype, access direction, or another diagnostically important field when the available report establishes it. Do not remove every number: an error code can distinguish failures even when an address cannot.

Exclude clearly volatile presentation fields such as wall-clock timestamps and process identifiers from the key, while retaining them in the original record. Consider function offsets as a strict within-build feature. Across builds, compare a more tolerant signature only as a secondary relationship for review; compiler and source changes can move offsets without preserving the underlying fault.

Preserve taint information as context. Linux documents taint as a retained indication of conditions affecting diagnosis, including events that may predate the current failure. It is a useful facet, not proof that a particular module caused the crash. Linux 6.12 taint interpretation.

Make normalization inspectable

Store both the structured fields and the final digest. A hash without its input features is hard to audit when unrelated incidents merge. Canonicalize field order and text encoding, distinguish missing values from empty strings, and include a rule identifier. Avoid concatenating unescaped fields with an ambiguous separator.

Proposed fingerprint input, not a kernel ABI:
rule = "crash-group-v1"
build = immutable_failed_build
class = parsed_failure_class
site = validated_fault_symbol
stack = ordered_relevant_frames
confidence = complete_or_partial

Do not let the confidence field imply an arbitrary score. Define “partial” using observable conditions such as a missing faulting line or truncated stack. A partial record should normally enter a separate review queue; forcing it into the nearest complete group can hide an emerging issue. Keep multiple diagnostic sections when one panic contains an earlier Oops and later consequences.

Test false merges and unnecessary splits

Build a hand-reviewed fixture set before running the policy over a fleet. Include a pair that differs only in timestamps, a pair with the same top helper but different fault sites, two distinct exception subtypes, a truncated report, a repeat upload, and two builds that expose similarly named functions. Label these synthetic or manually curated fixtures rather than claiming field coverage.

For the timestamp pair, the proposed fingerprint should remain stable. For the different-site pair, it should preserve a distinction even if a generic reporting helper appears first in both stacks. The repeat upload should not inflate incident count. A cross-build resemblance should appear as a suggested relationship while preserving separate primary groups.

Inspect every merge among known-different fixtures and every split among known-equivalent fixtures. Revise one rule at a time and compare the changed memberships. Keep difficult examples instead of deleting them to improve a headline accuracy figure. The acceptance question is whether the grouping helps an engineer find distinctions without concealing uncertain evidence.

Version the policy and preserve reversibility

Store raw evidence immutably under the project’s retention policy, with fingerprint assignments in a separate derived index. When the rule changes, recompute assignments and retain the old-to-new mapping. A resolved bug label should not automatically attach to every new report that resembles an old cluster; review the failure site and build relationship first.

• Artifact, incident, and cluster counts are named separately.
• Every group retains its failed-build boundary and raw evidence links.
• Normalization rules preserve important diagnostic differences.
• Partial and ambiguous reports remain visible.
• Regression fixtures detect false merges and unnecessary splits.
• A rule update can be audited and reversed without modifying original logs.

The practical output is a triage queue with explainable membership. Fingerprints reduce repeated reading; they do not replace root-cause analysis or prove that a fix covers every report in a group.