PCIe AER on Embedded Boards: Preserve the First Error Before Recovery Resets the Link
Preserve the first AER record, PCIe topology and recovery sequence before treating the post-reset device state as the original failure. A link or function reset can change what later reads reveal. The first reporting component, the device affected by recovery and the physical source of the defect need not be the same thing.
This guide is scoped to upstream Linux v6.12 AER reporting and recovery. It is not a PCIe electrical-compliance procedure and makes no signal-integrity diagnosis from a log alone. No link error, AER injection or reset was performed for this article.
Verify who owns AER handling
The v6.12 AER guide describes the root-service driver, its capability requirements and configuration dependencies. On ACPI systems, firmware must grant the relevant native control before Linux handles AER through that path. A configured kernel option alone is not evidence that the operating system owns the reporting workflow.
Record the exact root port or root-complex event collector, downstream ports, endpoint and bound drivers. Preserve domain:bus:device.function identifiers together with the hardware hierarchy and stable device identity. Bus numbers can change after re-enumeration; one remembered BDF is not a durable physical identifier.
Check startup messages, firmware-first or native handling policy where applicable, and the error collection channel. Treat missing capability, disabled reporting and unobserved events as different conditions. Do not force a firmware-control override merely to make a diagnostic option appear active.
Capture the first report as a structured record
Retain severity, reported layer/type, source or requester identifier, status and mask fields, first-error indication and transaction header when present. Keep the raw text and the exact build. A shortened headline often removes the field needed to distinguish a reporting endpoint from the link partner that observed the error.
The guide distinguishes correctable from uncorrectable errors, and divides the latter into nonfatal and fatal classes. Those categories describe reported protocol consequences and recovery needs. They do not, by themselves, identify a faulty connector, power rail or software command.
For a synthetic worksheet, a root-port message names endpoint E, records a transaction-related status and is followed by recovery notifications for the hierarchy. That supports investigating E’s transaction and the affected path. It does not prove that the root port’s silicon or E’s board layout is defective.
Interpret counters as reports, not a parts-failure tally
The AER statistics ABI states that device counters describe errors seen or reported by that device. A problematic endpoint can leave its own counters at zero while its link partner reports the event. Read the relevant hierarchy rather than checking only the endpoint.
Per-error counts and total error-message counts are not necessarily identical because one message can report several error conditions. Root-port totals aggregate messages from the associated hierarchy, including internally reported events. Avoid summing several layers and presenting the result as distinct physical incidents.
Save counter baselines, read times and device lifetime. Removal, re-enumeration, reboot or a changed reporting owner can make snapshots incomparable. Never reset or clear status merely to simplify the first collection; preserve the existing evidence before an approved maintenance change.
Draw a recovery timeline
Use separate rows for first report, driver error notification, quiescing, requested reset, observed link/device return and application recovery. The PCI error-recovery document describes the coordinated driver callback model. Actual driver support and returned recovery decisions affect the sequence.
A fatal link condition generally requires a different recovery route from a correctable event that hardware handled. A driver without the needed recovery callbacks may not recover through the expected workflow. Read the driver’s implementation and logs before promising that the AER core can restore every device automatically.
Preserve the first-error evidence separately from later reset errors. A timeout while a link is intentionally resetting may be a consequence of recovery. Conversely, a recovery callback that fails to restore resource ownership can create an additional software defect after the original event.
Build a controlled capture plan
- Record topology, firmware/OS ownership and normal device operation before the experiment.
- Start an independent log collector that survives the affected device’s recovery path.
- Run the smallest supported workload associated with the real error, recording request identity and relevant power-state transitions.
- Preserve the first report and the full recovery timeline without adding an ad hoc reset before collection.
- Verify device and application state after recovery, including outstanding requests and newly allocated resources.
If software AER injection is used, scope it as a separate isolated recovery-code test. It can exercise reporting and callback paths but does not reproduce the electrical conditions that cause a real link error. Do not label a successful synthetic recovery as proof of board signal integrity.
Connect protocol evidence to the right next test
| Evidence pattern | Next bounded question |
|---|---|
| Correctable reports recur under one workload | Which link, transition and operating condition correlates with the recurrence? |
| Transaction fails but link remains usable | Which request, completion and device state explain the transaction error? |
| Fatal event leads to reset | Did every affected driver quiesce and restore its contract? |
| Only a link partner reports errors | What does the hierarchy show about where the error was observed? |
For a hypothetical endpoint failing only when exiting a low-power state, compare the first report and transition sequence across controlled runs. A correlation justifies reviewing firmware, driver sequencing and hardware conditions. It does not justify declaring a particular lane marginal without appropriate electrical evidence.
Close with two acceptance results
Report error diagnosis and recovery qualification separately. The first asks what initiated the event and how confidently it is located. The second asks whether the affected device and application returned to a defined state without stale DMA ownership, leaked resources or duplicate work.
Deliver raw AER logs, topology, counter epochs, callback timeline, exact software versions and untested conditions. A clean post-reset status can coexist with an unresolved initiating fault. Preserving the first error keeps recovery success from erasing the evidence needed to find it.
