Append-Only Sensor Logs: Find the Last Valid Record After an Interrupted Write

Recover an append-only log by accepting a contiguous prefix of fully validated records and stopping at the first incomplete or invalid frame. Return both the accepted records and the exact stop offset. Do not search for the next familiar marker and silently join data across an unexplained gap.

This article defines an original, synthetic framing model and its recovery oracle. It is not a littlefs on-disk format, a Zephyr NVS decoder, or a production-ready raw-flash writer. The host fixture checks parser behavior; actual persistence, erase geometry, concurrency and power-loss behavior require separate target qualification.

1. State the failure model before choosing markers

The simplest useful model is a serialized append stream whose previously accepted prefix remains readable, while its newest append may be incomplete. Under that model, recovery should preserve complete earlier records and reject the unfinished tail. If earlier bytes can be corrupted too, the same parser stops earlier and reports the reason rather than promising recovery of every later record.

Distinguish a valid byte representation from durable acknowledgement. The littlefs v2.11.2 file API says file writes are not updated on storage until synchronization or close. A trailer in a host byte string therefore does not demonstrate that a real device persisted it in the intended order.

Choose one application writer or serialize entire records. Two tasks interleaving header, payload and trailer writes can produce a damaged stream even without power loss. A lock around each individual write call is insufficient if a logical record spans several calls.

2. Define a bounded illustrative frame

The fixture uses the following invented layout. All multi-byte integers are little-endian. Payload length is limited to 1 through 64 bytes for this exercise, and sequence numbers start at one. These values are teaching choices, not hardware requirements.

Field	Size	Validation
Magic SL.	2 bytes.	Exact match at the expected frame start.
Format version.	1 byte.	Exactly 1.
Payload length.	2 bytes.	Within the configured bound before allocating or advancing.
Sequence.	4 bytes.	Next expected value within this synthetic segment.
Payload.	Declared length.	Present in full; interpretation remains application-specific.
CRC-32.	4 bytes.	Matches header plus payload.
Completion trailer C3 5A.	2 bytes.	Exact bytes at the computed frame end.

The host runner uses Python’s zlib.crc32, whose Python 3.12 documentation describes an unsigned checksum result and warns that it is not cryptographically strong. This checksum detects many accidental changes; it does not authenticate records or prevent a deliberate rewrite with a new checksum.

3. Make the parser’s stopping rule explicit

1. If the cursor equals the captured byte length, return a clean end.
2. If fewer than nine header bytes remain, return an incomplete-header result at the current cursor.
3. Validate magic, version, length and expected sequence before trusting the frame size.
4. Check that the complete payload, checksum and trailer fit in the remaining input.
5. Validate the checksum and exact trailer.
6. Only then append the record to the recovered result and move the cursor to its end.

For a fixed-width implementation, calculate bounds without integer overflow. Comparing the required remaining size against available bytes is safer than blindly adding attacker-controlled lengths to a pointer. Bound the total input and recovered-record count as well as each individual payload.

Return a reason such as incomplete header, invalid header, sequence discontinuity, incomplete record, checksum mismatch or missing trailer. Keep that reason separate from the last accepted offset. A caller can then show what was recovered without claiming that the remainder was merely empty.

4. Do not turn a marker search into recovery proof

The magic and trailer bytes may appear inside a valid payload. Searching forward for them after a failure can select payload bytes as a false frame boundary. Even a well-formed later record does not explain the missing interval or establish a contiguous history.

A product may implement a separate forensic salvage tool with stronger segment indexes or independently validated boundaries. Label its output as discontinuous evidence with gaps. Do not silently feed salvaged fragments through the ordinary contiguous-prefix replay path.

Sequence policy also needs a real segment design. The fixture’s starting value of one does not solve reboot identity, counter wrap, ring reuse or multiple sources. A production segment header and generation policy must define those boundaries before sequence checks can carry meaning.

5. Use an independent prefix oracle

The executed host fixture creates three records with payload lengths 1, 4 and 64 bytes. The middle payload deliberately contains magic and trailer bytes. Their frame lengths are 16, 19 and 79 bytes, so the complete stream is 114 bytes and its accepted-end offsets are 16, 35 and 114.

For each of the 115 possible truncation positions, the oracle accepts exactly those complete frames whose known end offset is at or before the cut. This expected answer comes from the fixture construction, not from asking the decoder to validate itself.

The runner also flips one low bit in every byte position, one case at a time. Each of these 114 cases must stop at the affected frame and retain only earlier frames. Five further cases cover a valid-checksum sequence discontinuity, an empty stream, maximum payload and two rejected encoder lengths.

Observed host result: all 234 cases passed using Python 3.12.14. This is a bounded parser-model result, not exhaustive corruption coverage and not a littlefs, flash-driver or power-cut test. The self-contained runner and its recorded output accompany the offline article package.

6. Preserve the original tail before any repair

Run recovery against an acquired copy first. Do not truncate the original immediately at the returned offset: an invalid interior frame may require investigation, and later bytes may matter to forensic analysis. Decide separately whether the product should start a new segment, retain the damaged one, or perform an authorized repair.

Target qualification must include short writes, synchronization failures, storage-full errors, resets around each append stage, and the exact driver’s persistence contract. A complete structural record still needs application validation before its sensor value is used.

Hypothetical torn-tail result

A captured fixture ends at byte 80, partway through the third frame. Recovery returns the first two records and stop offset 35 with an incomplete-record reason. It does not discard the first two records, invent the third payload, or search the remainder for a lucky marker.

That precise output is the deliverable: accepted prefix, rejected boundary and explicit uncertainty. It turns an ambiguous “log recovered” message into a testable local-storage contract.