Zephyr OS RTIO Buffers: Release the Completion Slot and Payload Separately

An RTIO consumer can drain every completion entry and still exhaust its receive-buffer pool. The completion record and the data allocation describe different resources. Returning one does not automatically return the other.

For a Zephyr OS application using pool-backed receive operations, diagnose ownership at both boundaries. This discussion is anchored to Zephyr v4.4.0 source. The examples are proposed review and test scenarios, not measurements from a deployed device.

Identify the two things being retained

The tagged RTIO public header exposes separate completion and buffer APIs. A consumed CQE must eventually be returned with rtio_cqe_release(). That operation returns the completion object to its pool. A successful pool-backed read can additionally expose its allocation through rtio_cqe_get_mempool_buffer(). The allocation is returned separately with rtio_release_buffer() after its consumer has finished.

Capture the validated buffer pointer and allocation length before giving up the CQE. Check the buffer-extraction result rather than assuming every completion contains pool data. A failed operation, a non-pool request and a usable pool-backed receive are different cases.

For application design, draw two ownership lines: “CQE held by dispatcher” and “payload held by decoder.” They may end at different times. A decoder queued to another worker cannot safely keep a pointer into an allocation that the dispatcher has already recycled.

Avoid turning a copied notification into a copied payload

The same header's rtio_cqe_copy_out() copies completion records and releases their original slots. It does not turn the referenced receive allocation into an independent copy of the sample bytes. Similarly, flushing completion records is not a general payload-reclamation strategy.

A hypothetical sensor pipeline illustrates the distinction. A high-priority collector copies completion metadata into a work item, releases the completion slot and hands the work item to a lower-priority decoder. That can be a useful separation if the payload remains explicitly owned until decoding finishes. It leaks allocations if nobody returns them, and it risks reuse corruption if the collector returns them before decoding.

Choose one policy for that handoff. Either retain the original allocation with exactly one release owner, or copy the valid application data into separately owned storage and then return the original allocation. A copied pointer alone implements neither policy.

Do not apply the success-path cleanup to every result

The tagged executor implementation treats one-shot and multishot operations differently. In the multishot path, cancellation or failure releases the current receive buffer and submission entry; a successful pool-backed receive resets the submission's buffer information before resubmission. Response generation is also conditional on request flags.

Consequently, do not write a generic error callback that unconditionally frees whatever pointer appears related to a request. Review which path owns that allocation when the error is reported. Otherwise a cleanup routine intended to prevent leaks can introduce a second release.

Keep the first implementation simple: document supported operation types, whether they are multishot, and how their successful payloads reach the final consumer. Expand to cancellation only after tracing the corresponding source and device implementation. There is no universal “any callback means all resources are mine” rule.

Test depletion and delayed consumption separately

Use a deliberately small pool in a disposable test configuration. First delay payload processing while continuing to consume completions. Observe whether allocation pressure follows the number of retained payloads. Then delay CQE consumption while allowing already-owned payloads to finish. The two experiments target different capacity limits.

Add failure injection only after recording the expected owner at each transition. Include a decoder rejection, a consumer shutdown and a multishot cancellation. Check that each live allocation has one eventual release and that no worker accesses it afterward. These are suggested tests; their outcome depends on the selected device driver and operation.

A larger pool may postpone a failure, but it does not establish correct ownership. The useful fix is an explicit resource handoff that explains both completion-slot recovery and the lifetime of the bytes being processed.