Designing Zephyr Inter-Core IPC: When Control and Bulk Traffic Should Share an Instance
For a two-domain embedded product, put control and bulk traffic on one Zephyr IPC instance only when their combined resource demand and receive processing meet the agreed service requirements. If control needs transport capacity and receive scheduling separated from bulk, evaluate distinct instances and budget both peers accordingly. Two endpoint names alone do not provide that separation.
Consider a hypothetical acquisition product with independently built application and acquisition firmware. It exchanges control commands while moving sample batches. The decision below concerns its inter-core communication architecture, assuming startup ownership is already resolved. It does not report a customer deployment or measured performance.
Define what the control path must complete
Before choosing a backend, specify whether control completion means reception, validation, execution, or an acknowledged result. Record its deadline and maximum outstanding requests. For bulk, specify the largest permitted burst, sustained offered load, acceptable loss, and whether production may be throttled. Include traffic in both directions.
The hardware review must identify shared RAM accessible to both domains, mailbox routing, cache requirements, and scheduling headroom on each core. A requirement for separate message addressing is much weaker than one for separately budgeted transport resources. Neither establishes protection from a faulty peer or independent reset behavior.
Pin the integration before pricing implementation effort. This analysis uses Zephyr v4.4.0 and its west manifest, which selects OpenAMP commit 5efe7974f9546582e99f5a842a816ea4b65f5227. A vendor fork, manifest override, or different linked module needs a fresh ownership review.
Map endpoints onto the resources that serve them
The IPC service interface distinguishes an instance from its logical endpoints and makes endpoint priority backend-dependent. In the examined RPMsg static-vrings implementation, each instance has an endpoint array, one RPMsg virtio device, one static-vring pair, and a receive workqueue on each participating domain.
Host initialization supplies one shared-memory pool for that instance. The pair contains distinct directional queues; endpoints sending in the same direction use that instance’s corresponding queue and available transport buffers. This does not mean RX and TX descriptors are an interchangeable global allowance. More endpoint slots add addressing capacity, not a reserved transport pool or worker per endpoint. Instance structures and initialization
Receive dispatch is equally important. The backend mailbox callback submits instance work. That worker calls the virtqueue notification path; OpenAMP’s receive loop then invokes the selected endpoint callback synchronously before continuing. A long bulk callback can therefore delay later dispatch on the same worker. This is a call-path deduction, not a measured delay or a numerical worst-case bound.
ipc_ept_cfg.prio is not consumed by this tagged backend’s endpoint registration or send path. Its devicetree zephyr,priority property instead selects the instance receive worker’s priority. Raising that priority affects the shared worker; it does not manufacture endpoint-specific capacity or execution contexts.
Choose the smallest topology that satisfies the contract
| Topology | When to consider it | Cost or constraint to accept |
|---|---|---|
| Two RPMsg endpoints on one instance | Joint traffic fits; bulk admission and callback work can be bounded | Shared directional capacity and receive dispatch; no endpoint reservation supplied here |
| Control and bulk on separate RPMsg instances | Separate transport budgets or receive workers are required | Additional regions, signaling allocation, worker stacks and instance metadata on both peers |
| Plain ICMsg | Both peers support its simpler protocol and required feature set | One endpoint per instance; application multiplexing still shares each directional FIFO |
A shared instance is a reasonable candidate when bulk can be admitted in bounded amounts and receive callbacks quickly hand work to appropriately bounded application processing. Specify the handoff’s queue capacity and rejection policy. Moving work out of a callback does not remove transport pressure or make an unlimited application queue acceptable. A producer policy must actually enforce the promised bulk allowance; naming an endpoint “control” reserves nothing.
Separate RPMsg instances partition the configured transport regions and give each instance its own receive worker. That partition requires nonoverlapping regions accessible to both domains and correctly routed mailbox signaling. They can also use different buffer sizes and worker priorities, with matching peer configuration. The official nRF5340 sample overlay illustrates two regions and distinct mailbox channel pairs. It demonstrates a topology, not universal SoC support or load isolation.
Budget the additional shared-memory layout, ring metadata, instance state, and architecture-adjusted worker stacks. Check the final linked allocations rather than multiplying endpoint count by a guessed buffer budget. Also account for BSP work: resource routing, matching peer changes, configuration maintenance, and regression coverage. These are project effort drivers; no generic price or RAM saving follows from this comparison.
Distinct workers still compete for their core’s execution time. Interrupts, memory buses, application locks, and power or reset domains can remain shared. Two instances are therefore not a latency guarantee, security boundary, or safety certification. If the requirement demands those stronger properties, the architecture review must extend beyond IPC topology.
Keep the ICMsg alternative deliberately narrow
The plain ICMsg documentation specifies one endpoint per instance, two directional memory regions, and mailbox signaling. The peers must agree on the layout, with transmit and receive regions and channels reversed appropriately. Cache alignment must meet both sides’ maintenance requirements.
Its backend operations table supplies registration, deregistration, and send, without the RPMsg no-copy operations exposed by the backend examined above. Application-level control/bulk tags inside one ICMsg endpoint still use the same directional FIFO. They do not create separate transport capacity.
Zephyr also defines separately named multi-endpoint ICMsg backends; the one-endpoint restriction here does not describe that entire family. Qualify any alternative against the actual peer protocol and required operations. Neither choosing RPMsg nor seeing a Linux-related OpenAMP example proves this static-vrings setup is a drop-in Linux remoteproc integration.
Make the architecture deliverable independently reviewable
Require a paired-domain transport map before freezing the BSP: endpoint names, instance assignments, host/remote roles, physical regions, direction, mailbox channels, buffer sizes, and cache/alignment assumptions. Include both images’ resolved devicetrees, effective configurations, source revisions, linker maps, and worker-stack allocations. The static-vrings binding specifically requires the buffer-size setting to agree between host and remote.
Then require a callback execution map and the actual admission rules. Identify which worker receives each endpoint, where application work continues, and which resources remain shared after any split. If two instances ultimately feed the same blocking application worker, disclose that downstream bottleneck rather than calling the control path independent.
A proposed acceptance fixture should compare the candidate topologies using the same declared workload. Check quiet control, maximum approved bulk bursts, slow consumption, and transport exhaustion. Preserve endpoint-to-instance identities, callback entry/return observations, send outcomes, and application completion records. Report useful bulk delivery alongside all control attempts and missed deadlines, so starving bulk cannot masquerade as success.
Correlate cross-domain observations using a documented clock relationship or an external observer; callback entry alone is not command execution. Retain the exact two-image combination and resource map with the results. An observed maximum does not establish an absolute bound.
No firmware build or traffic experiment was performed here. Accept the shared design if it meets the joint contract with enforceable admission and bounded processing. Choose separate instances when their identified resource separation is needed and feasible. The handover should explain that decision, its verified resource costs, and the bottlenecks still requiring qualification.
