TinyML Edge AI Offload: Confirm the Model Contains the Ethos-U Workload
Adding an Ethos-U build option does not turn an ordinary TinyML graph into an NPU workload. The model must contain the compiled custom operation, and the runtime must connect that operation to the real driver-backed implementation.
Diagnose the model and firmware as a pair. A missing compiler transformation, a stub registration and an incomplete board integration fail at different boundaries.
Look for the compiled workload in the deployed model
The pinned TFLM Ethos-U integration notes explain that Vela adds an ethos-u custom operator containing the accelerator workload. TFLM dispatches to Ethos-U when it encounters that operator.
Inspect the actual model embedded in the firmware, not only the input passed to Vela. It is possible to generate an accelerated output successfully and still compile the original model array into the application. Compare the graph's custom-operation inventory and the generated model artifact selected by the build.
Arm's Vela guide, covering Vela 5.1 and later, explains that supported TFLite regions become NPU custom operations while unsupported regions remain CPU operations. Use the installed version's --show-cpu-operations output and supported-operator report to understand that partition. A successful compilation does not imply complete NPU placement.
A successful AddEthosU call can still add nothing
The examined resolver helper calls Register_ETHOSU. If the returned registration is non-null, it adds the custom operator. Otherwise it returns kTfLiteOk without adding one.
That fallback matters because the non-Ethos stub returns null. Checking only the helper's status therefore does not prove that an accelerated model can resolve its custom operator.
Confirm which implementation was compiled. The integration notes use CO_PROCESSOR=ethos_u and an appropriate architecture selection; an external build system must reproduce the selected sources and dependencies. Keep registrations for CPU operations remaining in the compiled graph as well.
Trace the real operator into the driver
The pinned Ethos-U kernel exposes the custom name ethos-u. Preparation reserves scratch arrays for base addresses and sizes. Evaluation reads the command stream from the first input, gathers the other memory addresses, reserves a driver instance and calls ethosu_invoke_v3 before releasing it.
This is a driver-backed execution path, not an automatic conversion of each ordinary convolution at runtime. The platform must initialize and integrate the matching driver before inference. The README's example requires supported hardware or a suitable Corstone virtual platform.
The examined kernel also documents an important memory-layout restriction: its command stream accesses input/output feature maps through the arena base region, so simply pointing those tensors outside the arena is unsupported in that path. Do not generalize newer compiler options to this pinned implementation without checking compatibility.
Separate placement, successful execution and performance
For a controlled integration, establish three observations: the deployed graph contains the expected custom workload; the real registration and driver path are selected; and the complete mixed CPU/NPU graph produces acceptable output for fixed inputs.
Only then interpret performance measurements. Arm distinguishes Vela's cost-model estimates from measurements on the intended hardware. CPU fallback, memory placement and data movement can all affect end-to-end time even when some operations run on the NPU.
Keep the Vela version, accelerator configuration, model artifact and runtime/driver combination together when investigating a discrepancy. This article reports source behavior only: no Vela conversion, NPU execution or speedup was measured.
