TinyML Numeric Regression: Match CMSIS-NN Rounding Options Before Comparing Outputs

An optimized int8 model produces the same classification as the reference build, but several output bytes differ. Accepting the result because the top label matches can hide a regression. Rejecting it immediately as a broken kernel can also miss a configured arithmetic difference.

CMSIS-NN exposes requantization options that affect how integer accumulators become output values. Establish the intended numerical contract before changing tolerances or regenerating expected results.

Identify the reference implementation precisely

The CMSIS-NN project documentation targets agreement with TensorFlow Lite reference arithmetic, following TFLM when the desktop and micro reference implementations differ. Therefore, “the TensorFlow result” is not a sufficiently precise oracle.

The unit-test documentation identifies LSTM and SVDF as cases requiring the TFLM interpreter for the stated comparison. Use the reference appropriate to the operator, its revision and the options being tested. Record whether expected bytes came from that reference or from a previously deployed optimized binary.

Keep model bytes, quantized input, tensor layout and quantization parameters identical across the comparison. Otherwise, a numerical difference at the output cannot isolate kernel arithmetic.

Single rounding changes an intermediate calculation

The pinned requantization helpers contain separate branches selected by CMSIS_NN_USE_SINGLE_ROUNDING. The ordinary branch performs a rounded high multiplication followed by rounded division by a power of two. The single-rounding branch combines the rescaling before its final rounding.

A small positive example exposes the difference without involving model accuracy: take accumulator 1, multiplier 1073741824 and shift -1. The ordinary helper rounds the high multiplication to 1, then rounds its division by two to 1. The single-rounding calculation corresponds to one quarter and produces 0.

This example is arithmetic derived from the examined scalar helpers. It is not a target test or a claim that every tensor element differs by exactly one. Accumulation, activation limits and subsequent operations determine how a local discrepancy propagates through a real graph.

Match options where the arithmetic is compiled

The project documentation flags single rounding as affecting headers and instructs integrations to enable the equivalent option in TFL/TFLM. Changing only an application-level definition does not retroactively rebuild a precompiled CMSIS-NN archive. Conversely, an archive rebuilt with one choice can disagree with inline code compiled elsewhere using another.

Inspect compile definitions on both sides and remove stale objects before making a controlled comparison. Header selection, compiler version and target architecture belong with those definitions. Do not infer the option from a library filename.

The separate CMSIS_NN_USE_REQUANTIZE_INLINE_ASSEMBLY option is documented as intended to preserve results, but the README reports observed differences for Arm Compiler with Cortex-M7. Treat that note as a reason to test the actual toolchain combination, not a blanket permission to ignore mismatches.

Localize the first unequal value before adjusting acceptance

Start with a fixed quantized input and compare complete output tensors. If they differ, instrument a development build to locate the earliest divergent operation. Capture its input bytes, weights, bias, per-channel multipliers and shifts, then reproduce that operation independently with the intended reference configuration.

The convolution implementation receives per-channel quantization parameters separately from tensor data. A mismatch in those arrays can resemble a rounding change; compare them before blaming the selected helper.

If bit-exact agreement is required, aligned configuration must pass byte-level comparison. If a product deliberately permits numerical variation, define that criterion in advance and evaluate affected decisions, especially near thresholds. Regenerating a golden output merely because the optimized build changed would erase the very evidence needed to distinguish an intended arithmetic choice from a defect.