TinyML Speech Releases: Keep Calibration and Final Evaluation in Separate Data Partitions

A TinyML speech model's reported test accuracy does not automatically describe an untouched release holdout. Check whether the conversion process already consumed inputs from the same partition. Quantization calibration is not supervised training on those labels, but it is still a data-dependent step in producing the deployed artifact.

The inspected Micro Speech training notebook provides a concrete example. This article proposes a release-evaluation design around that example; no model was trained or scored, and no accuracy inflation is asserted.

Read both consumers of the testing partition

In the pinned TFLM notebook, the representative-data generator makes 100 single-example calls to AudioProcessor.get_data() with mode testing. Later, its inference helper loads the testing partition to report accuracy for the exported model. The calibration generator discards the returned labels, whereas the evaluation helper uses labels to count correct predictions.

This is not a converter API violation. TensorFlow's RepresentativeDataset documentation describes input samples used to estimate quantization ranges and permits sampling from training or evaluation data. The narrower issue is the product claim: a partition that helped construct the converted artifact should not also be described as completely untouched by development.

Version boundaries matter here. The pinned notebook clones TensorFlow without a fixed revision. The TensorFlow 2.20.0 implementation discussed below is separately pinned for inspection; it is not claimed to be the dependency that the notebook would resolve today. A reproducible product workflow must select its own compatible versions.

Define the final set before choosing calibration samples

Maintain a development pool for fitting, model selection and calibration, plus a final release set kept outside those decisions. Calibration can use a suitable subset of development inputs; this design does not require a separate fourth partition simply for its name.

Document the membership and role of each collection. A final set should contain the product conditions being evaluated, including the intended speakers, microphones, environments and command classes. Keep those requirements distinct from merely obtaining enough samples to run conversion.

The separately inspected TensorFlow training script evaluates validation data during training and testing data at the end. A notebook's later conversion and comparison steps add another use of data after that training loop. Audit the entire pipeline rather than assuming that an earlier function's testing label protects the partition from all later consumers.

Speaker grouping depends on the naming contract

In TensorFlow 2.20.0 input_data.py, which_set() hashes the basename after removing the _nohash_ suffix and everything following it. With unchanged proportions, adding other files does not change an existing name's assignment. Related files sharing that prefix therefore remain together.

That mechanism cannot infer speaker identity from arbitrary renamed recordings. Audit speaker or recording-group identifiers before importing a private corpus. Verify that related clips are assigned together across labels, and check content duplicates separately. A stable filename hash is useful machinery, not proof that the corpus obeys its grouping convention.

For a release set, compare group membership with the development/calibration pool. If the question is generalization to unseen speakers, clip-level separation alone cannot establish the intended experiment. Resolve the grouping criterion before looking at which split produces a better score.

Generated silence requires more than a path comparison

The inspected loader inserts silence records using a training-file path, but later multiplies that foreground audio by zero. Seeing that path in multiple partitions therefore does not by itself show that its speech waveform leaked into evaluation.

Instead, follow the generated sample. Silence uses randomly selected background audio, crop and volume. Testing can also receive a random time shift when a nonzero shift is requested. Stable file membership consequently does not guarantee identical feature tensors between runs.

Freeze the final evaluation's transformation policy and its generated examples, or record enough inputs and random-state control to reproduce them under the pinned implementation. Track background-recording provenance where independence from development environments is part of the product claim. Do not mistake a muted placeholder path for the actual acoustic content.

Report what the release experiment actually establishes

Freeze the converted model before running the reserved final set. Evaluate float and quantized artifacts on the same prepared inputs when comparing conversion loss. Report class-level failures alongside the aggregate result so that an acceptable overall score cannot hide a weak command or silence class.

If the team changes calibration selection, thresholds or preprocessing after examining final-set failures, treat that set as development evidence for the next iteration. Reserve a fresh untouched evaluation for a new release claim. The objective is an honest, repeatable assessment of the exact MCU-bound model, not a particular percentage or a declaration that the original example is defective.