TinyML Voice Triggers: Check Event Matching and False Alarms on Continuous Audio

Good accuracy on isolated speech clips does not establish acceptable behavior for an always-on voice trigger. Continuous operation adds event timing, missed commands, repeated detections and long intervals without a command. A release experiment needs to say how those events are matched and what each reported rate divides by.

TensorFlow 2.20.0 includes a useful host-side streaming evaluator, but its implementation has specific measurement conventions. Audit them before treating its printed percentages as product acceptance metrics. This is a source-based procedure, with no model inference or MCU test executed.

Establish the audio, model and timestamp contract

The versioned streaming evaluator loads a frozen TensorFlow GraphDef, slides an audio window through a WAV file and passes model outputs through its host recognizer. It is not a runner for an arbitrary quantized TFLite Micro firmware image. Its results therefore do not establish that the deployed frontend, integer model and application decoder behave identically.

There is also a documentation conflict worth catching before any experiment: the evaluator's preamble describes ground-truth times in seconds, while the actual ground-truth reader rounds the numeric value without converting seconds to milliseconds. The evaluator produces detection times in milliseconds, and the companion stream generator writes label timestamps computed in milliseconds.

For this pinned implementation, supply a millisecond event ledger. A row intended to mean one second must use a value corresponding to 1000 milliseconds; entering 1.0 is parsed near the beginning of the stream, not converted to 1000. Check a known event against its actual audio location rather than trusting a column description inherited from another tool.

Understand which annotation a detection can claim

The statistics code scans time-sorted ground truth for each found event. It skips annotations before the tolerance window, then uses the first annotation inside the window and stops searching. It does not find the closest same-label event or compute an optimal one-to-one assignment.

Consequently, overlapping tolerance windows can make an earlier annotation claim a detection that a reviewer would associate with a later command. A label mismatch is counted as wrong; the scan does not continue looking for a preferable match. Choose annotation spacing and tolerance deliberately, and inspect ambiguous cases rather than increasing tolerance until a headline metric looks better.

Repeated detections also need care. A first matching label can count as correct, while another detection matching the same annotation time counts as wrong. Nevertheless, both append that time to the matched list used for the reported matched percentage.

As a scorer-only fixture, consider one yes annotation at 1000 milliseconds and two yes detections at 1000 and 1600, with a 1000-millisecond tolerance. The implementation yields one correct and one wrong match, with two matched entries for one annotation. Its matched percentage is therefore 200 percent. This is a calculation from the code, not a recognition result, and demonstrates why that field is not unique-event recall.

Read the false-positive denominator

An event with no annotation in its tolerance window increments the false-positive count. The printed false-positive percentage divides that count by the number of ground-truth events, not by audio duration. If there are no ground-truth events, the implementation prints a raw false-positive count instead of a percentage.

Those outputs cannot be compared directly as false alarms per hour across recordings with different command density. Preserve the raw counts and evaluated duration. For a background-only acceptance test, declare the exposure interval and compute the intended time-based rate separately.

Also define how the product treats wrong-label and repeated events. They may cause unwanted actions even though the stock scorer places them outside its false-positive category. Keep misses, substitutions, duplicates and background triggers distinguishable in the event report; do not silently rename one existing counter to cover all of them.

Connect host scoring to the deployed acceptance claim

Exercise the scoring contract with small event ledgers before running a large corpus: a known timestamp, an unmatched detection, a duplicate and two nearby annotations expose different assumptions. Keep this separate from testing the recognizer's averaging or suppression algorithm.

The evaluator timestamps detections using the start of the current audio window. That timestamp is not a measurement of when a physical product finishes inference or performs an action. Define the timing reference before reporting latency, and obtain target-side timing separately when it is part of acceptance.

Finally, use representative continuous recordings and preserve both annotations and emitted events for inspection. A host experiment can assess a declared matching policy and model pipeline. A release claim about the MCU additionally needs evidence for the actual converted model, frontend, decoder and scheduling path. Report the boundary each experiment covers instead of collapsing those results into one accuracy number.