Join the early list

Wake-word and stop-phrase detection for VAHA: dataset, validation accuracy and first field check

· Ritesh Bonthalakoti

  • On-device AI
  • TRL 3
Read the paper Download

PDF, 17 pages, 0.8 MB · v1.0

The paper's cover: Wake-word and stop-phrase detection for VAHA: dataset, validation accuracy and first field check

Summary

VAHA is Dharanova's offline voice-capture device. It starts recording when it hears the wake word "Marvin" and stops on a stop phrase (class im_done) or on silence. Detection uses an Edge Impulse keyword-spotting model, MFCC features and a Keras classifier, that runs on an Arduino UNO Q. This paper reports what the model was trained on and how well it classifies. The training project holds 8,466 one-second samples in three classes, split 80/20 into training and test. The marvin class comes from the public Google Speech Commands dataset and bg was imported from another Edge Impulse project; im_done was recorded for VAHA. On the platform's validation split the int8 model scored 97.6 % (95 % interval 96.6–98.3 %, calculated), below the 98.7 % quoted earlier for this model. Three unseen test samples, one per class, were all classified correctly with over 99.9 % confidence. A 34-trial check in a bathroom passed every trial, but individual trials were not logged. Every measured accuracy so far comes from the cloud platform, not from the shipped model on the device. The work is an experimental proof of concept (TRL 3).

Key findings

  • Accuracy. 97.6 % on the validation split, int8 model (96.6–98.3 %, calculated). Recall: bg 94.7 %, im_done 99.3 %, marvin 97.4 %.
  • Earlier figure. The 98.7 % quoted earlier for this model lies outside that interval. The run it came from has not been identified.
  • Dataset. 8,466 samples. marvin is public Google Speech Commands data and bg was imported from another project; im_done was recorded for VAHA.
  • Unseen samples. 3 of 3 correct at 99.9 % or more, 19.9 to 25.0 points above the firmware thresholds.
  • Field check. 34 of 34 passed in a bathroom, without a per-trial log. Logged, the same result would show at least 91.6 %.
  • Readiness. TRL 3. Next: the shipped model on the UNO Q over the 1,689 test samples.

Inside the paper

6 of the paper's 9 figures. Open any of them full size to zoom in.

Figure 2: The VAHA audio hardware with the enclosure open.Photo of the open enclosure on a desk, numbered: 1, the speaker, with a PAM8403 amplifier and driver, for chime and speech output and not in the detection path; 2, the Arduino UNO Q, which runs new-marvin.eim and the wake and stop logic; 3, the 3.5 mm microphone input jack, part of the audio input path; 4, the USB lavalier microphone, the input the model listens on, 48 kHz mono. THE AUDIO HARDWARE, ENCLOSURE OPEN 1 2 3 4 1 Speaker PAM8403 amplifier and driver; chime and speech output, not in the detection path 2 Arduino UNO Q Runs new-marvin.eim and the wake and stop logic 3 3.5 mm microphone input jack Part of the audio input path 4 USB lavalier microphone The input the model listens on, 48 kHz mono
Figure 2 The VAHA audio hardware with the enclosure open.
Figure 1: The detection path. Audio is classified in 1,000 ms windows, a new one every 255 ms (3.92 per second, calculated). A single window above threshold is enough to start or stop recording.Flow diagram of the detection path on the device. The USB lavalier microphone, 48 kHz mono, feeds a 1,000 ms window, a new one every 255 ms; MFCC spectral features of the window; a Keras classifier with three classes; and a confidence per window for marvin, im_done and bg. The model input is 16 kHz, and the resampling step is not yet characterised. Wake: marvin at 0.75 or more in one window (WAKE_CONSEC = 1) starts recording. Stop: im_done at 0.80 or more in one window (6-frame window), or silence, stops recording. On the device this runs as new-marvin.eim on the Arduino UNO Q, called by the VAHA backend. DETECTION PATH ON THE DEVICE Microphone USB lavalier, 48 kHz mono Window 1,000 ms, a new one every 255 ms MFCC spectral features of the window Classifier Keras network, 3 classes Confidence marvin, im_done, bg per window model input is 16 kHz; the resampling step is not yet characterised Wake marvin ≥ 0.75 in one window WAKE_CONSEC = 1 Start recording Stop im_done ≥ 0.80 in one window (6-frame window), or silence Stop recording On the device this runs as new-marvin.eim on the Arduino UNO Q, called by the VAHA backend
Figure 1 The detection path. Audio is classified in 1,000 ms windows, a new one every 255 ms (3.92 per second, calculated). A single window above threshold is enough to start or stop recording.
Figure 4: The dataset by class and split, with what is known of each class's source.Bar chart of 8,466 samples by class, split into training and testing. marvin, 2,096 samples: public Google Speech Commands, identified from filenames. im_done, 3,148: recorded for this project and stored in it. bg, 3,222: imported from another project, original source not yet traced. DATASET BY CLASS AND SPLIT, 8,466 SAMPLES Training Testing 0 500 1,000 1,500 2,000 2,500 3,000 3,500 Samples marvin 2,096 Public Google Speech Commands identified from filenames im_done 3,148 Recorded for this project stored in this project bg 3,222 Imported from another project original source not yet traced Provenance
Figure 4 The dataset by class and split, with what is known of each class's source.
Figure 5: Validation confusion matrix for the int8 model, row-normalised. The diagonal is each class's recall.Three-by-three matrix of true class against predicted class, each row summing to 100 %. bg: 94.7 % bg, 3.3 % im_done, 2.0 % marvin. im_done: 0.7 % bg, 99.3 % im_done, 0.0 % marvin. marvin: 1.9 % bg, 0.6 % im_done, 97.4 % marvin. F1 scores: bg 0.96, im_done 0.99, marvin 0.97. VALIDATION CONFUSION MATRIX, INT8 MODEL, ROW-NORMALISED Predicted class bg im_done marvin bg 94.7 % 3.3 % 2.0 % im_done 0.7 % 99.3 % 0.0 % marvin 1.9 % 0.6 % 97.4 % F1 score 0.96 0.99 0.97 True class Correct (recall) Misclassified Each row sums to 100 %
Figure 5 Validation confusion matrix for the int8 model, row-normalised. The diagonal is each class's recall.
Figure 6: What each piece of evidence can show about accuracy at 95 % confidence (calculated). The field-check bound treats the 34 passes as logged results.Interval chart of classification accuracy from 30 to 100 %, with the 98.7 % reported in the product README marked. Platform validation, int8 model, about 1,355 samples: 96.6 to 98.3 %. Live classification, 3 of 3 correct in the cloud: at least 36.8 %. Bathroom field check, 34 of 34 passed, summary only: at least 91.6 %. WHAT EACH PIECE OF EVIDENCE CAN SHOW, AT 95 % CONFIDENCE 30 % 40 % 50 % 60 % 70 % 80 % 90 % 100 % Classification accuracy (calculated bounds) 98.7 % reported in product README Platform validation int8 model, about 1,355 samples 96.6–98.3 % Live classification 3 of 3 correct, cloud at least 36.8 % Bathroom field check 34 of 34 passed, summary only at least 91.6 %
Figure 6 What each piece of evidence can show about accuracy at 95 % confidence (calculated). The field-check bound treats the 34 passes as logged results.
Figure 8: Path from this work to TRL 4.Five steps in sequence. 1, set limits first: accuracy, missed wakes and false wakes per hour, fixed before testing. 2, pin the model: record which run, and int8 or float32, ships as new-marvin.eim. 3, test what ships: run new-marvin.eim on the UNO Q over all 1,689 test samples. 4, log field trials: a per-trial log, at least 59 trials per phrase, plus hours of bathroom noise. 5, provenance: source and licence for every class, bg included. These lead to TRL 4, the shipped detector validated in the lab against defined criteria. Then: define the relevant environment (bathroom noise, echo, distance) and test there (TRL 5). PATH FROM THIS WORK TO TRL 4 1 Set limits first Accuracy, missed wakes and false wakes per hour, fixed before testing 2 Pin the model Record which run, and int8 or float32, shipped as new-marvin.eim 3 Test what ships Run new-marvin.eim on the UNO Q over all 1,689 test samples 4 Log field trials Per-trial log, at least 59 trials per phrase, plus hours of bathroom noise 5 Provenance Source and licence for every class, bg included TRL 4: the shipped detector validated in the lab against defined criteria Then: define the relevant environment (bathroom noise, echo, distance) and test there (TRL 5)
Figure 8 Path from this work to TRL 4.

About this paper

Version
v1.0, published 21 October 2026
Written by
Ritesh Bonthalakoti
Reviewed by
Bharat Meda
Readiness
TRL 3: experimental proof of concept
Tested on
An Edge Impulse keyword-spotting model (MFCC features, Keras classifier, three classes), on the platform's validation split and in live classification; the shipped model on an Arduino UNO Q with a USB lavalier microphone, in one bathroom check that was not logged
Standards
None for keyword-spotting accuracy; ISO/IEC/IEEE 29119-3 loosely for structure; no accredited testing
The paper
17 pages, 0.8 MB, PDF

The full paper has the method, every table, the defects found along the way and what is still needed for the next readiness level.

Read the paper

Related reading