Voice Activity Detection

How to use the Voice Activity Detection robot skill

The Voice Activity Detection (VAD) skill listens to your microphone in real time and fires scripts when speech begins or ends. It uses FFT (Fast Fourier Transform) analysis on the 300-3400 Hz speech band to distinguish voice from background noise, filtering out short impulsive sounds like knocks and clicks.

Typical use: trigger a speech recognition skill when someone starts talking, and stop it when they go silent.

Key Features:

• Speech Detection:
  • Detects when speech starts (Speech Begin) and stops (Speech End).
• Customizable Actions:
  • Allows users to attach custom scripts that execute automatically when speech starts or stops. For example, you can trigger robot movements, lights, or other interactions based on speech activity.
• Real-Time Audio Visualization:
  • Displays a live graph of the detected speech level, giving a visual representation of the audio activity.
• Adjustable Sensitivity:
  • Includes settings to fine-tune detection parameters, such as silence thresholds, for optimal performance in various environments.

Practical Applications:

• Interactive Robots:
  • Enable your robot to react to speech dynamically, such as greeting people when they start talking or pausing when they stop.
• Hands-Free Control:
  • Use speech detection to trigger actions without needing additional input devices.
• Speech Analysis:
  • Visualize audio levels for debugging or fine-tuning robot behavior in different environments.

This robot skill combines robust speech detection with seamless integration into Synthiam ARC, empowering your robot to understand better and respond to its surroundings!

Main Window

Voice Detected The graph will display green when the voice is detected, and the voice start script will be executed.

Voice Not Detected The display will be red when a human voice is absent, and the voice end script will execute.

Pause You can pause the detection of speech by pressing the pause checkbox on the main form. The PAUSE is also enabled when any of the Audio.say() scripting commands are used and unpaused when the speaking has been completed. This ensures the VAD does not detect the robot speaking and triggers a false positive.

Graph

A real-time level meter shows the detected audio energy relative to the sensitivity threshold:

The graph bar fills from 0-100% as energy approaches and exceeds the threshold.

Pause Checkbox

Checking Pause suspends all voice detection. The graph stops updating and no scripts will fire. Uncheck to resume.

The plugin automatically pauses while the robot's text-to-speech (TTS) is speaking, then restores your previous pause state when TTS finishes. This prevents the robot's own voice from triggering detection.

Log Tab

Displays diagnostic output. Every ~50 audio frames (~2.5 seconds) a line is written:

[VAD] energy=0.000042  threshold=0.000050  ratio=0.840

Use these values to calibrate Sensitivity - the energy and threshold should be close together for good triggering.

Configuration

Open configuration by clicking the gear/config button on the skill panel.

Sensitivity

Default: 0.00005

The FFT energy threshold for speech detection. The VAD compares the smoothed energy of the 300-3400 Hz band against this value.

• Lower value more sensitive (triggers on quieter sounds)
• Higher value less sensitive (requires louder speech to trigger)

Use the diagnostic log to tune this. If energy is consistently below threshold when speaking, lower the value. If background noise is triggering false detections, raise it.

Silence Duration

Default: 500 ms

How long (in milliseconds) silence must persist after speech before the Speech End event fires. Increase this if the script fires too early during natural pauses in speech. Decrease it for faster response after someone stops talking.

Minimum Speech Duration

Default: 150 ms

How long (in milliseconds) energy must continuously exceed the threshold before Speech Begin fires. This filters out short impulsive sounds (clicks, knocks, coughs) that are not sustained speech. Increase this to be more aggressive about ignoring non-speech sounds.

Scripts

Both scripts are configured inside the Config dialog.

Speech Start Script

Runs once when speech begins (after energy exceeds Sensitivity for at least Minimum Speech Duration milliseconds).

Typical use:

// Start a speech recognition skill
ControlCommand("Speech Recognition", "Start Listening");

Speech End Script

Runs once when silence has persisted for the Silence Duration after speech was active.

Typical use:

// Stop the speech recognition skill
ControlCommand("Speech Recognition", "Stop Listening");

How Detection Works

1. Microphone audio is captured at 16 kHz, 16-bit mono, in 50 ms chunks.
2. Each chunk is optionally filtered with a high-pass BiQuad filter (cutoff 300 Hz) to remove low-frequency rumble.
3. Samples accumulate into a 1024-sample FFT buffer. When full, a Hamming window is applied and the FFT is computed.
4. The average magnitude of all FFT bins in the 300-3400 Hz speech band is calculated.
5. That energy value is exponentially smoothed (factor 0.3) to reduce frame-to-frame jitter.
6. The smoothed energy is compared to Sensitivity:
   • If above threshold for Minimum Speech Duration SpeechBegin fires.
   • If below threshold for Silence Duration after speech was active SpeechEnd fires.

Tuning Guide

Robot is not triggering on speech

1. Check the diagnostic log. Note the energy value while speaking.
2. Lower Sensitivity until it is slightly below the energy value when speaking.
3. If still not triggering, ensure the correct microphone is selected in Windows Sound settings.

Too many false triggers (background noise)

1. Note the energy value during silence in the diagnostic log.
2. Raise Sensitivity to just above that noise floor value.
3. Increase Minimum Speech Duration (e.g., 200-300 ms) to require more sustained sound.

Speech End fires too quickly (mid-sentence pauses)

Increase Silence Duration (e.g., 800-1500 ms).

Speech End fires too slowly after speaking stops

Decrease Silence Duration (e.g., 300 ms).

Robot triggers on its own voice

This is handled automatically - detection pauses when TTS is active. If issues persist, verify that the TTS events are wired correctly (the plugin uses EZBManager.PrimaryEZB.SpeechSynth).

Technical Specifications

Control Commands for the Voice Activity Detection robot skill

There are Control Commands available for this robot skill which allows the skill to be controlled programmatically from scripts or other robot skills. These commands enable you to automate actions, respond to sensor inputs, and integrate the robot skill with other systems or custom interfaces. If you're new to the concept of Control Commands, we have a comprehensive manual available here that explains how to use them, provides examples to get you started and make the most of this powerful feature.

Control Commands

Other skills or scripts can control VAD behavior using ControlCommand().

Example from another skill's script:

// Pause VAD while playing audio
ControlCommand("Voice Activity Detection", "Pause");

// ... do something ...

// Re-enable VAD
ControlCommand("Voice Activity Detection", "Resume");