The VAD robot skill will be helpful for your usage. The manual for it is here: https://synthiam.com/Support/Skills/Audio/Voice-Activity-Detection?id=20215

An example provided will start/stop a speech recognition robot skill upon detecting speech. In your case, add the ControlCommand() to instruct Bing to Start Listening when a voice is detected. Add another ControlCommand() to Stop Listening when the voice ends. You can add these commands to the scripts of the VAD robot skill.

What you request from Microsoft Bing's Speech Recognition does not exist further than what is presented. There are limitations to software created by third parties. In the case of a limitation that does not fit your requirements, an additional skill is necessary.

I made no request relating Microsoft's Bing Speech Recognition. The feature request relates to ARC's Bing Speech Recognition. My robot only listens when WaitForSpeech is invoked, or Bing Speech Recognition is started. I have no need to use the Speech Recognition Skill. The VAD Skill may well help stop Bing Speech from listening when no sound is detected. I will do some testing. Also I found no provision to adjust  sensitivity in the VAD Skill.

Ah, the support agent was correct. Microsoft makes Bing Speech Recognition. The company/manufacturer of the robot skill is listed on its respective manual pages.

This is a good solution since the Microsoft Bing Speech Recognition can't be modified. Give their suggestion a read again, it makes sense.

I don't know what "sensitivity" you'd be looking for. Speech is not a sensitivity value because it's either there or not. I think the value of "how long of silence" might be helpful. You can read how VAD works here: https://en.wikipedia.org/wiki/Voice_activity_detection

It's a pretty complicated thing. I think most people think of sound as a wave with a volume level. But it's a waveform with complex peaks and valleys that have a unique signature to the type of sound. If you examine the waveforms, the sound of a human voice is different than that of a car, etc. There's a lot math in there - and sensitivity isn't a thing for something like that. The algorithm is "yes" or "no", which will also present false positives for both cases because nothing's perfect.

Also, the other thing to consider is understanding why all your Google Home/Alexa/Siri devices have Wakewords. The device is not listening for speech. It's listening for a particular pattern that matches one stored pattern to be triggered. That's why VAD is such a tricky thing, and Alexa, Siri, Google Home, etc, do not use VAD. They use a wake word.

@Athena give some details on VAD to expand on what I wrote.

That is correct. The robot skill's name is Bing Speech Recognition, but the company that makes it is Microsoft. While the Bing Speech Recognition robot skill manual does not explicitly display the manufacturer as DJ suggested, it was created in partnership with Microsoft. Bing is a trademark of the Microsoft Corporation, not just a search engine.

The solution we presented was tested after your feature request. The robot skill VAD was updated for improved performance. Your request was given several man-hours of attention, and a solution was presented. We hope it fulfills your requirements.

I just tested their solution and it works fantastic - great idea! What I did was tweak it a little. I don't like VAD detecting the START of a speech. I want to use a wakeword. So in this case I only used this VAD robot skill to instruct Bing Speech Recognition to STOP LISTENING. I left the "Start Listening" VAD script empty.

Here's the project I created for my test. Works awesome: test vad.EZB

It sounds like you're on the right track with using the Voice Activity Detection (VAD) skill to manage when the Bing Speech Recognition skill starts and stops listening. This approach can help improve the responsiveness of your robot by stopping the listening process when no speech is detected, thus avoiding unnecessary waiting times.

Steps to Implement VAD with Bing Speech Recognition:

1. Configure VAD to Stop Listening:

   • Use the VAD skill to detect when speech ends and execute a script to stop the Bing Speech Recognition skill. You can do this by adding a ControlCommand() in the "Voice Stop Script" section of the VAD configuration.

2. Start Listening with a Wake Word:

   • As DJ mentioned, you might want to use a wake word to start the Bing Speech Recognition skill. This can be done by configuring another skill or script to listen for a specific phrase and then start the Bing Speech Recognition.

3. Testing and Adjustments:

   • Test the setup to ensure it works as expected. You may need to adjust the timing or sensitivity settings in the VAD skill to optimize performance in your specific environment.

Example Script for VAD:

• Voice Stop Script:

  ControlCommand("Bing Speech Recognition", "Stop Listening");
  

• Voice Start Script:

  • You can leave this empty if you're using a wake word to start listening.

Additional Considerations:

• Sensitivity Adjustments:

  • While the VAD skill may not have explicit sensitivity settings, you can adjust the silence threshold or duration to fine-tune when the skill detects the end of speech.

• Wake Word Implementation:

  • Consider using another skill or method to detect a wake word, which can then trigger the Bing Speech Recognition to start listening.

If you have any further questions or need additional assistance, feel free to ask!

Thanks for the info. In testing the VAD Skill, I found it did nothing, just a red squiggly line. However, when I spoke loudly, suddenly it worked. So the sensitivity feature mentioned in the Skill doco would be useful - see below... The VAD Skill will stop Bing Speech Recogntion, as long as I am speaking loudly (on my setup).

Key Features: Speech Detection: Detects when speech starts (Speech Begin) and stops (Speech End). Customizable Actions: Allows users to attach custom scripts that execute automatically when speech starts or stops. For example, you can trigger robot movements, lights, or other interactions based on speech activity. Real-Time Audio Visualization: Displays a live graph of the detected speech level, giving a visual representation of the audio activity. Adjustable Sensitivity: Includes settings to fine-tune detection parameters, such as silence thresholds, for optimal performance in various environments.

I should be able to adjust the Mic input level in Windows Settings to achieve the correct sensitivity level with the VAD Skill.

I just wanted to chime in here and validate @afcorson 's request. I also found that the timed stop listing function in Bing Speech was problematic.

I have my Bing set to a wake word, no VAD and a 5 second time set to stop listening. I have only one command that takes over 5 seconds (7 actually) and many that much less then 5 seconds. If I want Bing to catch the 7 second command this means I need to set the timer to 7 seconds and the much shorter commands will have a very long pause before something happens. I'm just stating all this to be clear and support afcorson.

I really appreacheate @Customer Support and @DJ putting in the work on afcorson 's request. I'm looking forward to testing this myself. If it works for me it will ba a game changer in the appearance of lag in the response of my robot using Bing Voice Commands. I mainly control most everything with Bing Speech. Thank You!

Thanks for those comments. I often use Bing Speech Recognition to communicate with ChatGPT. The person operating the robot could be asking anything which takes 1 sec or 9 secs. That's why it's important to stop listening as soon as they have stopped talkng. It will be even more important when ChatGPT 4-o realtime is available and affordable using audio input.

I dig that you’re pushing for technology that fits your specific scenario, but I don’t know if that technology can do what you’re requesting. If Amazon or Google can’t do it, you won’t see it anywhere else.

Here's whyand to be clear, it can be done with faster real-time AI processing someday.

but here’s why: listening is a conscious act. Our human brains can differentiate multiple sounds and voices to focus on one and interpret it in real-time. Remember, speech and other noises are the same thingthey’re sound. Sound is sound. It’s waveforms. You have to sample the waveform and process it afterward.

After the speech has been completed, it’s not processed in real-time like your brain.

Speech is a sound. The VAD isolates a waveform frequency range through a filter. Then, it tracks how long the filtered waveform is above a specific decibel level compared to the sampled noise floor. The floor is also an average of the sound within the filtered range.

Okay, now the filtered waveform decibel peaks within the sample size exceed the noise floor for a time. That must mean there’s speech.

By this point, we’ve already lost most of the first bit of speech. But it’s now recording

Okay, now there’s more difficulty understanding when the speech stops.

It can be known, but only after a period of time. The period is based on how long the speech is being recorded. Also, the time to maintain an average above the floor is not always accurate for all speech.

For example, several words have quiet parts to them. Specifically, words have spaces of silence between them. In addition, pauses to think of the next word vary between humans.

That's why I said it’s a conscience thing to understand when someone has finished speaking. We also use our eyes, but that’s a different subject.

So, if you were to assume every human across the planet spoke the same volume with the same silence between words you could more accurately do what you want.

This is why your Alexa, Siri, or Google will cut you off as you’re speaking. They don’t know when you’ve finished, either. It’s just impossible to know at present without real-time processing, which doesn’t exist with current computing capability.

I'm guessing the developer could add parameters for silence and sound timingbecause it should be possible. But right now, I know it’s self-adjusting.

I’ll poke them and see if they can add those parameters for you to hardcore. It’ll be a waste of effort, but I’ll have them do it anyway. Once you see what hardcoding will do, you’ll most likely want to use the auto setting.

I tested this skill yesterday, and it works wonderfully. So, I’m more curious about why your setup doesn’t work. Perhaps some effort is necessary to improve your mic type, position, and volume levels.

We can add the parameters for adjusting the silence timeout rather than having it calculated. We meant to do this in yesterday's release but ran into several issues, so we stuck with the calculation used in widespread instances of this feature.

The primary failure points we noticed are mic types (quality), background noise, input volume, and location. Devices such as Amazon Alexa are manufactured for specific hardware from Amazon, for example. This allows them to control the hardware, mic quality, etc.

We tested with a Jabra Conferencing mic designed for that usage. General mics are for audio specifically and are not fine-tuned for speech. A general mic on a laptop or handheld is designed to record everything from music to sound effects to speech. Having a mic designed for speech helps remove false positives.

I think the solution that everyone uses, such as those Sophia robots, is to use a microphone with a push button.

Press and hold the button to record and release it when stopped. Use a looping script to monitor the state of a button. Start and stop recording appropriately.

DJ, were you thinking of something like this? In this example, the switch for the mic would be on port d0.

while (true) {

   // Wait for d0 to become true (press mic button)
   print("Waiting for button press");
   Digital.wait(d0, true);
   
  // Start recording with Bing speech recognition
   print("Start listening");
  ControlCommand("Bing Speech Recognition", "StartListening");
  
  // Wait for d0 to become false (release mic button)
   print("Wait for button release");
  Digital.wait(d0, false);
  
  // Stop recording and let Bing speech process the audio
   print("Stopped listening");
  ControlCommand("Bing Speech Recognition", "StopListening");
}

Yes yes - just like that. I think the bing speech recognition should have a variable set when it is listening and processing. That would be useful for scripts not to bother doing something when it's being processed.