Sherpa-ONNX Speech To Text

How to use the Sherpa-ONNX Speech To Text robot skill

Offline speech recognition with the open source sherpa-onnx toolkit from the Next-gen Kaldi team. It runs a neural network model on the computer ARC is running on, with no account, no cost per phrase and no internet connection once the model has been downloaded.

Unlike the cloud engines, which record a few seconds and send them away, this robot skill holds the microphone open and recognizes continuously. It answers in a fraction of a second and costs nothing to leave running all day - but it also hears every word said anywhere near the robot, which is why it has a wake word.

Unlike ARC's own Speech Recognition robot skill, it is not limited to a list of phrases. It can recognize anything that is said and put it in a variable, which is what a conversational robot skill downstream wants, and it will still run scripts from a phrase list if you give it one.

It does not do text to speech. Pair it with a speech synthesis robot skill if the robot should also talk.

Requirements

• ARC on Windows.
• An internet connection the first time a model is used, to download it. After that, none.
• Disk space for the model: 30MB to 640MB depending on the one chosen.
• A microphone on this computer. Choose which one in the configuration.
• Only one instance per project.

Getting started

1. Add the robot skill to the project. It starts with a small streaming English model selected and begins downloading it (57MB) straight away, and it is listening within a minute.
2. Say one of the example phrases - "robot turn left". It appears in the log with how closely it matched.
3. Open the configuration to change the phrases, choose a different model, or turn on the wake word.

Models are shared by every project on the computer and are only ever downloaded once. They are kept in Documents\ARC\Speech Models\Sherpa-Onnx - or in ProgramData\ARC\Speech Models\Sherpa-Onnx when the Windows user name has accented or non-Latin characters, which the model loader cannot open. The configuration shows the exact folder and has a button to open it. Delete a model's folder to remove it.

Choosing a model

There are two kinds, and the difference is the one you will notice:

• Streaming models recognize while you speak. The text appears in the window word by word, and the answer is there the moment you stop. Best for commands and for a robot that should feel responsive.
• The others wait until you stop speaking, then recognize the whole utterance at once. They are usually more accurate, and add punctuation and capitals (which are removed by default). A small voice activity detector (Silero, 2MB) decides where each utterance starts and ends; it is downloaded with the model.

Any other sherpa-onnx speech recognition model can be used too: download and extract one from https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models, choose Custom model folder, enter the folder and pick its kind. The files are found by name (encoder, decoder, joiner, tokens.txt and so on), and int8 copies are preferred when a model ships both.

The window

• The status says what the robot skill is doing: Downloading or Unpacking with a percentage, Loading model, Listening, Hearing speech, Robot is talking, Paused or Error.
• Under it is the model that is loaded, or the reason it could not be.
• The words being spoken appear in italics as they are recognized, with streaming models.
• The level bar is the microphone. It turns red while the robot skill is paused or while the robot's own voice is being ignored. If it never moves while you speak, the problem is the microphone, not the model.
• Pause stops the robot skill acting on what it hears. The model stays loaded, so unpausing is instant.
• With a wake word configured, Wake and Sleep do by hand what the wake word does by voice, and the label between them counts down how long the robot stays awake.
• The log holds the last 200 lines: model loading and how long it took, every phrase with how closely it matched, wake words, and anything that went wrong.

Wake word

Because the engine hears everything, all the time, a wake word is what stops every stray sentence in the room from being a command.

There is no second recognizer and no extra processor cost. The wake word is matched against the text that was already recognized, which means:

• it can be any phrase, in any language the model knows, not a keyword from a fixed list;
• it still counts when the model mishears a letter or two;
• several alternatives can be given, one per line, so people have more than one way to get the robot's attention;
• the wake word and a command can arrive in the same breath. "Robot, turn left" wakes the robot and turns left, with the wake word taken off the front. Turn Accept the wake word and a command inone breath off to require a pause instead.

After the wake word the robot stays awake for Stay awake for (seconds), so a conversation does not need the wake word in front of every sentence. Set it to 0 for one command per wake word. With Every command starts the timer again on, the robot keeps listening as long as somebody keeps talking to it.

Match threshold is the one to tune. A short wake word wants a high threshold, because short words resemble each other - if "robot" keeps waking on "rowboat", either raise the threshold or use a longer wake word such as "hey robot".

Phrases and scripts

Phrases are matched loosely rather than compiled into a grammar, because none of these models can be restricted to one. Everything is recognized and then compared with the list: the phrase may be at the start of a longer sentence ("turn left please" matches turn left), and small mistakes are forgiven according to Phrase match threshold. How close the match was becomes the confidence.

Leave the phrase list empty to have everything that is heard go into the variables and run the anything recognized script - that is the setup for feeding a conversational robot skill.

Global variables

Any of them can be renamed, or emptied to not set it at all.

Configuration

Model

Recognition

Pausing

A pause phrase and a resume phrase let the robot be told out loud to stop and start listening. Setting a resume phrase keeps the engine recognizing while the robot skill is paused, so that it can hear it - which costs processor time. With no resume phrase, pausing stops the decoding altogether.

Control commands

Troubleshooting

See also

• sherpa-onnx - https://github.com/k2-fsa/sherpa-onnx
• The models - https://github.com/k2-fsa/sherpa-onnx/releases/tag/asr-models