Rhasspy offline voice assistant toolkit

synesthesiam · January 30, 2019, 8:27pm

Once I have the latest version of Rhasspy uploaded (should be this week), you will be able to train a custom hotword with snowboy. Unfortunately, there is no Vietnamese model for pocketsphinx that I know of, so it won’t be able to recognize Vietnamese voice commands. I found this person’s Github project that seems to be trying to create a model, but I can’t read their notes.

Creating an acoustic model from scratch takes a lot of work and data. There are a number of projects on VoxForge, but nothing that I see for Vietnamese. If you can find an acoustic model or have the means to create one, I would happily include it in Rhasspy.

synesthesiam · January 30, 2019, 8:29pm

Are you wanting to record the audio from the voice command after the hotword is spoken? Or do you just want to record audio until silence is heard, and then save it somewhere?

befaint · January 31, 2019, 8:59am

Thanks for your reply. I want to record the audio from the voice command after the hotword is spoken, and then save it in “config/www” folder.

synesthesiam · January 31, 2019, 4:37pm

Do you want Rhasspy to do any speech recognition, or just save the WAV file?

If you don’t want speech recogntion, then you could configure Rhasspy to use an external HTTP server for “speech recognition” and get what you want. Rhasspy will do an HTTP POST to the URL you give with the recorded WAV data. If you run a little HTTP server, you could take in the WAV data and just save it to wherever you want.

befaint · January 31, 2019, 5:13pm

I am needing a custom component for Home Assistant to record audio just after hotword, and then save it in “config/www” folder.
I simply need that.

synesthesiam · January 31, 2019, 9:15pm

OK, this should be doable with the hotword_snowboy and command_listener components, plus one or two Home Assistant automations.

I’m not in a place where I can try this out right now, but you’ll need something like this in your configuration.yaml file:

hotword_snowboy:
  model: /path/to/snowboy.umdl

command_listener:

And then something like this in your automations.yaml file:

# Automatically listen for the hotword when Home Assistant starts.
- alias: "start hotword"
  trigger:
    platform: homeassistant
    event: start
  action:
    service: hotword_snowboy.listen

# Start listening for a command when hotword is detected.
# Save to /config/www/audio.wav
- alias: "hotword -> command listener"
  trigger:
    platform: event
    event_type: hotword_detected
  action:
    - service: command_listener.listen
      data:
        filename: /config/www/audio.wav

# Have snowboy listen again after recording is finished.
- alias: "command listener -> hotword"
  trigger:
    platform: event
    event_type: speech_recorded
  action:
    - service: hotword_snowboy.listen

You’ll want to copy snowboy.umdl somewhere and change the /path/to/snowboy.umdl to wherever you put it. I’d recommend training your own custom model and using that eventually. For now, you should be able to say “snowboy” to wake it up, speak a command, and (after a second of silence) have the /config/www/audio.wav file written.

synesthesiam · January 31, 2019, 9:29pm

To everyone:

I pushed a new version of Rhasspy up today. Some notable features are:

Embedded snowboy (amd64/armhf) and Mycroft Precise (amd64 only) - no MQTT or extra add-ons necessary!
Updated the Profile Settings page with a way to test all available microphones and let you know which ones are receiving audio
Added a “Tap to Record” button that lets you use two clicks instead of holding down (which didn’t work on mobile)
Added a “Wake up” button that causes Rhasspy to listen for a command (silence stops it)
Fixes for web interface to make it work on mobile

Going forward, I’d like to focus on fixing outstanding bugs. The last two Rhasspy versions have involved some major re-tooling under the hood as I’ve settled on a reasonable architecture for this increasingly complex system.

Please let me know if you find bugs!

falphonso · January 31, 2019, 9:39pm

Still not available on aarch64 I guess ?..

19-01-31 21:36:41 INFO (SyncWorker_14) [hassio.docker.addon] Start build 75f2ff60/aarch64-addon-rhasspy:1.20
19-01-31 21:36:42 ERROR (SyncWorker_14) [hassio.docker.addon] Can't build 75f2ff60/aarch64-addon-rhasspy:1.20: no matching manifest for linux/arm64 in the manifest list entries

I’ll wait…

synesthesiam · January 31, 2019, 10:05pm

It’s actually building right now on my server I don’t own a B+, so I have to compile everything through QEMU, which is slooooooow

synesthesiam · February 1, 2019, 12:45am

Ok, give it a try. I had to leave out snowboy because it’s only got a 32-bit library. I’ll revisit that later.

Romkabouter · February 1, 2019, 12:28pm

Tried a fresh install of the Hassio Addon:

/run.sh: line 7: jq: command not found
/run.sh: line 8: jq: command not found
mkdir: cannot create directory ‘’: No such file or directory
 * Serving Flask app "app.py"
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
DEBUG:app:Namespace(profile=None, set=[])
DEBUG:RhasspyCore:Loaded profile from /usr/share/rhasspy/profiles/en/profile.json
INFO:root:++++ Actor System gen (3, 9) started, admin @ ActorAddr-Q.ThespianQ
DEBUG:root:Thespian source: /usr/local/lib/python3.6/site-packages/thespian/__init__.py
DEBUG:DialogueManager:Actors created
DEBUG:DialogueManager:started -> loading
DEBUG:DialogueManager: -> started
DEBUG:PhonetisaurusPronounce: -> started
DEBUG:FuzzyWuzzyIntentTrainer: -> started
DEBUG:PocketsphinxSpeechTrainer: -> started
DEBUG:JsgfSentenceGenerator: -> started
DEBUG:HomeAssistantIntentHandler: -> started
DEBUG:FuzzyWuzzyRecognizer:Loaded examples from /usr/share/rhasspy/profiles/en/intent_examples.json
DEBUG:APlayAudioPlayer: -> started
DEBUG:WebrtcvadCommandListener:started -> loaded
DEBUG:PyAudioRecorder: -> started
DEBUG:FuzzyWuzzyRecognizer:started -> loaded
INFO:PocketsphinxDecoder:Loading decoder with hmm=/usr/share/rhasspy/profiles/en/acoustic_model, dict=/usr/share/rhasspy/profiles/en/dictionary.txt, lm=/usr/share/rhasspy/profiles/en/language_model.txt
DEBUG:WebrtcvadCommandListener: -> started
DEBUG:FuzzyWuzzyRecognizer: -> started
DEBUG:HermesMqtt: -> started
ERROR:SnowboyWakeListener:receiveMessage
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/snowboy/snowboydetect.py", line 14, in swig_import_helper
    return importlib.import_module(mname)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 658, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 571, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 922, in create_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ImportError: /usr/local/lib/python3.6/site-packages/snowboy/_snowboydetect.so: unexpected reloc type 0x03
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/share/rhasspy/rhasspy/actor.py", line 38, in receiveMessage
    self.transition('started')
  File "/usr/share/rhasspy/rhasspy/actor.py", line 59, in transition
    getattr(self, transition_method)(from_state)
  File "/usr/share/rhasspy/rhasspy/wake.py", line 142, in to_started
    self.load_detector()
  File "/usr/share/rhasspy/rhasspy/wake.py", line 186, in load_detector
    from snowboy import snowboydetect, snowboydecoder
  File "/usr/local/lib/python3.6/site-packages/snowboy/snowboydetect.py", line 17, in <module>
    _snowboydetect = swig_import_helper()
  File "/usr/local/lib/python3.6/site-packages/snowboy/snowboydetect.py", line 16, in swig_import_helper
    return importlib.import_module('_snowboydetect')
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
ModuleNotFoundError: No module named '_snowboydetect'

synesthesiam · February 1, 2019, 4:13pm

OK, thanks for the quick feedback. I’ll try to adjust the Docker base images. Seems like there’s no single image for the Raspberry Pi that simultaneously supports Python 3.6, tensorflow, and snowboy’s pre-compiled binaries…

hassping · February 1, 2019, 4:35pm

Just installed, feels much easier to use, being able to just test the mics and having it mark the active one is great. Awesome work.

falphonso · February 1, 2019, 8:35pm

/run.sh: line 7: jq: command not found
/run.sh: line 8: jq: command not found
mkdir: cannot create directory ‘’: No such file or directory
 * Serving Flask app "app.py"
 * Environment: production
   WARNING: Do not use the development server in a production environment.
   Use a production WSGI server instead.
 * Debug mode: off
DEBUG:app:Namespace(profile=None, set=[])
DEBUG:RhasspyCore:Loaded profile from /usr/share/rhasspy/profiles/en/profile.json
INFO:root:++++ Actor System gen (3, 9) started, admin @ ActorAddr-Q.ThespianQ
DEBUG:root:Thespian source: /usr/local/lib/python3.6/site-packages/thespian/__init__.py
DEBUG:DialogueManager:Actors created
DEBUG:DialogueManager:started -> loading
DEBUG:DialogueManager: -> started
DEBUG:HermesMqtt: -> started
DEBUG:PhonetisaurusPronounce: -> started
DEBUG:FuzzyWuzzyIntentTrainer: -> started
DEBUG:JsgfSentenceGenerator: -> started
DEBUG:PocketsphinxSpeechTrainer: -> started
DEBUG:FuzzyWuzzyRecognizer:Loaded examples from /usr/share/rhasspy/profiles/en/intent_examples.json
DEBUG:APlayAudioPlayer: -> started
INFO:PocketsphinxDecoder:Loading decoder with hmm=/usr/share/rhasspy/profiles/en/acoustic_model, dict=/usr/share/rhasspy/profiles/en/dictionary.txt, lm=/usr/share/rhasspy/profiles/en/language_model.txt
DEBUG:WebrtcvadCommandListener:started -> loaded
ERROR:SnowboyWakeListener:receiveMessage
Traceback (most recent call last):
  File "/usr/share/rhasspy/rhasspy/actor.py", line 38, in receiveMessage
    self.transition('started')
  File "/usr/share/rhasspy/rhasspy/actor.py", line 59, in transition
    getattr(self, transition_method)(from_state)
  File "/usr/share/rhasspy/rhasspy/wake.py", line 142, in to_started
    self.load_detector()
  File "/usr/share/rhasspy/rhasspy/wake.py", line 186, in load_detector
    from snowboy import snowboydetect, snowboydecoder
ModuleNotFoundError: No module named 'snowboy'
DEBUG:HomeAssistantIntentHandler: -> started
DEBUG:FuzzyWuzzyRecognizer:started -> loaded
DEBUG:WebrtcvadCommandListener: -> started
DEBUG:FuzzyWuzzyRecognizer: -> started
DEBUG:PyAudioRecorder: -> started
DEBUG:PocketsphinxDecoder:started -> loaded
DEBUG:PocketsphinxDecoder: -> started

It installed fine. Thanks

Have to reboot the RPi on which it runs as I pluged the USB mic by installing the Addon and I think the mic is not seen. But will do the reboot only tomorrow. Will come back here to tell you.

A question though, do I have to install the aside Snowboy Addon still or is it incorporated within the Rhasspy Addon now ?

synesthesiam · February 1, 2019, 9:21pm

When I get the Docker issues sorted out for Raspberry Pi’s, you should be able to run snowboy inside of Rhasspy without anything extra.

But beware, I haven’t tested with aarch64. I see a library for it in the snowboy install file, so I’m hopeful. Worst case you can at least try it out with pocketsphinx.

Romkabouter · February 2, 2019, 9:42am

Hmm, I have hassio running on a 3+.
I have another Pi somewhere so I will try that as well

falphonso · February 2, 2019, 1:04pm

Ok…rebooted the RPi with the USB Mic plugged in…
but seems I am a bit out of luck here…
The addon starts well.
I can see the right output, but no luck in seeing the entry select for the input.
When playing a bit with portainer and the console of the rhasspy container the output of ‘arecord -l’ however displays:

root@75f2ff60-rhasspy:/# arecord -l
**** List of CAPTURE Hardware Devices ****
card 1: CD04 [CD04], device 0: USB Audio [USB Audio]
  Subdevices: 1/1
  Subdevice #0: subdevice #0
root@75f2ff60-rhasspy:/# 
root@75f2ff60-rhasspy:/# arecord -L
null
    Discard all samples (playback) or generate zero samples (capture)
default
mic
speaker
sysdefault:CARD=CD04
    CD04, USB Audio
    Default Audio Device
front:CARD=CD04,DEV=0
    CD04, USB Audio
    Front speakers
surround21:CARD=CD04,DEV=0
    CD04, USB Audio
    2.1 Surround output to Front and Subwoofer speakers
surround40:CARD=CD04,DEV=0
    CD04, USB Audio
    4.0 Surround output to Front and Rear speakers
surround41:CARD=CD04,DEV=0
    CD04, USB Audio
    4.1 Surround output to Front, Rear and Subwoofer speakers
surround50:CARD=CD04,DEV=0
    CD04, USB Audio
    5.0 Surround output to Front, Center and Rear speakers
surround51:CARD=CD04,DEV=0
    CD04, USB Audio
    5.1 Surround output to Front, Center, Rear and Subwoofer speakers
surround71:CARD=CD04,DEV=0
    CD04, USB Audio
    7.1 Surround output to Front, Center, Side, Rear and Woofer speakers
iec958:CARD=CD04,DEV=0
    CD04, USB Audio
    IEC958 (S/PDIF) Digital Audio Output
dmix:CARD=CD04,DEV=0
    CD04, USB Audio
    Direct sample mixing device
dsnoop:CARD=CD04,DEV=0
    CD04, USB Audio
    Direct sample snooping device
hw:CARD=CD04,DEV=0
    CD04, USB Audio
    Direct hardware device without any conversions
plughw:CARD=CD04,DEV=0
    CD04, USB Audio
    Hardware device with all software conversions
root@75f2ff60-rhasspy:/#

And aside this, the web UI doesn’t open neither.
The addon config displays that the 12101 port of the container should be forwarded to the TCP/12101 port of the host, but it’s impossible to reach the web interface.

I admit I am a bit lost here…

synesthesiam · February 2, 2019, 2:06pm

Great, thank you for your time! I’ll add some more MQTT functionality once we get these fundamental issues sorted.

synesthesiam · February 2, 2019, 2:08pm

The critical line to look for in the log is this:

INFO:werkzeug: * Running on http://0.0.0.0:12101/ (Press CTRL+C to quit)

This indicates that the Flask web server started, which comes at the very end of the start-up process. I just pushed a new version that has a 15 second timeout, so the web server should start even if one of the sub-components (microphone, etc.) fails to start.

synesthesiam · February 2, 2019, 4:09pm

OK, I’m finally getting a chance to test again on Hass.IO and I’m really disappointed in snowboy. It seems to be useless on ARMv7 due to the unexpected reloc type 0x03 bug, and it won’t even install on ARMv8 (aarch64)!

I can at least try and back off to an ARMv6 docker base for Rhasspy on the Pi 3, but I’m not sure what to do for the Pi 3 B+. There is clearly an aarch64 library for snowboy, but it doesn’t seem to work. Mycroft Precise is also out on the Pi’s unless I can get tensorflow installed there. Any thoughts?