Rhasspy offline voice assistant toolkit

litinoveweedle · April 19, 2020, 7:17pm

Great,
thank you. I can confirm I can start add-on without any issue.

Problem with downloading ANY files on Rpi 3via webconsole is confirmed by synesthesiam himself: https://github.com/synesthesiam/rhasspy/issues/208

tc23 · April 20, 2020, 3:20pm

I am having an issue where I am trying to use a server/client setup, not to dissimilar from the above issue. But I am not running the add-on but through docker. When I try and use the remote STT, recognition is very bad, but when I run on-device (Pi3B+) it works approx 95% of the time. The remote server I am trying to use is an Intel NUC. Both are on the latest stable release, and are using Kaldi. Mic is a Matrix Voice. Has anyone experienced similar?

prologic · May 4, 2020, 3:01am

Hi, Just come across this great project. Unfortunately it all looks a bit too hard and fiddly for me at the moment. We probably need to collectively work on making the setup and productionizing a bit easier

My setup here is a Proxmox VE Virtual Machine running HASS with a bunch of other gear in a Rack cabinet in my home. I can’t see myself adding an external Microphone/Speakers to the Hypervisor nodes just to make this work.

I need to figure out how I can have “Network”(ed) Mics/Speakers in the house which Rhasspy seems to support but I guess it’ll take ma a while to figure this out given the documentation so far.

Keep up the great work!

This is (btw) the type of voice-assistant I really want, Almond is great but it still uses a cloud service for the NLU part. And Ada doesn’t/won’t work for me at all due to my setup and need to have “network”(ed) mic/speaker(s).

Romkabouter · May 5, 2020, 3:32pm

Maybe this tutorial can help:
Tutorials - Rhasspy.

But 2.5 will be out soon, so I suggest checking the community as well:

prologic · May 5, 2020, 6:10pm

I’ve read through this a few times but its not super clear to me how I would apply this type of setup to my virtual environment here. I run Proxmox VE on a 3x-node Cluster. HASS just so happens to be one of the VMs I run. I’m not at all sure how I get for example something like what you describe there as a “Seed Array”? I kind of need to look into some kind of TCP remote Mic/Speaker setup (if that’s even possible).

Romkabouter · May 6, 2020, 7:01am

I’d suggest creating a new VM, and install Rhasppy on there.
You can install the current version:
https://rhasspy.readthedocs.io/en/latest/installation/

Or you can also install the 2.5 prerelease, depending on how comfortabel you are with beta stuff:

For a microphone/speaker, you can use a Pi for that for example. The Pi does not have to have the micarray mentioned, but it needs a microphone. A ps3eye is mentioned as well in various posts

voronovsd · May 17, 2020, 3:27pm

Rhasspy 2.4.20.3 addon, HA image 0.109.6 on RPi 3 B+ with USB Logitech webcam mic.

Now that after 2 days I have a successful trigger by voice interaction outside Web UI, let me share the path:

Web UI error at the Speech tab on getUserMedia - Chrome blocked the mic on http://homeassistant.local:12101/ as insecure.
Fixed by enabling chrome://flags/#unsafely-treat-insecure-origin-as-secure with this URL.
In Web UI Settings tab, the Hass URL was different from the default one you get after flashing Hassio image.
Fixed by setting it to http://homeassistant.local:8123/
In Web UI, intent was recognized correctly and resulted in the error 401 Unauthorized during POST to HASS.
Fixed by generating a Long-lived token at the bottom of the Hassio profile http://homeassistant.local:8123/profile
and setting it in Web UI Settings → Access Token ~~(automatically filled in HassIO)~~ - didn’t happen.
In Web UI Speech tab, intent was sent successfully to Hassio and registered in http://homeassistant.local:8123/developer-tools/event
(Listen to events: rhasspy_GetTime)
Worked for both local and RPi microphone!

So I tried to achieve the same from voice input: no effect due to Settings → Wake word: “No wake word on this device”.

Switched wake word to Pocketsphinx:
“Okey Rhasspy! [high-pitch sound] [another high-pitch sound] [error sound]…”

no effect, and after some time the Web UI was becoming unresponsive, requiring a restart.

Found the info about Snowboy wake word option being more reliable, switched to it.
It downloaded the default file snowboy.umdl automatically!
Found out the default wake word for Snowboy is “snowboy”.
After restart, got the error in Web UI: Snowboy model could not be loaded from profiles/ru/snowboy.umdl
Fixed by changing in Settings the snowboy.umdl to snowboy/snowboy.umdl

And voila, the automation triggered!

- Snowboy!
- [high-pitch sound]
- What time is it?
- [another high-pitch sound] Time is 14:38
# Can’t tell you how happy I was.

Learned a bit of JS, Python and other HA newbie things in the process

It would be nice to fix these issues to simplify the entry for new users. I really like the experience when everything started working, hope not all features will be so challenging.

Alomon · May 20, 2020, 7:57am

Hi,

I currently have HA installed on my RPi. I’ve got a Respeaker 2-mic which I would like to use with Rhasspy, but I understand ther is no driver for it in native HA, so I’d better install Debian or Raspbian (which is best choice?) on the RPi and then Docker then Hassio and Rhasspy. Am I right?

Thanks for your help.

jaburges · June 10, 2020, 8:39pm

I cant imagine you’ll be able to run Rhasspy AND HA on a pi without impacting either system in terms of performance.

you can install the drivers on either Rapbian, or inside the docker, or create a separate docker. check out the wiki for the repseaker

totof-l · June 17, 2020, 8:36pm

Hi ! So I have started using HomeAssistant on a Pi 4 a few days ago, and added Rhasspy on it.
It works great, I can switch off my lights by voice, pretty, pretty, pretty cool !

I am now looking for a way to :

Tell Rhasspy to add some movies / tv shows to Ombi (with Ombi integration on HA) ;
Tell Rhasspy to play some movies / tv shows to Plex / Kodi (with Plex integration on HA) ;

So telling to Rhasspy to send “name: beverly hills cop” on the “ombi.submit_movie_request” service.

I’m also trying to using this kind of sentences/slots with HA.:

Is there a way to set it up with variables ? I would like to say “change light to green” or “change light to red” with one Rhasspy Intent, and one HA automation as well.

manju-rn · June 25, 2020, 4:05pm

in Rhasspy you could define like

[BulbControl]
change (red|green|blue) {color}

in HA intent sciprts

  BulbControl:
    speech:
      text: Bulb Control
    action:
      - service: notify.telenotify
        data_template:
          message: color is {{ color }}

The above is an example to get a notification and display the color information sent from Rhasspy.

Rhasspy can also send a event directly, if you configure the " Send events to Home Assistant (/api/events)" . in HA automation, you just need to define the event type as rhasspy_<Intent_Name>. So in above example it would be event_type: rhasspy_ BulbControl

Each approach have their advantages, since i use both HA internal intent and rhasspy, I tend to prefer send the intent directly from Rhasspy to HA

totof-l · July 8, 2020, 1:37am

Hi, thanks for the reply

For now I’m gonna focus on the Events aspects in Rhasspy and HA, triyng to understand the automations.yaml configuration file.

After a look to this documentation (https://www.home-assistant.io/integrations/plex/), I’ve made this sentence in Rhasspy :

[PlexMovieRequest]
( mets | lance |  joue | lis ) [(la | le | l)] film ($Films){video_name}

It’s supposed to say to HA to tell Plex to play a movie.
I’v already set a slot file (named Films) with all my movies.

Rhasspy can find the movie properly, the name of the movie is linked to the tag “video_name” :

But I don’t know how to set the automations.yaml file to link the “movie_name” variable to the name of the movie.

My automation currently looks like this :

- id: '1592172342812'
  alias: '[PLEX]PlexMovieRequest'
  description: ''
  trigger:
  - event_data: {}
    event_type: rhasspy_PlexMovieRequest
    platform: event
  condition: []
  action:
    service: media_player.play_media
    data:
      entity_id: media_player.plex_plex_for_kodi_kodi_add_on_kodi
      media_content_id:
        \“library_name\”: \“Films\”
        \“video_name\”: \“video_name\”
      media_content_type: VIDEO

I’v tried to set the “event_data” like this, :

 - event_data: {video_name}

but it doesn’t work :


[PLEX]PlexMovieRequest: Error executing script. Invalid data for call_service at pos 1: value should be a string for dictionary value @ data['media_content_id']

How can I tell to HomeAssistant to consider a variable in Rhasspy as a correct data ?

HorizonKane · July 8, 2020, 10:24am

I am using Node-Red, however the variable I have to use there are called slots.whatever (slots.video_name in your case). Maybe try that in HA, too.

Looking at your previous post it might also be event_data.video_name . I’m not sure, just guessing as I use Node-Red like I said. But the principle should be the name, the variable is a property (don’t know i my terminus is correct) of event_name / slots.

I am also not sure if you include the variable correctly in the script. Looking at my old Snips intent scripts, I used to include variables like this (changed it to fit a little to your example and still think you MIGHT need slots. or event_data. prior to video_name):

      media_content_id:
        \“library_name\”: \“Films\”
        \“video_name\”: '{{ video_name }}'

totof-l · July 8, 2020, 2:07pm

Thanks for your suggestions

Since this morning I have made some changes in the " media_content_id" (seems like the exact synthax in this recent documentation : https://www.home-assistant.io/integrations/plex/). So I don’t have any longer the "value should be a string for dictionary value " error.

I have tried a bunch of combinaison, but none of them makes any difference :

      media_content_id: '{ "library_name": "Films", "video_name": event_data.{{ video_name }} }'
      media_content_id: '{ "library_name": "Films", "video_name": "event_data.{{ video_name }}" }'
      media_content_id: '{ "library_name": "Films", "video_name": "event_data.video_name" }'
      media_content_id: '{ "library_name": "Films", "video_name": event_data.video_name }'
	  	  
      media_content_id: '{ "library_name": "Films", "video_name": slots_data.{{ video_name }} }'
      media_content_id: '{ "library_name": "Films", "video_name": "slots_data.{{ video_name }}" }'
      media_content_id: '{ "library_name": "Films", "video_name": slots_data.video_name }'
      media_content_id: '{ "library_name": "Films", "video_name": "slots_data.video_name" }'
	  
      media_content_id: '{ "library_name": "Films", "video_name": {{ video_name }} }'
      media_content_id: '{ "library_name": "Films", "video_name": "{{ video_name }}" }'
      media_content_id: '{ "library_name": "Films", "video_name": video_name }'
      media_content_id: '{ "library_name": "Films", "video_name": "video_name" }'
	  
      media_content_id: '{ "library_name": "Films", "video_name": trigger.event_data.{{ video_name }} }'
      media_content_id: '{ "library_name": "Films", "video_name": "trigger.event_data.{{ video_name }}" }'
      media_content_id: '{ "library_name": "Films", "video_name": trigger.event_data.video_name }'
      media_content_id: '{ "library_name": "Films", "video_name": "trigger.event_data.video_name" }'

I think the problem is for now related to Plex. I can’t call properly the service with a specific value :

entity_id: media_player.plex_plex_for_kodi_kodi
media_content_id: '{ "playlist_name": "Synth", "shuffle": "1" }'
media_content_type: PLAYLIST

Plex don’t react at all. I also see that my “media.control” cards don’t have any buttons to control Plex. I can only see what is currently playing :

Tested on different client (Plexamp, Plex for Android, Plex Extension for Kodi, Plex on Firefox), most of them locally. HomeAssistant is listed on the Plex “Authorized device”, but is not visible on the logs (same for the local IP).

But if this is a Plex issue, i’m gonna look in a different topic, and came later once this is fixed.

HorizonKane · July 8, 2020, 2:18pm

Sorry I am out here as I don’t use Plex No idea how to talk to it.

Did you try to control Plex from the developer tools -> services menu (don’t know if thats the exact names in english UI) in the Home Assistant UI to verify it is working from there? Then you would know if you have a problem in your scripts or with Plex / HA in general.

totof-l · July 8, 2020, 2:33pm

I was wrong, I can see the Play button for the Plex Android client :

But in Developer Tools -> Services , if I test the service like this :

entity_id: media_player.plex_plex_for_android_mobile_xz2_compact
media_content_id: '{ "playlist_name": "Synth", "shuffle": "1" }'
media_content_type: PLAYLIST

It doesn’t work, even with the Plex Android client. I’m gonna try to find some other media player to try this service.

totof-l · July 8, 2020, 4:23pm

I’ve got the same “variables” issues with Ombi.

The sentence in Rhasspy :

[OmbiMovieRequest]
Ajoute le film ($Films){name}

The query :

If I test the service “ombi.submit_movie_request” from Developer Tools -> Services, it work :

But no way to got it to work with automation.yaml, i’v tried again a lot of combinaison :

  name: '{ event_data.{{ name }} }'
  name: '{ "event_data.{{ name }}" }'
  name: '{ "event_data.name" }'
  name: '{ event_data.name }'

  name: '{ slots_data.{{ name }} }'
  name: '{ "slots_data.{{ name }}" }'
  name: '{ slots_data.name }'
  name: '{ "slots_data.name" }'

  name: '{ {{ name }} }'
  name: '{ "{{ name }}" }'
  name: '{ name }'
  name: '{ name }'

  name: '{ trigger.event_data.{{ name }} }'
  name: '{ "trigger.event_data.{{ name }}" }'
  name: '{ trigger.event_data.name }'
  name: '{ "trigger.event_data.name" }'

Romkabouter · July 8, 2020, 9:07pm

Try:

{{ trigger.event.data.name }}

I am using this in my automation:

- id: '1581372525473'
  alias: EventLampen
  trigger:
  - event_data: {}
    event_type: rhasspy_Lights
    platform: event
  condition: []
  action:
  - data_template:
      payload: OK, {{ trigger.event.data.location }} {% if trigger.event.data.action
        == "on" %}aan{% else %}uit{%  endif %}
    service: rest_command.rhasspy_speak
  - data_template:
      entity_id: light.{{ trigger.event.data.location }}
    service_template: light.turn_{{ trigger.event.data.action }}

Also check if Rhasspy is sending events and not intents

totof-l · July 8, 2020, 10:10pm

Hi ! Thanks, it finally worked, after some trying ! The final automation :

- id: '1592172281599'
  alias: '[OMBI]OmbiMovieRequest'
  trigger:
    event_data: {}
    platform: event
    event_type: rhasspy_OmbiMovieRequest
  action:
    service: ombi.submit_movie_request
    data_template:
      name: "{{ trigger.event.data.name }}"

I’m gonna try this synthax for Plex, if I found a workaround for the issue with it

totof-l · July 8, 2020, 10:23pm

Sorry, I forgot… I have choose Ombi to find the right synthax to use for Plex, but I don’t have a slot file for Ombi, only for Plex.

Plex can choose a movie to play from the “Films” slot file, but Ombi is a service who add movies to Plex.

So I have to set Rhasspy and Home Assistant to recognize a word who is not in the sentences/slots.

For exemple :

Ajoute le film ($){name}

Where ($) is the unknown variables.

Is it something possible to set ?

In the worst case, I think I could make a slot file with thousands of names of movies and TV Shows, but I’s gonna take for ever to Rhasspy to train with all this data.