Solved: Voice Assist to run an automation in Node-RED

I have all my automations in node-RED, where they are currently triggered from Rhasspy voice assistant by a MQTT-In node. This totally bypasses Home Assistant, and works well.

I am testing HA Voice Assist with a RasPi and wyoming-satellite, and it is turning my study light on/off (now that I have added half-dozen aliases for “study light”) … and I see there are instructions for getting Assist to run a HA Automation …

Has anyone got Voice Assist to run an automation in Node-RED ?

Most of my automations are manipulating devices in HA - eg “turn on/off the study light” - which are already handled by Voice Assist :slight_smile:

According to the documentation for “Sentence” node in node-RED:

A sentence trigger fires when Assist matches a sentence from a voice assistant using the default conversation agent.

I copied the example … but it seems to leave out as much as it mentions. Seems exceedingly easy to feed a text command in to Assist; but I don’t see much value for that. The example “sentence” node appears to be missing information; I am particularly puzzled at the “Expose As” parameter.

Looking through my Rhasspy sentences there are not many which will need node-RED. I have added these to a node-RED “Sentence” node

and to configuration.yaml

automation:
  trigger:
    - platform: conversation
      command:
        - "( what time is it [please] | whats the time )"
        - "good morning"
        - "( Dinner time | Time for dinner [please] )"
        - "( Goodnight | good night | nighty night | night night )"

Reboot - too often I have found that a restart hasn’t made a difference - and try again

When I ask, I hear a message “Sorry, I’m not aware of any device called time”.

In the Whisper logs:

INFO:faster_whisper:Processing audio with duration 00:01.510
INFO:wyoming_faster_whisper.handler: What is the time?
ERROR:asyncio:Task exception was never retrieved
future: <Task finished name='wyoming event handler' coro=<AsyncEventHandler.run() done, defined at /usr/local/lib/python3.9/dist-packages/wyoming/server.py:31> exception=ConnectionResetError('Connection lost')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/wyoming/server.py", line 41, in run
    if not (await self.handle_event(event)):
  File "/usr/local/lib/python3.9/dist-packages/wyoming_faster_whisper/handler.py", line 95, in handle_event
    await self.write_event(self.wyoming_info_event)
  File "/usr/local/lib/python3.9/dist-packages/wyoming/server.py", line 29, in write_event
    await async_write_event(event, self.writer)
  File "/usr/local/lib/python3.9/dist-packages/wyoming/event.py", line 131, in async_write_event
    await writer.drain()
  File "/usr/lib/python3.9/asyncio/streams.py", line 387, in drain
    await self._protocol._drain_helper()
  File "/usr/lib/python3.9/asyncio/streams.py", line 190, in _drain_helper
    raise ConnectionResetError('Connection lost')
ConnectionResetError: Connection lost

but nothing in Node-RED. Nothing in debug, and no time stamp on the node to indicate it was activated.

Ohhh, just noticed in the core log on starting HAOS:

Logger: homeassistant.components.automation
Source: components/automation/config.py:107
integration: Automation (documentation, issues)
First occurred: 8:17:12 pm (1 occurrences)
Last logged: 8:17:12 pm

Unnamed automation could not be validated and has been disabled: required key not provided @ data['action']. Got None

Do I need to create dummy Automations in HA, which do nothing ?

Whats the time and what is the time are two different sentences that cannot be matched.


Thinking that I may need to create a dummy Automation, I go into the Settings > Automations and Scenes menu option … and discover an “automation 0” last triggered “Never”. Clicking to open reveals

Tried the [Migrate] button, edited and saved. Don’t know where it went, but not on the Automations page no matter how many times i refreshed.

Looked in automatoins.yaml, and there it is … along with some other (presumably old) crap

  • id: ‘1703386043388’
    alias: test1
    description: ‘’
    trigger:
    • platform: conversation
      command: this is a test
      condition:
      action:
    • service: mqtt.publish
      data:
      qos: 0
      retain: false
      topic: hermes/intent/#
      payload: test test
      mode: single
  • id: party_time
    trigger:
    • platform: conversation
      command:
      • ‘[it’‘s ]party time’
      • happy (new year|birthday)

Lets try the “Edit in yaml” option, and that looks surprisingly similar to what I was told to put into configuaration.yaml … so remove.

I just copied from Rhasspy’s sentences file. I rarely ask the time, but it seems easier to test with than the “good morning” or “good night” which each control several devices.
I have changed to “what time is it | (what is | whats) the time”

1 Like

OK. Managed to delete the Automations and add a new one with only trigger of conversation. Reboot and the error has gone from the log file.

Interesting ! I could add automations through the UI, but the Settings > Automations page keeps saying “Start automating” … despite my new automations being in automations.yaml.
I had previously moved my automations from HA to node-RED and had commented out the “automation: !include automations.yaml” line in configuration.yaml, but maybe I had left the automations.yaml file unused ?
No. Deleting automations.yaml, removing the automation: line from configuration.yaml and restarting HA brings me back to “Start automating” … but creating an automation creates the automations.yaml file but still “Start automating” :frowning:

OK, my new automations work in HA … but still nothing triggering the Sentence node in node-RED

Thanks to zachowj over in Sentence Node (beta) · zachowj/node-red-contrib-home-assistant-websocket · Discussion #981 · GitHub I got the HA Automations working


Just updated to HA 2024.4.1 and the HACS Node-RED Companion to v3.1.4.

I have one HA automation (triggered by “hello”) setup and working, giving the voice response. The Node-RED Sentence node is not triggered.
image


When I add “hello” to the sentences in the node-RED “sentence” node, then say “hey jarvis, hello” I now hear “Done” as the response - but the NR Sentence node is now triggered ! Progress !!

Now to figure out and translate the HA Voice Assist fields into the format my flows expect.

With Rhasspy all the intents are defined in sentences.ini by defining the intent name (in square brackets), then defining the syntax of the various sentences which would trigger that intent.

Ah-ha ! I was trying to use one sentence node to define all the sentences, then wondering how to work out which intent they are from … BUT now I realise each intent should be defined in its own separate sentence node, and this allows multiple sentences to be viewed easily.

Hmmm … sometimes it works but sometimes it doesn’t. More research required.

Also, having got a NR flow started, how do I (a) stop the “done” response from Voice Assist, and (b) send a response back from Node-RED to the initiating satellite ?

(a) “Done” is the default “Response” in the NR sentence node, and is given immediately the sentence is triggered - ie before NR has even started the flow. This is not right.

If someone asks you to get the milk from the fridge do you reply “done” before you start the job ? What if there is no milk in the fridge ? “hang on”, “thinking” or “roger” are more appropriate … to be followed up with “done” only when the job has been completed.
And “Done” is not an appropriate reply to “what is the weather today?”

(b) there appears to be no way in HA or NR to send a message to a Wyoming / Piper / voice satellite device. These devices have entities - but no services !

Probably makes more sense to separate the input (microphone, Whisper) functions from the output (Piper, speaker) functions; or at least expose the speaker as (a subset of) media_player.

TTS.speak already supports output to Piper (to keep the voice consistent) but requires a media_player entity to send the audio to.

Using one device for media player and voice assistant has been requested many times, and seems a logical combination. Echo cancellation requires that the audio going out the speaker be subtracted from the audio coming in the microphone.

As mentioned before, HA Voice Assist provides voice responses

  • to built-in device based intents (eg turn on the kitchen light), and
  • from HA Automations (Set conversation response action).

All my automations are in Node-RED, so neither of these will work for me.

  • In Node-RED a standard Call Service node with conversation.process can be used to get HA Voice Assist to process the command text. No voice response is given.
  • There is a “sentence” node in the v0.64.0 “node-red-contrib-home-assistant-websocket” Node-RED Palette addon - however it sends the response immediately the sentence is detected - ie before any HR flow can be executed.
  • Wyoming-satellite has a few entities, but no services … thus no way to send a response to the satellite.
  • We can use TTS.speak, with entity piper.tts - but this requires a media_player … which wyoming-satelite is not.

Maybe we can create a dummy HA automation which triggers a NR flow and returns to the HA automation in time to give the conversation response. Rather messy.

The solution is to install MPD on the RasPi satellite and use that as the destination for TTS.speak. This requires:

  • installing and configuring MPD on each satellite
  • MPD does not have a HA UI, so requires setting up mpd device(s) in configuration.yaml
  • a translation between the wyoming-satellite’s device id (a unique long internal identifier) and the appropriate media_player entity id. Fortunately a Change node does this easily in NR.
  • once all that is done, we can use a call service node calling the tts.speak Service with Entity of tts.piper (to maintain the same voice), and Data of {“media_player_entity_id”:“{{deviceId}}”, “message”:“{{payload}}”}