Typed intent works but spoken does not

Configuration Voice Assistant
1

I am trying to implement a voice shopping list. What I’ve done so far responds correctly if I type the command into the assistant or if I speak it into the assist pop-up on my android phone app.

But, if I speak the command to my Voice Assistant PE it does not work. The Voice Assistant PE responds to the wake word and after I say the trigger and item “Add Eggs” for example. I get three quick blue flashed from the Voice Assistsnt PE light ring and nothing further happens.

Here’s what I have configured:
I found this blog post describing how to setup an intent and custom sentence. Adding shopping list items via Home Assistant voice commands

I added the intent to my configuration.yaml

# Voie Shopping List
conversation:
  intents: {}

intent_script:
  ShoppingListAddItem:
    action:
      action: todo.add_item
      target:
        entity_id: todo.shopping_list
      data:
        item: "{{ shop_item }}"
    speech:
      text: "{{ shop_item }} added to your shopping list."

and created a custom sentence file custom_sentences/en/shopping_list.yaml

language: "en"
intents:
  ShoppingListAddItem:
    data:
      - sentences:
          - "Add {shop_item}"
lists:
  shop_item:
    wildcard: true

I only have one local Assist configured. I am using the Home Assistant conversation agent, and speech-to-phrase for the STT.

Do I need to do something additional to get the Voice Assistant PE unit to work with a custom intent and sentence like this?

Thanks!
Tom G.

2

I believe there’s now a built-in sentence to add items to the shopping list - could there be some conflict there?

3

Mine does this on one of them. What’s the debug logo the voice pipeline say for that device when it happens?

4

If so, then perhaps the issue I’m having with Voice Assistant PE was preventing the shopping list adds from working in the first place. Not knowing that was there I attempted the above solution. Time for further investigation…

5

I’m at work now but I will look at the debug logs this evening when I am back at home. Thanks!

1 Like
6

Yeah my master bathroom vpe is dropping a LOT. It’ll start the voice response then… Stop.

I THINK it’s a connectivity issue to that device as it also has audio playback issues. The rest of mine (5) don’t do this. So it’s probably not the intent itself but the vpe.

7

I haven’t played with STP much, but my understanding is that has its own set of built-in sentences as well as customizations (See here for shopping list), which will differ from the regular built-in sentences that are used when one types in a text box or uses with a voice assistant that uses STT.

I presume you are putting your custom sentence under the ./configuration/, but for STP, the AddOn doc says:

  • You can add custom sentences to /share/speech-to-phrase/custom_sentences/<language>/sentences.yaml
8

Thanks!

A quick switch from Speech To Phrase to Fast Whisper proved the issue was with STT.

When on FW the Voice Assistant PE handles shopping list updates properly.

I added the custom sentence code to

share/speech-to-phrase/custom_sentences//sentences.yaml

and switched back to STT. But, no joy. I guess I have more readiing to do on STT customization…

1 Like
9

What does the debug log show when it fails with Whisper?

My satisfaction with voice improved a lot when I started using Whisper Large Turbo. Unfortunately Large Turbo is too resource intensive to be offered as a standard option from within HA.

10

To clarify, it’s working with Fast Whisper. I turned on debug logging and spoke the sentence “Hey Jarvis, Add Eggs” Eggs appeared on the shopping list and here’s what showed in the log:

2025-03-21 14:47:05.908 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_STT_VAD_END

2025-03-21 14:47:05.909 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00025c080c]
2025-03-21 14:47:08.484 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_STT_END
data {
  name: "text"
  value: " Add eggs"
}

2025-03-21 14:47:08.484 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00155c080412110a04746578741209204164642065676773]
2025-03-21 14:47:08.484 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_INTENT_START

2025-03-21 14:47:08.484 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00025c0805]
2025-03-21 14:47:08.497 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_INTENT_END
data {
  name: "conversation_id"
  value: "01JPX08N7RJN7FHP5ETAWN3HHF"
}

2025-03-21 14:47:08.497 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00315c0806122d0a0f636f6e766572736174696f6e5f6964121a30314a505830384e37524a4e3746485035455441574e33484846]
2025-03-21 14:47:08.498 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_TTS_START
data {
  name: "text"
  value: "eggs added to your shopping list."
}

2025-03-21 14:47:08.498 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [002d5c080712290a047465787412216567677320616464656420746f20796f75722073686f7070696e67206c6973742e]
2025-03-21 14:47:08.500 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_TTS_END
data {
  name: "url"
  value: "http://192.168.6.10:8123/api/tts_proxy/kLbsM507L_Gxiz2SWloczA.flac"

I switched back to Speech to Phrase and repeated the sentence. It either ignored me or mistakenly activated a scene. Here’s what I found in the log:

(I’m not sure how it got “activate Read scene” from me speaking the words “Add Eggs” but apparently it did…)

2025-03-21 14:56:13.562 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_STT_VAD_END

2025-03-21 14:56:13.562 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00025c080c]
2025-03-21 14:56:13.946 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_STT_END
data {
  name: "text"
  value: "activate Read scene"
}

2025-03-21 14:56:13.946 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [001f5c0804121b0a0474657874121361637469766174652052656164207363656e65]
2025-03-21 14:56:13.949 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_INTENT_START

2025-03-21 14:56:13.949 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00025c0805]
2025-03-21 14:56:15.451 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:15.652 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.688 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.704 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.704 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.721 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.722 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.746 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.747 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.774 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.775 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.798 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.799 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.802 DEBUG (MainThread) [homeassistant.components.esphome.assist_satellite] Received unknown pipeline event type: intent-progress
2025-03-21 14:56:16.804 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_INTENT_END
data {
  name: "conversation_id"
  value: "01JPX08N7RJN7FHP5ETAWN3HHF"
}

2025-03-21 14:56:16.804 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00315c0806122d0a0f636f6e766572736174696f6e5f6964121a30314a505830384e37524a4e3746485035455441574e33484846]
2025-03-21 14:56:16.805 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_TTS_START
data {
  name: "text"
  value: "The Read scene in the Master Bedroom has been activated."
}

2025-03-21 14:56:16.805 DEBUG (MainThread) [aioesphomeapi._frame_helper.base] home-assistant-voice-097259 @ 192.168.6.11: Sending frame: [00445c080712400a047465787412385468652052656164207363656e6520696e20746865204d617374657220426564726f6f6d20686173206265656e206163746976617465642e]
2025-03-21 14:56:16.807 DEBUG (MainThread) [aioesphomeapi.connection] home-assistant-voice-097259 @ 192.168.6.11: Sending VoiceAssistantEventResponse: event_type: VOICE_ASSISTANT_TTS_END
data {
  name: "url"
  value: "http://192.168.6.10:8123/api/tts_proxy/AgWgxjc4Vwu_9Qu89LzPIA.flac"