Sorry, I couldn't understand automation?

Configuration Voice Assistant
1

Hi everyone,

Is it possible to create an automation that is trigerred when assistant cannot understand a command and then sends a notification with the transcript of what it heard? Bonus points if it somehow also sends a link to the audio recording.

Thanks!

2

Sidenote: Whisper’s logs will show you the transcript of what it heard/processed.
Go to Settings > Add-ons > Whisper > Log tab
Example:
INFO:wyoming_faster_whisper.handler: Hey there, community!

I couldn’t find any event fired by the voice assist pipeline that would readily allow to automate as you want.
But if you use ESPHome, you can do it by recording the various values from voice_assistant events and then detecting the proper set of conditions from an automation. Bonus, it can also serve as a history for voice commands, and allow replay of last TTS through the recorded URL or text.

In ESPHome device:

text_sensor:
  - platform: template
    name: "Last STT"
    id: last_stt
    entity_category: diagnostic
    update_interval: never
  - platform: template
    name: "Last TTS"
    id: last_tts
    entity_category: diagnostic
    update_interval: never
  - platform: template
    name: "Last TTS URL"
    id: last_tts_url
    entity_category: diagnostic
    update_interval: never

voice_assistant:
  # add the rest
  on_stt_end:
    - text_sensor.template.publish:
        id: last_stt
        state: !lambda 'return x;'
  on_tts_start:
    - text_sensor.template.publish:
        id: last_tts
        state: !lambda 'return x;'
  on_tts_end:
    - text_sensor.template.publish:
        id: last_tts_url
        state: !lambda 'return x;'

In automation:

alias: Voice Assist Error
description: ""
trigger:
  - platform: state
    entity_id:
      - binary_sensor.voice_box_assist_in_progress
    from: "on"
    to: "off"
    for:
      hours: 0
      minutes: 0
      seconds: 2
condition:
  - condition: state
    entity_id: sensor.voice_box_last_tts
    state: Sorry, I couldn't understand that
action:
  - service: notify.persistent_notification
    data:
      title: Voice Assist Unrecognized Intent
      message: >-
        Understood: {{ states("sensor.voice_box_last_stt") }}
mode: single

Not sure about the STT recording, I couldn’t find anything being published to the /api/stt/{provider} endpoint or anywhere else… Maybe someone else will have better luck.

3

Good question. I hope there’s eventually a trigger/automation for “Sorry, I couldn’t understand that”.