Get conversation inputs and outputs to dashboard

Configuration Voice Assistant
1

Hey guys, i try to populate my dashboard with notifications about my voice assistant / assist questions and responses.

My idea is to put up a notification on my dashboard, when i ask my voice assistant anything, but i don’t know how to get conversation inputs, intents, responses and so on to node red to pass it to some entity.

i can just monitor conversation.home_assistant (or any other agent) and the entity just changes its timestamp on activation.

is there a specific event i can filter for that holds this data or do i need to create an automation to get all incoming conversations?

Greetings :slight_smile:

1 Like
2

I would also like to find a solution for this. I want to listen to the agent’s response. I haven’t figured out how to do it yet.

3

Agreed, it’s impossible to debug what’s happening without this.

4

Nobody else interested in putting this kind of information on their dashboards?

It seems like there is already something out for the voice-assistant ESP32-S3-Box-3, but not available in HA dashboards?

5

If you are looking for a way to capture what the assistant heard and says then you can add the following to your voice assistant device yaml configuration. This gives you a text sensor within Home Assistant with the last conversation request and response.

text_sensor:
###### Text Request ######
  - platform: template
    id: text_request
    name: Text Request
    icon: mdi:message-processing
    internal: false
    on_value:
      lambda: |-
        if(id(text_request).state.length()>32) {
          std::string name = id(text_request).state.c_str();
          std::string truncated = esphome::str_truncate(name.c_str(),31);
          id(text_request).state = (truncated+"...").c_str();
        }


###### Text Response ######
  - platform: template
    id: text_response
    name: Text Response
    icon: mdi:message-reply-text
    internal: false
    on_value:
      lambda: |-
        if(id(text_response).state.length()>32) {
          std::string name = id(text_response).state.c_str();
          std::string truncated = esphome::str_truncate(name.c_str(),31);
          id(text_response).state = (truncated+"...").c_str();
        }


voice_assistant:
...
  on_listening:
    - lambda: id(text_request).publish_state("...");
    - lambda: id(text_response).publish_state("...");
  on_stt_end:
    - text_sensor.template.publish:
        id: text_request
        state: !lambda return x;
  on_tts_start:
    - text_sensor.template.publish:
        id: text_response
        state: !lambda return x;

Hope this helps you out.

2 Likes
6

Thank You very much!
I already found this solution and thought about why i have to send data so many times from server to satellite to server again, if i could just get everything on the server.

But your solution is at least a small workaround! :slight_smile:

Another Question is, if we can get all conversations in and outputs from assist.
Probably to act on anything written or said. Also to get conversations from smartphone and smartwatch apps.

7

@john500y2k
thanks for sharing details.
I copied text_sensor and voice_assistant into esp home config.

How to use this now in a dashboard?

8

Your esphome devices should have a new entities with the names Text Request and Text Response in Home assistant. You can use these entities in your dashboard configuration how you like.

1 Like
9

Based on the same idea as @john500y2k , you can find here a long post with a solution which seems to work to log to a file everything asked and answered. Log to a file everything "discussed" with ChatGPT on all Voice Assist devices - #3 by adynis

10

Thanks for the hint!
Also not very elegant and i still hope for an implementation on “assist” level :sweat_smile:

11

With all the new voice features for assist it is a shame that there is still no way to extract what assist gets as input and outputs. This would be the perfect Question for the WTH month (for me! :wink: ).

With my VoicePE i used the described workaround to get at least the LLM input (STT) and output (TTS).
But if a question is processed locally there is no way to get any feedback.

There could at least be an event i could watch for.

There would be many possibilities if we also add parameters (agent, intent, raw input, confidence, …)

We could add timers started by voice to our dashboard
Or we could build a little response screen to our dashboard-tablet showing Questions and Answers we said/got by voice.