Query Voice Assistant about room level presence

Configuration Voice Assistant
1

Has anyone figured out a efficient solution for integrating Bermuda with voice assistant?

The problem im having stems from having two entities that i want to share the same name. So I would like to be able to query assist to be able to tell me whether Mary is home or away, or query “what room is Mary in?” I can create custom sentence/response, and expose the bermuda derived location sensor to assist, but if the goal is to the have a natural language convo with the voice assistant, i dont want to ask “what room is Mary’s BLE tracker in”, i want to just ask about Mary.

You could rename the sensor to just be Mary, but that will conflict with the person entity Mary.

So I created a MQTT device tracker from the Bermuda sensor so I can add it to the person entity i want to track, but assist cannot seem to recognize anything other than the home/away status of Mary.

Anyway, if anyone has any thoughts im very curious to see how this might be done.

2

You are ascribing too much to the llm.

You need to give it enough information to be able to interpret the data in front of it. If you want it to understand that

Mary has two devices and one of them is usually in her pocket so if your user asks about Mary’s location you should check these devices and use them (with these rules) to answer your user.

The way I describe it in my prompt engineering seminars… Assume you’re explaining a task to your grandmother and want her to look at the data and give you an answer. You have to give her context of what she’s looking at else she’ll stare at you and say don’t ask me dumb questions, kid.

You need to tell the LLM what it’s looking at with either your prompt, aliases, an index or SOMETHING so it can interpret the data

I couldn’t tell you what was what with conflicting names. Neither can the LLM without that context.

3

Yes I understand why the conflict exists, and likely my post was a bit all over the place.

Here is an example of a pre-built sentence from HA:

  • sentences:
    - “is [] [the] {zone:state}”
    response: one_yesno
    requires_context:
    domain: person
    slots:
    domain: person

Essentially, what I would like is to be able to query the area:state using the same type of prompt. The person entity, even when I add a room location sensor to it, does not seem to be able to output the area.

1 Like
4

You will need to write your own intent that when that sentence is triggered.

[where] [is] [find] {person}

Matches to custom intent let’s call it find_person.

Make it accept slot name {person}

And then returns the preferred answer all without the llm (i have a voice pipeline using plain assist to test these)

THEN once it works there - enable LLM. (use your llm pipeline) and it should ‘just work.’ until you do that the llm will stumble all over itself.

The llm ONLY provides translation from speech to text or text to speech in a creative manner but can only use a tool if the tool works.

This one

1000004189
10000041891080×2340 145 KB

is asking mine about ‘room details’ for the Garage. This requires that I created a custom intent that accepts room (area) as input and I told the LLM that there is such an intent called room details how to use it and what it returns. (and note it errors at the end and adds some tidbits about water utilities)

This one I have an entire section in the prompt describing the kinds of information I want when I want a home report. The LLM correctly identified when I said Whats up that I wanted a report. But the it ran to the intent I built to grab the ‘report data’ (note the room presence in there - that’s Bermuda) and the spit it out in ‘plain text’

1000004190
10000041901080×2340 169 KB

5

Thanks. This is helpful, though i obviously need to read up some more.