Trouble getting voice to execute / comprehend commands

Configuration Voice Assistant
1

Hi,

(Context)
(Sorry for the long post. This part is not that important)

Let me preface this by saying that understand that it’s a work in progress and that it will get better. It’s already amazing how fast it got to where it is, but overall I already spent almost 15h between watching videos, reading online, checking parts and setting up everything. So I’m trying to decide whether to keep spending time on it now or revisit the topic in 6 months or so. The videos I saw (Network Chuck, etc) made it seem like it works way more reliably that it does in my experience.

With all the hype online about the voice assistant device. I decided to give voice a try. The voice assistance preview is out of stock (I’ll most likely buy one just to support home assistant when it’s available) so I just setup everything and tried it via the android app. If I get it working reliably enough I might setup a Wyoming satellite until I can get my the preview.

Right now, I have Google Home linked with HA and it can control all my devices really well. I don’t care that much about the local / cloud part. I was just hoping to get a smarter voice assistant and maybe have small conversations with it like I do in open-webui.

(Setup)

I already have an local AI server with a 3090 so I set it up to use llama 3.2 in Ollama. I also setup remote whisper and remote piper. That whole part works fine and it’s pretty fast.

(Problem)

My problem is that once the command is parsed voice assistant has a lot of trouble executing the commands. The voice commands are transcribed correctly. For testing I only added two devices for the voice assistant: Thermometer (aquara thermometer) and Office Fans (a tariff plug).

When I test the sentence in developer tools I get matches for the devices.

Here are the problems that I encountered so far:

  1. Turning on the office fans. I had to try a bunch of stuff. I finally got it to work by adding an alias Office Fans (same as the device name).

Anybody knows why it couldn’t just find it directly?

  1. Getting the temperature. I tried changing the name to temperature and adding aliases, but nothing worked. That one I can’t get it to work. Sentences:
  • “What’s the office temperature” → no climate device found.
  • “What’s temperature” → no clima device found
  • “What’s the thermometer” → no climate device found / if it type it rather than saying it it works even tough the voice command text when transcribed is exactly the same.
  1. Ask the assistant about old information / small talk.
    Ex: What are the 5 tallest mountains in north america?

A third of the time it works. A third it tries to execute random commands like turn off, turn on imaginary devices. The other third it tells me that it doesn’t have access to the info.

All these questions work perfectly fine when I run them in open-webui with Ollama and llama 3.2.

Has anybody else experienced these issues? If so did you find solutions for it.

Thank you

2

Correct naming and binding entities to areas will help solve many problems.
If you want to see all the command options, you should look at the project’s github.

You are requesting the thermostat temperature value. For a separate sensor, you should use the name . The command for getting the value in the area is not yet implemented.

I can offer my template, if you really need it, but don't ask what to do with it

{{ area }} temperature is {{ states.sensor|selectattr(‘attributes.device_class’, ‘eq’, ‘temperature’)|selectattr(‘entity_id’, ‘in’, area_entities(area))|map(attribute=‘state’)|min }}

Select the “no control” option for LLM, then it should correctly answer the questions.

In summary, everything works stably, but requires a higher entry threshold. Video alone is not enough, you need to read the documentation

3

Thanks for the reply.

Based on what you said about a higher entry threshold and some more research I did online. It seems like it might require a bit more prep / organization for all the devices that I have in HA to make it work. I’ll thinker with it a little bit more. I mainly use it for basic stuff and it looks that right now the assist option might not really be worth it for me since my google home actually works fine without having to add areas, alias etc.