Going all-in with Voice Assistant (help needed)

[Thyraz]

@Sawtaytoes

@NathanCu I’m using Home Assistant Cloud with “Ok Nabu” as the keyword and “prefer local”.

I set up Ollama with a bunch of different models and the only decent experiences were with llama3.2 and qwen3. Anything else I tried had a lot of trouble. llama3.2 was the best overall, but it seemed to get worse as I used it. Like it would respond correctly a couple times, then suddenly gave me weirdo responses a minute later for the same question even with a new context. In the end, I decided to avoid any Ollama setup for now.

I also tried changing the prompt, but it seemed like that only made things worse.

After that I’m still not sure what you use now for processing the commands itself.
Is it another LLM model now?

If you’re not sure if the problem is your local model, or if HA simply doesn’t provide the needed tools / capabilities for your requests, you could try one of the cheaper cloud models (e.g. gtp-<any-number>-mini would be a good starting point).
Once you got things running here, try a local model and find out what’s not working anymore and investigate why.
Might be easier than trying to solve all problems out there at once.

As you seem to use LLMs instead of speech to phrase based on your comments, I would suggest also disabling “prefer local” for the time being.
Otherwise you’re fighting against 2 systems, where one or both might not support what you’re testing and you end up with mixed results.

What you then need, is a good prompt to tell the AI important stuff about how you want to control things to get repeatable results and how it should behave / control things.

The other part are user intends / scripts that the AI can call to do what you want.
HA simply hasn’t everything on board so far to make power voice users happy.
That might change over time, as they are adding more and more intends.

Even a search tool for entities is very helpful if you expose a lot entities, so the LLM can easily lookup the needed ones for rooms, user tags, …
Home Assistant only provides a large lists of entities with state and some attributes, so based on how “smart” your LLM is and how large the entity list
Otherwise simple requests like “turn off all lights in the living room” might not always work as expected.

Once you're at that point you should be ready to add your own scripts to add "capabilities" to your assistant.

Not sure how Tater’s setup is exactly structured. I’ve seen it’s a whole addon, so wasn’t able to find any single scripts at a glance.
Is it using HA intents / scripts Tater, or are you using a different approach to control things externally, as your package also adresses other things beside HA?

Nate’s Friday’s Party thread that he linked above, offers the most advanced collection of tooling and prompt shared in the forums.
He’s also bundling up everything for an easier start to use it at the moment.

About your music control: You most likely use the script from TheFes to control Music Assistant (as it offers more options compared to the internal Music Playback script, at least this was the case until 1 or 2 months ago)?

I wrote another script to do music search capabilities for the LLM instead of just playback by search terms, which opens new possibilities.
Wrote about that here.

If you need even more inspiration on some simple scripts or prompting, I try to write down all the problems / solutions I run into while setting up Assist to use in our house:

https://community.home-assistant.io/t/about-making-inexpensive-models-smarter-by-providing-tools-and-context-local-models-gpt-5-mini-gpt-4-1-mini-gpt-4o-mini/

Edit, about your list:

1. “What time is it?”
   Should be possible out of the box with an LLM based assist.

2. “What day is it?”
   Should be possible out of the box with an LLM based assist.

3. “What’s the temperature [outside]?”
   Needs some prompting how it should retreive that to be reliable and always the same way. Also tell it in the prompt if you want to use a temperature sensor or a weather entity for questions like that (which also needs a tool if that didn’t change recently. I use this one).

4. “Turn on Exhaust Fan for 10 minutes”.
   I think delayed commands are already possible with a default intent from HA. Not 100% sure, don’t use this often myself.

5. “Broadcast (or Announce) Dinner’s ready!”.
   Will need a script to do TTS to a media player in a specific room.
   Also currently on my todo list.

6. “Drop in on Kitchen”: intercom feature.
   As you mentioned, this is most likely not easily possible atm.

7. “Find my phone” feature (rings the phone by calling it or playing an alarm).
   A script that allows to send notifications to mobile phones would allow for that too, if you allow the script to also send criticalk notifications (and allow this for the HA companion app in the iOS settings).
   Then you can send notifications with sound that is also played if the device is muted.

8. “What did you do?” for recalling the last action.
   An LLM based assist should be able to tell you that without additional tools / information. If not, something is broken with conversation history in your LLM setup or the LLM integration you use (which means each request will become a new conversation and the LLM won’t have any information about what you asked before)