I have used home assistant for many years and I believe it’s the best thing since sliced bread.
However, since november last year I’ve struggled to get voice assistants to work with my configuration.
I have spent countless hours trying to get whisper, piper and wyoming protocol to work with an AI model that actually works. There are two integrations that I tested: extended openAI conversation and recently anthropic hass api.
I spent most of my time trying to get extended openai to work properly but to no avail. I must have tried 20 different prompts and 20 different versions of functions. I also tried many different models, Top P settings, temperature settings, function call settings per conversation, context thresholds etc.
I’ve greatly reduced the number of entities exposed to the voice assistant pipeline (down to 130 entities).
I’ve created aliases for all entities exposed with a more meaningful name.
It basically boils down to this:
The response is very slow (even when I use text input to bypass the speech-to-text delay)
Extended OpenAI can turn individual lights on and off but areas already become problematic.
Any entity status is problematic (f.i. is there any motion in room x?)
I also use Amazon alexa in every room and that works without fail. I was hoping that the voice assistant pipelines could allow me to get rid of alexa, allow me to switch to Dutch for all devices and entities and would provide more ‘intelligence’ and flexibility with questions like, 'I anyone home?", "It’s a bit cold here, can you turn up the heater? etc.)
Maybe I’m asking too much of my Home Assistant installation but it looks like I won’t be able to get rid of alexa anytime soon.
If anyone has any suggestion how to achieve this, I’d very much like to know and will greatly appreciate any advice given.
These are the details of my installation:
Installation methodHome Assistant OS
Core2026.3.4
Supervisor2026.03.2
Operating System17.1
Frontend20260312.1
Running on a n100 with 32gb memory
Update: I now use openrouter with anthropic/claude-haiku-4-5 and I’m starting to get some results but it requires a lot of updating the prompt to make it work. I never realized how much work goes into ‘coaching’ a model to get a result.
From my exeperience the LLM makes a huge difference. One thing I am working on at the moment is daily generating an AI image of the past. The first action is to tell the LLM that it should pick a historic event from the past (something an LLM should be easily able to do). All the smaller models failed (using Gemini 3.1 Pro as speed is not that relevant ther) and halucinated about things that never happend.
I experienced similar issuse using Assist. Have you tried it with something like Gemini 3.1 Flash Lite? Had a lot of success with it lately. Very reasonably priced and pretty fast as its a model trained for fast respones but still powerful enough for assists. The OpenRouter integration is pretty great for that as you can easily switch models and test them out.
Thanks!
Could you elaborate on the quality of context? I did a lot yesterday to select the entities to expose and to create consistent aliases. One item I’m struggling with is to define the lighting per room: if I have more than one lamp in the room, I cannot get the LLM to recognize them and turn them all on or off. I assume it’s something that can be fixed in the prompt but I haven’t found out yet.
If you have any suggestions or if your previous posts have suggestions, please let me know!