Since we have the feature to have continued conversation with LLM (voice pe / assist), the assist “prefer handling commands locally” option is quite confusing: it works well when the user initiates the conversation.
However, it doesn’t fallback on local intents if the user asks for a “local command” during a continued conversation handled by the LLM (tested with Ollama in my case). That is an issue when the LLM doesn’t have control of assist exposed entities. I guess it’s often the case for users who have local LLM.
So, it would be a nice feature to get this fallback capability during a continued conversation.
This was supposed to be fixed by pull request #139421. However, it looks to me like the ChatGPT integration is handling even simple commands to turn on lights.
I was thinking of checking local intents during a conversation with the LLM agent - I mean in the middle of a continued LLM conversation.
Are you sure this PR #139421 serves the same purpose? It seems to handle the prefer locally sentence setting even if the LLM agent has control on assist exposed entities (well, it seems related to your issue).
As for local LLMs, I’m currently using the HA MCP server as a partial workaround. While it’s faster than giving control to the LLM agent within HA, it’s still too slow.
By the way, another useful feature would be a dataset generator from the Home Assistant instance, which could help fine-tune local models.