Im not one to usually post on forums, but Im going to post this in case anyone is wondering how to do it. I got my home assistant voice preview edition to talk to my ollama llm conversation agent. I run Piper, Whisper, Ollama, and Home Assistant in docker containers. I start my automation by saying, hey nabu, followed by, ask ollama, followed by whatever i want to ask. The behavior is similar to asking alexa questions.
If ollama (or any other llm) is your primary agent for assist, then you don’t need additional automation. Just ask a question after activating the assist.
But your way makes sense if you want to limit false positives. You limit yourself to the standard set of commands (Home Assistant agent) and add custom sentences for some template requests.
Or if the task requires processing data using a specific model.
When i tried just asking my assist sometimes it would respond with function { some data } instead an answer and others it would time out responding. This way forces assistant to go to conversation agent and reply back with everything it responded with. I’m not asking it to control anything in the house, it’s like opening up chatgpt and saying something you would normally google.
A lot depends on the model used. Promt can also affect the answers.
Which one do you use?
Try downloading gemma3 and ask it the questions you have problems with on the current version.
I use something like this, but I have it as a regular script that I expose to Assist.
My main conversation agent is Ollama with a local LLM, but I have also set up an OpenAI agent with GPT-4.5 (without the ability to control HA), which Ollama can decide to invoke via this script. Essentially, it’s another tool in its disposal.
I’ve provided instructions in the script description, informing the LLM that it can ask GPT if the question is particularly hard, or if I explicitly ask it to “Ask GPT”. If it invokes the script, it is instructed to relay the response back while prefacing it with “Here’s what GPT said”.
I have not used sentence triggers, maybe that is a better option. To be honest, I kind of like the fact that it’s the LLM itself making the decision to invoke the tool, as opposed to a deterministic sentence trigger.
If anyone’s curious, here’s the (very simple) script. I don’t have much experience, so let me know if it can be improved:
alias: Ask GPT
sequence:
- action: conversation.process
metadata: {}
data:
text: "{{prompt}}"
agent_id: conversation.chatgpt_expert
response_variable: gpt_response
- variables:
result:
instructions: >-
Make sure to say that you got the response from GPT. Preface the answer
with "Here\'s what GPT said: "
response: "{{gpt_response.response.speech.plain.speech}}"
- stop: Complete
response_variable: result
fields:
prompt:
selector:
text: null
name: Prompt
description: >
The query prompt to ask the expert model. Set this to the full question
with all the required context
required: true
description: >-
You can use this to ask an advanced GPT model for assistance with harder
questions, especially when the users asks the assistant to Ask GPT.
mode: parallel
max: 3
