Ollama integration (offcial one) How to send data to the LLM?

[ToonK (Toon)]

I think I tried all possible ways and workarounds to send data to the Ollama voice assistant.

Can someone please tell me how I can do it? I have a function I call that should output the result back to the LLM, because it can run the script itself.

I also tried using a sensor template, but while I can expose it, the Ollama voice assistant is not able to see the entity, and will make stuff up (hilarious, but not when you try to debug something).

I also tried using the conversation.process directly (it seems it does not see anything from that). I tried having the conversation.process run my intent to run the accompanied intent_script to run a speech command, but appearantly allthough it retrieves the intent function name, it gives an error, and I don’t even know if that would have solved anything.

Help

[Sir_Goodenough ((SG) WhatAreWeFixing.Today)]

How to create multiple phrases to send at random to TTS?

I took Greg’s code and did it for Ollama and llama3.1 locally. get_ollama_response: alias: Get ollama response sequence: - data: agent_id: conversation.llama3_1 text: Prepare a {{tone}} notification about "{{subject}}" using no more than {{length}} words and not using any emoji. response_variable: llama3_1 alias: ollama Prompt enabled: true action: conversation.process - metadata: {} data: value: '{{llama3_1.response.speech.plain.speech | tr…

[ToonK (Toon)]

Super! Thanks! It looks like the input textfield for communication? I’m going to carefully study this!

[ToonK (Toon)]

Thank you so much!! I got it somewhat functional, however I have a question. Is there a way to direct the response that now goes to the (helper) input_text.llm_response to go to the actual voice conversation? It almost seems like I have a split personality like this.

It seems currently I have it setup so my voice assistant Ollama can run a script and that processes stuff, and then the output of that process is fed to this automation and into the llm_request. There comes the weird part, there seems to be a ‘split personality’ since there seems to be another Ollama entity that is replying in the llm_response to the content in the llm_request.

I created this automation:

alias: LLM Broadcaster
description: Broadcasts to LLM (Ollama)
triggers: []
actions:
  - data:
      agent_id: conversation.llama3_2
      text: "{{ broadcast_text }}"
    response_variable: llama3_2
    action: conversation.process
  - data:
      entity_id: input_text.llm_request
      value: "{{ ( broadcast_text | trim | replace('\"', '') )[:255] }}"
    action: input_text.set_value
  - data:
      entity_id: input_text.llm_response
      value: >-
        {{ (llama3_2.response.speech.plain.speech | trim | replace('"', '')
        )[:255] }}
    action: input_text.set_value
mode: single

[ToonK (Toon)]

Ok, it seems it sends it to the wrong ‘instance’ of Ollama. The agent_id is fine, but it seems I can run multiple instances on the same model, there must be some kind of unique id of the conversation I have via the HA Mobile app, not sure if this could be a current conversation id, or something similar. Would really appreciate some help!

[ToonK (Toon)]

If anyone reads this, I was able to figure out that with setting up a response_variable I was able to get the return from the pyscript back into the script in Home Assistant. (without the need for a textfield workaround)

However, the Ollama LLM voice assistant is not able to ‘see’ the contents of the result.

(the result is parsed back, since I’m able to see it in my notify.file output)

It seems that the Ollama LLM voice assistant that I ask to run the function in Home Assistant is not able to see the contents of the input_text or a template sensor.

Anyone have any hints in what I could do?

[ToonK (Toon)]

Again talking to myself, I finally found out that there was an issue with the response_variable not being returned to the LLM. Appareantly there was a topic, a bug report and it seems marked at *solved and the topic is closed.

WTH – Why aren't response variables from scripts available to conversation agents?

Thanks for the reply! There are always files present in the directory with those names, new ones are overwritten so worst-case it would be interpetting outdated images. But no that’s not what’s happening, I can run the script manually and everything works as intended. I did have it set up with the stop action originally but it didn’t change anything so I removed it. I put it back in just now, still with no change. It’s as if the script is not exposed; Assist does not even attempt to run the scr…

However, I still am unable to get the response_variable to my LLM (and yes it contains the results as I can view them in my notify.file and output them to a input_text)

This is getting really disheartening, since every single thing or workaround I tried seems to end up in nothing, so far I learned:

Ollama:

-can’t view the response_variable (or it is a timing issue? It does say that it was a succes, but that the actual result is an empty object *which it is not)
-can only view the initial state of a input_text at the start of the conversation
-cannot run the get_state() command or anything similar, so it’s not able to ‘see’ what’s in a text_input or even a sensor template, beyond the initial state of the conversation
-same seems to apply form the sensor template, and it’s also not able to see any custom parameters from that entity.
-cannot see exposes automations
-says it can’t list entities?
-does not receive any information trying the conversation.process function (see code previously pasted) with the correct model ‘agent_id’, since it seems there are multiple ‘entities’ of this agent_id?
-is not able to receive more than 4 lines of text, because then my speech-to-text stops (would be nice if you could alter this amount), to clarify, I use the voice-assistant on my phone.

Do I keep talking to myself here, or should I just add stuff in GitHub? How can I move forward?

[strato]

Can you post the latest version of your script/automation yaml?

My issue was that I renamed the script and also changed the script ID. Apparently there is a bug where if you rename a script, Assist LLMs may lose the ability to access the script as a tool even if it looks exposed in the UI.

Duplicating my script, exposing the new one, deleting the old one fixed my problem.

I should also note, prior to release 2024.12, LLM assistants were not able to see script responses or variables. So this is a new feature.

The biggest difference between our setups is that I am using OpenAI whereas you are using Ollama – this thoeretically shouldn’t matter if everything else is working correctly.

The last thing I’ll suggest is trying to get this simple blueprint working with your LLM. If it does work, reverse engineer and see if you can apply the logic to your usecase.

[ToonK (Toon)]

alias: Check Memory
mode: single
sequence:
  - data:
      id: "{{ id }}"
      title: "{{ title }}"
      value: "{{ value }}"
      category: "{{ category }}"
      subcategory: "{{ subcategory }}"
      subsubcategory: "{{ subsubcategory }}"
      priority: "{{ priority }}"
    action: pyscript.search_memory
    response_variable: results
  - data:
      entity_id: input_text.llm_request
      value: "{{ ( results | string | trim | replace('\"', '') )[:255] }}"
    action: input_text.set_value
  - target:
      entity_id: notify.file
    data:
      message: |
        {% if results %}
          Memory check (notify.file) complete:
          {{ results }}
        {% else %}
          No data found.
        {% endif %}
    action: notify.send_message
description: >-
  This script checks or searches for memories based on the given title, value,
  category, subcategory, subsubcategory, and priority. It sends the results to
  the LLM through the conversation.process service and logs them in notify.file.

This is what I use with a pyscript the output of the notify.file is something like this:

2024-12-15T13:41:17.172224+00:00 Memory check (notify.file) complete:
{‘results’: ‘445 How to Evolve as a Home Assistant AI This is a sample memory value Test Category Test Subcategory Test SubSubcategory High’}

[ToonK (Toon)]

SUCCESS!! I was finally able to get the response_variable to the LLM voice assistant!

Thnx to Balloob on github providing me with the blueprints!! (Also thanks to Defes, for providing them!) Offcourse I didn’t really know how to implement the essence of the bleuprints into my code, so my good friend ChatGPT told me to use this (and it worked!):

alias: Check Memory
mode: single
sequence:
  - data:
      id: "{{ id }}"
      title: "{{ title }}"
      value: "{{ value }}"
      category: "{{ category }}"
      subcategory: "{{ subcategory }}"
      subsubcategory: "{{ subsubcategory }}"
      priority: "{{ priority }}"
    action: pyscript.search_memory
    response_variable: results
  - data:
      entity_id: input_text.llm_request
      value: "{{ ( results | string | trim | replace('\"', '') )[:255] }}"
    action: input_text.set_value
  - target:
      entity_id: notify.file
    data:
      message: |
        {% if results %}
          Memory check (notify.file) complete:
          {{ results }}
        {% else %}
          No data found.
        {% endif %}
    action: notify.send_message
  - stop: ""
    response_variable: results
description: >-
  This script checks or searches for memories based on the given title, value,
  category, subcategory, subsubcategory, and priority. It sends the results to
  the LLM through the conversation.process service and logs them in notify.file.

As you can see the response_variable is duplicated at the bottom together with a stop.

So, I now have a working pyscript that the LLM can use to store memory items in a SQLite database and now finally is also able to check their memories, super fascinating! If anyone is interested I could clean up the mess and share it?

[kombihacke]

Hey Toon,

What you did here sounds interesting! I was actually searching for a way to have a “long term memory” of my LLM.

In my configuration I have one Raspi running homeassistant and another Raspi running ollama with the LLM. I am using the Ollama integration to communicate between theae two.

Can you explain me (in more detail) what the necassary steps are to get a Sqlite database with the entries (generated by your automation?) running? Can I still use the Homeassistant companion app to write with my LLM but with long term memory?

Best regards,
Kombi

Edit: How does your pyscript.search_memory look like, can you share it?

[ToonK (Toon)]

Ok, yeah I have a fairly similar setup, I am using my laptop for Ollama, and have an RPi4 for HA itself.

What I did to play with the memory function, was use a script to interact with a pyscript. The LLM (hosted in Ollama) would be able to run functions in the script (that would call a function in the pyscript, that would deal with a database).

However, it was nice playing around with it, but I also concluded that this is not the way to create a long-term memory for an LLM. What I noticed was that I needed to instruct the LLM too much for it to understand when to add something to it’s memory and by the time it understood how to use the arguments (this depends on how ‘smart’ your LLM is) it had forgotten what we were talking about.

I’m now looking into a more automated way with using LangChain and somekind of memory system. But I have not yet come around with getting this all set up.

I do however think that creating a memory function would greatly benefit the whole experience, so perhaps it would be nice to have more people involved in realising something like this?