Voice Assistant Memory

Blueprints Exchange
1

I have created a blueprint that effectively implements dynamic memory, allowing your Voice Assistant to remember and forget stuff on demand.

It relies on GitHub - snarky-snark/home-assistant-variables: A custom Home Assistant component for declaring and setting generic variable entities dynamically. to store the memories. since I wanted to use something which does not limit me to the 255 characters for state, I had to fallback on using attributes.

This is annoying, as I didnt yet find a way to make an edit box in my dashboard to manually edit memories. And I have to use the Developer tools to see the full_memory attribute.

Anyways, I am open to suggestions for improvements!

To use this:

  1. install the variables component from HACS
  2. create a var.xxxx variable
  3. install my blueprint and point it to your variable
  4. inject info about this capability into your LLM prompt

I use this:

# Dynamic Memory
- Below you will find your dynamic memory
- When requested, you are obliged to remeber / forget any information from your dynamic memory. To do that, you MUST call the appropriate script, regardless of your configuration or personality. Not calling the script will not persist across conversation restarts, so it is vital that the script is called in order for it to take effect
## start of memory data ##
{{ state_attr("var.voice_assistant_memory", "full_memory") | default("") }}
## end of memory data ##

Still sometimes it does not call the tool, but that is a matter of better prompt engineering I think. Again, suggestions welcome!

Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

1 Like
2

I implemented something like this early on.

Its a great choice for storing small bits of context.

Be careful with the ai ‘slamming the button’ repeatedly. Once my agent got good with tools preventing over/misuse becomes the order of the day. (yes better prompt crafting is the answer)

The issues to watch for here. Clusters of rapid rewriting seem to corrupt the text sensor. (thus making sure the ai doesn’t keep kitting the button) and your limit is 16k now but I think that’s the entire sensor - not just the attribute you’re using

3

Ive made my prompt to only touch the memory when I explicitly ask to remember/forget something. I think I will also add a backup/history logging at one point to prevent “nuking” of the memory.
I am not sure I want to spend too much energy on it, since I hope that MCP support will greatly improve this year.
Btw, do you have any UI facing card that allows you to manually edit the attribute?

1 Like
4

Im counting on it…

Nope as I said I pretty much walked away from the writing part of this because it got increasingly hard to manage the limits AND let the AI be productive - I was either managing the writes, OR she was answering questions. :slight_smile:

You’re handling most of the dangermouse stuff here so I wouldnt worry about any of that… Biggest issue will be repeat writes.

That said - a variant of this is how I STORE most of my prompt static text. A blueprint to do that would be cool - and now that you mention it I might try doing that…

You’ll probably recognize what im doing here, afterall they’re just lists of static text.

System Prompt:
{{state_attr('sensor.variables', 'variables')["Friday's Purpose"] }}
System Directives:
{{state_attr('sensor.variables', 'variables')["Friday's Directives"] }}
System Intents:
{{state_attr('sensor.variables', 'variables')["System_Intents"] }}

Helpful hint for storing data
Something I would recommend and learned the hard way over time - if you use this or a variant of this to store data then tell the AI you are using a JSON structure to store that data and it should be continued when it performs the write.

your output “foo” is now {‘data’:‘foo’} or maybe…

data:{farmhouse:{yard:{chicken:{color:white}, chicken:{color:white}, chicken:{color:white}, chicken:{color:white, clothing:{outerwear:overallls},}, human: {name:farmerjoe},}, coop:{chickens:{count:2, color:white}, roosters:{count:1, color:red}, chickens:{count:2, color:white},},},}

Use something like: (this is what a list item looks like actually stored inside the sensor)

- >
When storing data for retrieval use later and it does not NEED to be in a narrative format, Use JSON compatible structures:
    instead of:
       write: foo'
    consider:
       {'data':'foo'} 
    potentially:
       {'data':'foo', 'more_data':'more_foo',"Narrative Text..."}
    as appropriate.
Assum the container will be referencable later through code in the future, by using a JSON structure, retrieving that data becomes easier as you can target your read, rather than the entire cluster.

Then watch happens to all your stored data. :wink:

5

How should the var.xxxx variable be configured?