After starting to use the Voice Assistant, I quickly realized that it takes some time and dedication to get the Instruction-prompt just right. I started to think about ways to improve this, and came up with a feedback loop which instructs the LLM to author instructions to itself any time it is corrected by the user. This is especially powerful after the recent improvements that allow asking the LLM follow-up questions.
My solution was to give the Voice Assistant access to a “persistent memory storage” by allowing it to write from a (named) To-do list and feeding that text back into the Instruction-prompt.
Examples
- The first time I asked to turn off all the lights in the outside, I noticed that the lights on the garage were still on (because I never tagged them as “outside”).
- In the follow-up conversation I simply stated “When turning off the lights outside, also turn off the lights on the garage”.
- The Voice Assistant then turned off those lights too, plus it created an instruction of “When the user asks to toggle the lights outside, toggle those in the garden and the garage”.
- After asking to enable the “air circulation”, the Voice Assistant did not understand me because I’ve named the device “ventilation”.
- I replied that I meant “ventilation” after which an instruction “When the user asks to enable the air circulation, enable the ventilation.” was added (and the air circulation was enabled).
Configuration
- Create a new To-do list named
ai_persistent_memory. - Add the below input_text, script, and automation to your YAML configuration.
input_text:
ai_persistent_memory:
name: AI Persistent Memory
max: 255
script:
update_ai_persistent_memory:
alias: Update AI Persistent Memory
sequence:
- service: todo.get_items
target:
entity_id: todo.ai_persistent_memory
data:
status: needs_action
response_variable: ai_persistent_memory_response
- service: input_text.set_value
target:
entity_id: input_text.ai_persistent_memory
data:
value: "{{ ai_persistent_memory_response['todo.ai_persistent_memory']['items'] | map(attribute='summary') | list | tojson }}"
automation:
- alias: Update AI Persistent Memory
trigger:
- platform: time_pattern
minutes: "/5"
action:
- service: script.update_ai_persistent_memory
- Add these instructions to your AI based conversation agent by going into Settings | Voice assistants, opening your assistant and from there going into the settings of your Conversational agent and pasting the below into the Instructions.
# AI Persistent memory
When the AI misunderstands a command and the user provides a correction, it must carry out the corrected action and also add a new instruction to `todo.ai_persistent_memory`. This instruction should describe the misunderstanding and explain the correct behavior to follow in the future. New instructions are always added to the bottom of the list and cannot overwrite or remove existing ones. Each item should be general enough to help prevent similar mistakes later. The goal is to enable the AI to learn from corrections and improve over time.
# Existing memory
{% set items = states('input_text.ai_persistent_memory') | from_json %}
{% for item in items %}{{ loop.index }}. {{ item }}
{% endfor %}
Outstanding
- Entity states can only contain up to 255 characters, which is clearly not enough. If anybody can provide feedback on how to better convert the To-do list data into a string that can be fed into the conversational agent’s instructions, please let me know.
- Instead of using a
time_patternautomation, the script should run any time there is a change on the todo list. - Some better way to share this and/or make it part of the main Home Assistant software?
