Hi, I’m looking to make an HA-integrated shopping list / to-do list that I can integrate with ChatGPT. Of course, I want the regular functionality of the to-do list that can be read/modified by voice command or by the Home Assistant dashboard. But I also want ChatGPT integration to sort the list by grocery aisle and translate back and forth between different languages.
I installed the OpenAI-Conversation add-on with the hopes to set it as my voice assistant and expose the Home Assistant To-Do List to control my shopping list. However, I was disappointed to find the voice assistant doesn’t have access to the items in the list, so it can’t do any sorting or translating.
After lots of Googling, I’m struggling to find inspiration on another path forward. I was just wondering if anyone here has experience with this, or maybe just some keywords I can add to my Google searches.
But I’m VERY verbose in the prompt telling the LLM about what things are. The second image is a picture of the UI for a self hosted version of Mealie - that is driving the actual shopping list, integrated into HA as shopping (ToDo) lists and that picture was the direct result of the ask.
What I mean by that. You can’t just assume adding llm just knows how to do stuff with no context. It may see the intents but it may not know how to apply them
What have you added to your prompt to explain to the llm what each of your lists are and what they contain or are you running defaults?
Whats going on in your case is the intent to read the list exists (see image) but your llm hasn’t made the connection that that’s exactly what you asked.
In mine I combat that with nearly 8 pages (single space) of typewritten instructions and somewhere in there I explain to the ai that xyz are the shopping lists and they contain items, please use the standard intents to manage the including adding and removing items from the list.
Interesting. How did you expose the list contents to Assist? I’m using the Anylist integration, which helpfully adds the list contents as attributes on the entity - but I can’t see how to expose those attributes to Assist. The only way I’ve managed to do it so far is to create a new Template sensor to read the attributes, and then add instructions to the prompt so that the llm knows to use a different entity to read the list contents. But this is not a satisfactory solution.
I’m having the same issue… Using local to-do or shopping list and ollama api to a local ollama agent. The assistant will do a lot of things, even give the number of items like above, but is adamant it can’t read the contents. Asking to add to a list has resulted in a hallucination, admission of failure, or what I believe is a HA response that it failed to use the tool. Something similar is happening when I ask to set light brightness.
Other things, blinds, lights on/off, etc work fine. But, I’m struggling with lists and light brightness, so I’m not rolling it out to the family.
Edit: I’m running the default system prompt. I searched for hours but couldn’t find any documentation or working examples that seems to help with these. If you have them, sharing them would be helpful…
I´m afraid that in the entity what is exposed to LLM the content of the shopping list is missing. So LLM has no information what is included in shopping list.
Shared to LLM is only amount of items in shopping list.
The problem is to get this information out of the action so the LLM can access it. I tried automation to trigger the action expecting the data would be available for the LLM but without success.
I would also love to know how to grab that information.
@NathanCu Others nailed it, but, yes, it seemed to me like the Home Assistant to-do list just doesn’t have access to the individual items on the list. See the attached image. For clarity, do you believe there is a way to fix this with improved instructions to the LLM or do I need a different to-do list integration (Mealie, for example)?
Im looking at this today. I’m definitely seeing some differences in how lists are stored v. ToDo items. Some scenarios look like they’ll need a tool while if your llm sees the items in the ToDo as attributes and can call it that way it can get them… Haven’t figured out the rhyme or reason yet… Should be able to work around it.
In any case I think im handling it long term like how I handled calendars. For the ones I think I need to preload I’m building a triggered template sensor that stores ‘events’ as a bunch of attributes (the old way it was done)
Ill probably just do that for any list I can’t read from get_tasks() (which is what I think is trying to read them now, and if your list is stored that way you get them)
Im writing this now, haven’t forgotten. There DOES need to be an intent at least for SHOPPING lists, looks like there’s a difference in how the system handles a shopping list v a task, and in some task lists you can just roll out the attributes of a todo item and boom there’s the data.
Part of the difference comes in December-Jan timefroam todo base handling became part of HA Core.
For shopping lists: this is doing it for me. My original response of I didn’t do anything special is incorrect because I installed this to handle something entirely different and it didnt click that shopping was being piped through these. And when I finally dug in - the LLM was using it anytime I asked about a shopping list. (Confirmed through the debug logs)
get_tasks:
parameters:
name: # name (required) of an existing, valid task entity to read
required: true
action:
- action: todo.get_items
target:
entity_id: >
{%- if (name == '') or (name == '*') -%}
{% set entity_ids = states.todo | map(attribute='entity_id') | list %}
{{entity_ids}}
{%- else -%}
{%- set dot = name.find('.') -%}
{%- if dot == -1 -%}
{%- set entity_id = 'todo.'+ slugify( name | replace(':','') ) -%}
{%- else -%}
{%- if name[0:5] == 'todo.' -%}
{%- set entity_id = name -%}
{%- else -%}
{%- set entity_id = '' -%}
{%- endif -%}
{%- endif -%}
{{entity_id}}
{%- endif -%}
response_variable: result # get action response
- stop: ""
response_variable: result # and return it
speech:
text: >
{%- for todo in action_response %}
{%- for item in action_response[todo]['items'] %}
list: '{{todo}}'
item: {{item.summary}}
due: {{item.due}}
status: {{item.status}}
description:
{{item.description}}
{%- endfor -%}
{%- endfor -%}
mark_task_complete:
parameters:
name: # name (required) must contain an existing, valid todo entity to target
required: true
item: # an existing, valid task item (required) to be marked completed
required: true
action:
- action: todo.update_item
data:
item: "{{item}}"
status: completed
target:
entity_id: >
{%- set dot = name.find('.') -%}
{%- if dot == -1 -%}
{%- set entity_id = 'todo.'+ slugify( name | replace(':','') ) -%}
{%- else -%}
{%- if name[0:5] == 'todo.' -%}
{%- set entity_id = name -%}
{%- else -%}
{%- set entity_id = '' -%}
{%- endif -%}
{%- endif -%}
{{entity_id}}
speech:
text: >
'{{name}}': '{{item}}'
'status': 'completed'
So these two get it where you can add them and mark them off. the other parts are good context building in your prompting.
Where/how did you install these? I tried pasting them as scripts, but they don’t seem to match the script syntax. Thank you for all your help–I think I’m almost there.
