There’s a difference between knowing something and when to use that knowledge.
Try asking a question where using the labels function is the best option, without priming it, or give it a solution from a time before labels and see if it will improve it using labels.
No LLM knows anything. They’re just statistical models showing which words normally come next to each other. They work as well as they do because of the vast amounts of data they’re trained on.
They’re as plausible as they are because that’s the way people think too. 
OK. Moving on.
Changed the OpenAI model to gpt-4o-mini, which has a higher TPM limit. (Apparently in English a token is about 4 characters.) Still using the default prompt.
This is much better, but still an issue.
I’m assuming that the errors are partly down to the lack of a prompt, so that I’m getting a large amount of LLM waffle.
Observations about usage:
The fact that the conversation is cleared when you navigate away from the web UI is becoming increasingly annoying. I find I often want to cut and paste things like entity IDs from elsewhere. You can do it in a browser, of course, with two windows open.
When there is an error it would be useful to be able to edit or resend the question. At the moment the field is cleared after each send.
And a question…
I have a number of tools set up for a voice AI along the lines described here:
If I include them in the prompt, will they work? They usually invlove running a script and the description fields in the script may contain extensions to the prompt.
I asked it what its training cutoff date was. October 2023. This is gpt-4o-mini, of course.
100%, I know that very well, since I work with the tech. I was pointing out that the previous poster’s proof didn’t actually say much. You need to ask a question where it needs to come up with a solution using the latest information. Merely having access to the information doesn’t mean it has learned from examples in order to apply it – this relates to the point of these models being quite a bit behind in general.
My “proof” was just to point out that Nathan’s comment “I suspect it does t know anything about labels() or label_entities()” wasn’t true ( ignoring the weasel word “suspect”. Maybe LLMs will become first class citizens when they start using weasel words 
). I didn’t go any farther because I just wanted to see if people, like the LLMs that emulate them, are occasionally wrong. Turns out they are. That doesn’t reduce their value. We’ll cherish Nathan in spite of his missing keyboard keys from the statement above
It totally missed it not only a month ago. The point stands. You MUST ACTIVELY manage your prompt for this right now. Because it lags and you don’t know when it will make it into the training corpus.
I flat out argued with Veronica (my debugger, gpt-5.0) that they were valid. Not even four months ago.
I guess that’s a point of discussion. Language is hard. How do we define “it”. For example, I don’t go to chatgpt.com and start typing, I use the awkwardly named “Home Assistant Assistant” agent that someone posted in the forum months ago. I’m sure it has better prompt language and references to the forum and documentation behind the scenes. Maybe there are ways to see what was input to generate the agent. In my experience it does a pretty decent job.
If nothing else, LLMs are forcing people who do get value out of these systems to improve their reasoning and communication skills by learning “how” to ask the question in a way that gets the best answer. This can be helpful in dealing with everyone, not just computers.
Well… as an experiment I asked the Configuration Agent how many entities were labelled “sos” in my Home Assistant instance (the correct answer is 8). It thought for several minutes, then generated a response of 777937 tokens, and a TPM error. I take this to mean that it had no idea what I was talking about.
I then added a prompt which included a link to the docs on labels, restarted and asked again. It replied immediately “There are 16 entities that have the label “sos” in your Home Assistant instance”. Double counting for some reason, but at least it knew what I meant.
When I asked it to check again, it said “Upon reviewing the entities again, there are 18 entities that have the label “sos” in your Home Assistant instance”.
So… probably couldn’t find the information, hallucinating, but at least it knew what I wanted.
Edit: On further investigation I find that gpt-4o-mini doesn’t actually follow links given in the prompt - you have to paste in the text fom the linked page.
So presumably the Config Agent deduced from my prompt that labels were a thing, but otherwise was none the wiser. 
No llm will in the initial prompt.
In practice then, I’m going to use my entire token allowance on the prompt.
Now you know why I wrote the summarizer. 
If you can’t use gpt-5-mini I would recommend using OpenRouter with google/gemini-flash-2.5. It has the best response time and is very cheap to use. It also worked great with the tools.
@jackjourneyman Thanks so much for testing out my add on!
I could put a whole history component into it but I was trying to keep it super simple. I guess I could make it so you could save conversations to your browser so you can resume where you left off. Be careful though, your entire conversation is in context with each request to the LLM so you can grow quickly by having big conversations. Thoughts? How would you prefer it worked?
Regarding integration with tools exposed to Assist, no this add on won’t work with any of those tools. I would have to make some generic way to expose a tool that would expose scripts to the LLM.
Thank you. Seems to be a choice between selling your soul to Microsoft or Google…
I just released version 0.1.6 that enables “Prompt Caching” Prompt Caching | Reduce AI Model Costs with OpenRouter | OpenRouter | Documentation you can save some money with both Gemini and Claude models (both supported through OpenRouter). They make long conversations or ones where you had the LLM read your entire HA configuration cost you substantially less (1/4).
This version also fixes a number of bugs that were preventing new files from being created (and other issues).
Enjoy!
And I found my first critical security defect. Please upgrade to 0.1.7 (or greater if I’ve released it by then). Previous versions can leak your secrets.yaml files to the LLMs!
My AI service of choice is Perplexity. That keeps a “library” of all your conversations and you can resume any of them. It also has “spaces” where you can collect conversations relating to the same research topic - this is just for the user’s benefit, though, the AI makes no use of them.
I’m not saying that your add-on needs anything that elaborate, but it’s responses do seem to be rather ephemeral. Given that LLMs in HA are problematic anyway, I think I would need some sort of record of conversations, even just a .txt file.
I think your add-on is great, by the way. But like most of the people commenting in this thread I have concerns about the damage LLMs could do, both to users’ installations and to confidence in HA. It’s a philosophical thing.
Personally, I think people are going to use AI whatever we say. I’d rather have them use your add-on than some random chatbot. 
Ask and you shall receive!
0.1.8 is released
This now includes Import and Export conversation functionality. Click the Export button and download a pretty printed JSON export of your conversation and all tool use. Click import and upload a file you exported to resume a prior conversation.
Wow!
Where are the files stored? When I click import it just opens a local folder on my laptop.
Edit: When I use a browser to access HA, exported files are stored in the Downloads folder, as you’d expect. Using the android HA app on the same machine (a chromebook), I’m not sure where they go.