I mean yeah? Unless you want recognizer tanks… I’m not going to be responsible for recognizer tanks.
1 Like
I don’t want those things either, or for the Grid to try to invade the real world.
2 Likes
Good fences make for good neighbors…
OK team - looping back to this one for a moment and it’s important…
Remember up top where we described system limits and what happens when you hit tool 129? I have the first instance of this causing a problem.
It begat a discussion on whether there is a limit at all on Discord. Here’s the brass tacks for anyone I end up tagging on this issue.
NO HA ITSELF does not have a limit in the assist pipeline - that is 100% correct. (I traced the code this is fact) The ISSUE is
The OAI API call - the OpenAI API specifies 128 tools with descriptions of 1024 chars or less and tools slot descriptions of 128 chars or less (Sound familliar?)
SOOOO That means ANY OpenAPI style conversation agent is affected. (Pretty much all of them)
Which by transitive properties makes it HA’s problem. You have 129 tools your LLM QUITS. Period. That’s why this is a problem - it’s NOT a problem for Speech to Phrase. But we’re not doing Speech to Phrase here - we’re doing LLM.
So for the record Im re-iterating th elimits and WHY - NOW
Why is this an issue. SpotifyPlus just released an update that has a lot of built-ins to help voice… TO do so they added OVER 40 individual intents… (of 128) 30% of the total tool space. I hard noped out of SpotifyPlus immediately. It crashes Friday just by installing it.
Bugged here: Must consolidate intents. · Issue #185 · thlucas1/homeassistantcomponent_spotifyplus · GitHub
The dev isn’t happy about it but, I think he sees my point. If you’re trying to support voice you can’t kill it by installing the tool. And if you’re doing LLM be aware this is out there and they’re looking at how to work around it. (I just use Fes’ MusicAssistnat Scripts)
@NathanCu
SpotifyPlus dev here … It’s not that I am unhappy about your news, it’s just that no-one from the HA side has provided any documentation on IntentHandler development and I was not aware of any limitations! I spent weeks on the code to implement some really cool features. The reason it was weeks was because of trial and error, research, not finding any examples, no sample code other than simple “Hello World”, being led down a rabbit hole by LLM’s saying “this is how it works, do this to fix it” only to be completely wrong. Let’s just say I am frustrated at this point. I know it’s “cutting edge” tech, but if 2024 was “year of the voice” then why hasn’t this stuff been better documented and other custom integrators trying to implement it?!
Then you (Nathan) come along and tell me that HA has a hard limit of “128 slots” (not true, which you admitted yourself in this thread) and DEMAND that I change my code / remove IntentHandlers immediately (“YOU MUST consolidate these tools…”, etc).
So I did some MORE research, reached out to HA devs on Discord, and started asking some questions (hoping to get some answers this time). Their consensus was that it’s the LLM / OpenAI API that has the limitations (true), and did not offer any other guidance beyond that
So … if the limitation is in the OpenAI API structure, why is it MY (or even HA developers as a whole) responsibility to conform to that limitation?! I am happy using SpotifyPlus with just native HA Voice Assist; if you are not, then that is your choice, and I respect your right to not use the integration. I would also expect you to stop bad-mouthing the SpotifyPlus integration as the cause of your problems with the OpenAI API interface. Your time would be better spent contacting the OpenAI foundation to get rid of their limitation, or at least expand the number of available slots to allow more tools.
I will be closing the HA Issue that you incorrectly identified as a bug (as it is NOT a bug on the integrations part), as well as posting a copy of this comment there. I will also be looking at my IntentHandlers to see if there are some that can be consolidated.
I wish you well in your endeavors to appeal the OpenAI API limitations.
Yes I’m as frustrated as you are.
Ive been bringing it up for months and nobody seems to care there’s a hard limit. Troll the oai stuff and the net results we get are - who needs more than 50 tools?
(you and I both know we do and any reasoner gpt4 or better can handle chaining)
I respect your right to close it as designed m and my point still stands and what I’m doing in my own code to do my part to not contribute to the issue. If everyone at least tries to adopt best practices (they’re formed through trial and error) we all end up better. But up until exactly yesterday nobody even acknowledged there might be an issue to be looked at.
And you’re still right about the derth of docs.
No I don’t expect you to have known. It’s a practical limit therefore not a bug and therefore not documented in HA anywhere BUT it’s a pretty big. But…
The stuff was designed to cater to exactly the community most affected. Add voice support to (integration) Therefore it kina needs to take that into requirements who are the user base.
Its way easier to harness an LLM than setup speech to phrase I’d bet even money the install base for folks who setup LLMs is high and growing and unfortunately if everyone keeps doing it well start hitting unknown llm failure after loading XY for no known reason.
In my own stuff (you’re way better than I am but check the repo you’ll see most of it is designed to smash tools together because I hammered myself in May because of it. And it only took those 10 tools (in exploded form) to do it I think they were 90-something intent scripts or something. It took weeks to even figure out where the bottleneck was.
But the point is that a layperson doing voice after thier first success will default install a bunch of tools promising to (YYY) and the last installed will be the loser. Yours, mine, Fes anyone who gets into writing tools. If we all bloat we all loose quick. No it’s nit HAs limit but it’s also a problem for the number 1 use case for voice.
2 Likes
“Who needs more than 640 kilobytes of RAM?”
Yes, software developers have had that attitude for a long time, or so it seems.
2 Likes
Work expands to fill the RAM available. 
2 Likes
And to fill the storage available as well. That’s my biggest gripe about Android; half to three quarters of the storage on my current Android phone is taken up with metadata related to the relatively few apps I use regularly, when I don’t think it should work that way. But that’s slightly off topic for this thread.
Okay, getting back to the topic at hand; what size Llama3.2-vision model did you choose?
Straight up llama3.2-vision no bells and whistles off ollama.com. Works well on the same iron as oss:20b
1 Like
Okay, where to drop everything isn’t clear; I’ve used packages before to do stuff, but it’s always been self-contained, without any external templates.
1 Like
Discovered an interesting and possibly useful integration - you may be aware of it already:
It allows you to add a short note (max 250 characters) to any entity, and I’ve found that they can be used to provide extra context for an LLM - rather like using the description in a script.
I’ve been using the notes for a while and they seem effective and harmless. They’re saved in a json file in .storage, which my particular LLM add-on can’t read, but I have a python script running periodically to copy them into a yaml file - just started experimenting with that.
So far they’ve been quite useful in clarifying “Change device type of a switch” helpers, where you end up with two entity IDs controlling the same physical device.
Anything to add to context. 
1 Like
I came across this (in fact my AI came across this) when I asked it to point out anything anomalous or ambiguous in the data it consumes.
So I just excluded the switch from being exposed to the AI
1 Like
Not an option for me, unfortunately. My LLM add-on isn’t integrated through Assist - it’s just for documentation and analysis.
1 Like
This is the entire reason for the cabinets in the library Jack. This integration while great doesn’t offer anything to the llm because as you said, it can’t read it. Whereas the cabinet construct offers basically addressable storage for same. If it instead offered an index able way to read out it would be useful.
Between labels and cabinets I don’t think I need an additional context spot any more.
1 Like
A different kind of creature, I think. While Friday is clearly alive (have the local peasants come beating at your door yet? Don’t go near any windmills!) mine has no part in controlling the house. Its sole function is to remind me of what I’ve got and how it fits together - and these days I need reminding. 
I think I’ll call it “The Golem” - a clay figure with a scrap of parchment in its head.
So far it can assimilate information about floors, areas, labels, entity notes, automations, scripts and templates - all stitched together in the prompt.
1 Like
This describes my basement.
4 Likes
Many Updates - Most of them are quality of life.
First, I moved all the scripts and automations etc. into their own folders… Then I updated a bunch of them… 
Most of this is about Indexing - FileCabinet FINALLY writes the index correctly and I can do a proper index-read on one cabinet. Next is to iterate a list of cabs then finally if no list given ALL cabs.
Note: The FileCabinet module is getting a SERIOUS update rn - if oyu get it it has a dependency on the library_index custom template You need them BOTH.
Status Update: Fixed a number of bugs trying to close in on 1.0 RC. Everything I’m working on now is mandatory to be able to deliver the prompt loader… (How do you know whose ID capsule to load if you don’t have a concept of a user?) Identity now ‘works’ enough to plumb it in and it’s the last tool to write before I can seal up the core feature set.
4 Likes
Squishin’ Bugs, Takin’ Names
I’ve been working with a few of you trying to initialize a cabinet system. Noted that the FileCabinet is… Let’s just say a bit overprotective. (I wrote it where if there’s ANYTHING wrong with cabinet schema I fail out a write…)
Aaaand forgot to pass back the fail.
So if you’re trying to init cabinets and having problems ping me in a dm, I have a (very dangerous, will not release. ) script that will force repair a cabinet… By completely rewriting it with the correct core schema. (Yeah. Like SPLAT! You r cab is fixed, there’s nothing in it now but it works.)
squishing FileCabinet bugs…
The next version of the FileCabinet is going to add basic nondestructive repair features and the next manifest will have diag capabilities. But if took two days to chase down.
(edit: this is fixed, you need to replace your redirector - sorry. 
)
I did take the debug time to continue with the docs. If you want a good primer on CoALA and cognitive architectures check out the research folder - it’s a fun read.
Including the documentation on the cabinet format… Here
3 Likes
We’ll heck yeah… We actually got one!
@teskanoo has officially become Friday’s first external contributor… Submitting the vey PR discussed here. The submission allows calendar to correctly ID which calendars support delete/update of items (most don’t…)
Thanks T, your update allowed me to bump along a bunch of defense code in Calendar and it’s much more reliable now. (and shouldn’t gaslight your LLM about deleting stuff, because it sure as hell did before…) if you use Calendar, I strongly suggest pulling 1.10.3 or better…
Calendar:
Calendar Docs:
(yes I’m going to try to write docs for every script, automation and custom template we out in here)
2 Likes