I use ollama localy and have tried many models like functionary-small3.1 and llama3.2:3b. These models are not the best but they should be good enough for decently reliable tool calling right. I cant seem to get them to control anything but simple switch devices. even a multi color light is to much for it.
I think it has somthing to do with the Ai not understanding how to use toolcalls but I dont know what to put in the system prompt to get it to understand.
So in short you’re kinda right. It’s a context problem. Solvable. There are a couple of posts you will need to read. They’re both ridiculously sprawling but tell the same story
You can’t just drop in an LLM and expect plug and play magic.
And thyraz’ post
Now as to your device issue
The prompt context does not contain info about ‘devices’ it does ‘entities’ … So are you matching the right item.