I’ve recently configured a fully local LLM on my desktop with 12GB of VRAM. I have tried running 7B and 14B models and they both run pretty well. I have set up some routing via LiteLLM. I wanted to play with these in Home Assistant so I installed the Local OpenAI LLM integration (via HACS) and connected my LiteLLM endpoints. Everything in basic testing seems to work well through Voice Assist with this LLM. Where I am hitting a bump is getting the LLM to actually call any scripts. It seems to be able to interact with the home just fine, and it recognizes that the scripts exist, but it won’t actually run the script…
I am specifically trying to call the LLM script for Music Assistant that’s configured with a Blueprint. It just wants to keep outputting the JSON it would use to call the script.
Anyone have any ideas of what could be happening here? I’ve confirmed I have the ‘tools’ calling capability enabled, and the script is exposed.
Here is a recent conversation I tried:
Me:
play music by taylor swift in the office
Assist:
To play music by Taylor Swift in the office, I’ll use the llm_script_for_music_assistant_voice_requests function.
{ “name”: “llm_script_for_music_assistant_voice_requests”, “arguments”: { “media_type”: “artist”, “artist”: “Taylor Swift”, “album”: “”, “media_id”: “Taylor Swift”, “media_description”: “music by Taylor Swift”, “area”: [“Office”], “media_player”: , “shuffle”: false } }
Me:
Yes, now actually do the tool call…
Assist:
I’m sorry for any confusion, but as an AI assistant, I don’t have the capability to directly execute tool calls or interact with your home automation system. I can only provide the instructions on how to do it.
To play music by Taylor Swift in the office, you would need to use the llm_script_for_music_assistant_voice_requests function with the provided arguments. Here is the JSON object again:
{
"name": "llm_script_for_music_assistant_voice_requests",
"arguments": {
"media_type": "artist",
"artist": "Taylor Swift",
"album": "",
"media_id": "Taylor Swift",
"media_description": "music by Taylor Swift",
"area": ["Office"],
"media_player": [],
"shuffle": false
}
}
You can copy this JSON and use it in your Home Assistant environment to execute the command.