Since some time Large Language Models like OpenAI/ChatGPT or Gemini can be used for voice commands, they can even control your house now. 2024.12 will even bring an option to prefer local processing of the command first, so the LLM will act as a fallback. This opens a whole new world of possibilities.
If you expose a script to Assist, a LLM can access and use that script. Using fields you can guide the LLM how to use the script. Using this I created a script which gathers calendar event data from one or more calendar entities (using the calendar.get_events action) and then feeds that data to the LLM, so it will give a nice summary. It will only send the events (ordered by time) to the LLM, so not the calendar it originates from.
How to use it
Use the button to import the blueprint
Create a script using the Blueprint, select the calendar entity or entities you want to use.
Save the script. IMPORTANT: The LLM will use the description of the script to determine how to use it. So it is important to provide a good description. I use Fetch calendar events from my calendar. In case the data for the weekend is requested, this means Saturday and Sunday
IMPORTANT: Expose the script to Assist
That’s it, you can now ask an LLM voice agent for calendar entries and it it will tell you what’s planned (provided the data is available in your calendar entity).
Thanks to:
@jenova70 for providing the groundwork for the script
I removed the inititally added LLM instructions. If you get responses including * which are spoken out by the LLM, change your LLM configuration to exclude them.
How do I change my LLM configuration to exclude * from being read? IT was not reading them at first but now it is saying asterisk asterisk in it’s response. Thank you for this
What’s an example command for getting this to trigger? Google AI and ChatGPT both don’t seem to trigger it, so I’m not sure if it’s not installed right or I just don’t get how to trigger it to activate. Obvious things like “What’s on my calendar today?” don’t do anything.
Ok, well this is weird. This script generated from the blueprint worked fine, but, without me changing anything, now I’ve gone back to getting “I can’t access your calender” response variations, and I can’t quite figure out why
EDIT: nevermind, appears to have been a glitch with Gemini, it sorted itself out.
@wpigulce I’ve seen that with the weather forecast to. It’s not the script, but the LLM. If you ask it for Wednesdays date, it will return the date for Friday. You can try this
@TheFes - any ideas what could be causing the issues for me? Used your blueprint, worked fine for a bit then decides it no longer has access to my calendars.
It actually feels like the the voice assistant is defaulting to local and not actually using Gemini.
But then I ask for the weather forecast (using your weather blueprint), and it works fine, so I’m confused. I’ve deleted the script, and configured it again, tried a million different ways of asking for the information, but it just won’t give me anything any more.
It was working when I went to bed last night and not working this morning, despite exactly nothing changing in between
Using Google Gemini. I’ve tried telling it to use the script, the helper script, the full name of the script, same response. When it worked yesterday, I just asked Jarvis to, for example, “tell me what’s on my calendar for Tuesday 28th”, and it checked my calendar and told me.
I had thought it was maybe an issue with Google calendar authentication, but I’ve just added a test entry via Google, reloaded the Google Calendar integration in HA, and the new event shows up just fine, in both the Google calendar integration, and the Home Assistant Calendar.
“Use the LLM Calendar script to fetch my calender events for XXXX” got it working again. Eventually.
After that I can ask normally, for example, “What’s on the calendar for 7 February”, or “are there any hospitals appointments in my calendar for the next two weeks”.
However, if it then gets a day wrong, and I ask it to check and confirm more than once, it then takes a sulk and tells me because it’s not been accurate it can no longer recover data from my calendars. I then have to rerun the original "Use the LLM Calendar script…"command to get it running again!
I’m fairly certain this is a Gemini issue though. AI is still pretty dumb atm.
