šŸ’™ Blueprints for Voice Commands (Weather, Calendar, Music Assistant)

Version

Buy me a coffe

Paypal

Home Assistant Voice Blueprints

:construction: Under construction

The blueprints are still under construction and might change quite frequently. Iā€™ll try not to introduce any backwards incompatible changes.

:blue_heart: About these blueprints

Since I received my Voice PE units, Iā€™m a happy user of Assist in my house. The recently added LLM fallback functionality also gave a lot more functionality to voice in Home Assistant.

However, there is still some functionality lacking. Out of the box you canā€™t ask for things like weather, calendar entries because this information can only be retrieved with actions and action responses.

To bridge this gap, I created a repository with voice blueprints

On this respository I will put my blueprints which are created to help issuing advanced (voice) commands using Assist.

:loudspeaker: Voice configuration

There are basically 5 ways to configure a voice pipeline in Home Assistant:

# LLM Home Control (Assist) enabled Local preference enabled Description
1 :x: NA NA Local only
2 :white_check_mark: :x: :x: LLM as voice agent, no home control, no local preference
3 :white_check_mark: :x: :white_check_mark: LLM as voice agent, no home control, with local preference
4 :white_check_mark: :white_check_mark: :x: LLM as voice agent, with home control, no local preference
5 :white_check_mark: :white_check_mark: :white_check_mark: LLM as voice agent, with home control, with local preference

On this repository different blueprints will be placed with the same goal. The will differ in which op the above mentioned configurations are supported. Based on that, also other aspects will differ.

:wrench: Blueprint options

There are 3 options for the blueprints

  • Option 1: Local automation

  • Option 2: LLM Enhanced automation

  • Option 3: Full LLM script

Description Option 1 Option 2 Option 3
Supported voice configurations 1, 2, 4 1, 3, 5 3, 4, 5
Fully local, no LLM required :white_check_mark: :x: :x:
No home control for LLM required :white_check_mark: :white_check_mark: :x:
No strict voice commands :x: :white_check_mark: :white_check_mark:
Complete freedom in voice commands :x: :x: :white_check_mark:
Variance in responses for each request :x: :white_check_mark: :white_check_mark:
Works without need for translations :x: :x: :white_check_mark:

The groundwork for option 2 was done by JLo in his blog post on GPT-powered music search. This blog post was a big inspiration for the LLM enhanced automation, and also enabled the full LLM script. So big thanks to JLo!

The Blueprints

:sun_behind_rain_cloud: Weather Forecasts

Using these blueprints you can ask for the weather forecast from a weather entity in your system.

Option Import Button
1: Local Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.
2: LLM Enhanced To be created
3: Full LLM Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

:date: Calendar

Using these blueprints you can ask for the calendar entries from one or more calendar entities in your system.

Option Import Button
1: Local Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.
2: LLM Enhanced To be created
3: Full LLM Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

:white_check_mark: ToDo

Using these blueprints you can ask for the todo list items from the todo entities in your system.

Option Import Button
1: Local Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.
2: LLM Enhanced To be created
3: Full LLM To be created

:notes: Music Assistant

For completeness I also mention these blueprints here. They are located on the Music Assistant Voice support repository.

Iā€™m also a maintainer of that repository and did a lot of work on those blueprints together with some other guys.

Using these bleuprints you can issue voice commands to play media on your Music Assistant media players.

Option Import Button
1: Local Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.
2: LLM Enhanced Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.
3: Full LLM Open your Home Assistant instance and show the blueprint import dialog with a specific blueprint pre-filled.

:memo: Planned

  1. Add missing options for weather and calendar

  2. Add blueprints to create reminders

  3. Add blueprints to add and list todo items, including due dates/times

  4. ā€¦

:question: Questions/issues/bugs/feature requests?

In case you have a question, you found a bug, or have a feature request open an issue on the GitHub repository.

In case something isnā€™t working or you found a bug, a trace of the script will be needed in most cases to determine the cause. The trace can be downloaded as a json file. To do this follow the steps below:

  1. Depending on the blueprint type go to Open your Home Assistant instance and show your scripts. or Open your Home Assistant instance and show your scripts.

  2. Find the relevant script or automation in the list.

  3. Click on 3 dot menu icon left of the script and then select Traces

  4. Make sure to select the right trace which contains the error

  5. Press the 3 dot menu icon in the top right corner, and select Download trace

  6. Press the download icon in the top right corner.

In case you create the issue on GitHub you can upload the json files, in case you create the issue as a post here, you can copy the json files to a code sharing website like dpaste.org. Please create a different link for each json file.

:coffee: Coffee

If you think I deserve a coffe, please feel free to buy me one (I might spend it on another beverage though).

In case you decide to do so, thanks a lot!

Buy Me A Coffee

Or you can do a small donation using PayPal.

paypal

8 Likes

Updates

20250130

Blueprints

:white_check_mark: ToDo

  • Local Assist blueprint added to list todo items

20250129.1

Blueprints

:date: Calendar

  • Local Assist blueprint added to retrieve calendar entries

Iā€™m looking forward to giving these a try, thanks!

Iā€™ve been adding items to my todo lists by voice for a while without any need for additional intent sentences etc. Never tried anything more than that though. I did just try getting Voice to list my items and that failed miserablyā€¦ Local only just said I had no area called ā€œsupermarket listā€ and using AI backed Voice it ended up adding 3 random items to my supermarket listā€¦ (eggs, milk and bread)

I have something working on my system, but only for a fixed list for now.
Iā€™m not sure if I can make something working locally as a blueprint, as it seemed to conflict with other built in intents. Or I should really make a really strict sentence like what is on the list named {list_name}

For the LLM variant it will be quite easy to add.

There was also a specific request in one of the other former topics to add support for due dates and due times, which isnā€™t supported in the built in intent

Update

Blueprints

Todo

  • Add local Assist blueprint to list todo items

you can try out the newly added blueprint.

LLM enhanced will come when I have time. Then you can also ask stuff like do I have carrots on the shopping list

1 Like

Thank you for such useful blueprints, especially the music one. It works like a magic!

During the setup, I encountered and fixed an issue that I want to share. Steps I took:

  1. Created a new player in Music Assistant.
  2. Connected Spotify to Music Assistant.
  3. Created a script from the Music Full LLM blueprint.
  4. Exposed the new script and new player to Voice Assistant.
  5. Tried using the script but got the error ā€œUnable to find valid target.ā€
  6. The issue disappeared when I assigned the new media player entity to the house area.

I hope this helps anyone with the same problem.

1 Like

What was your (voice) command?

I tried many different commands, from as simple as ā€œplay artist Metallicaā€ to specifying the media player and location. It never worked until I assigned the device to an area.

Im trying to get your blueprint ā€œweather/3_voice_weather_forecast_full_llm.yamlā€ working with ā€œExtended OpenAI Conversationā€ā€¦ i wrote a ā€œspecā€ to tell Extended openAI Conversation how to deal with your blueprint, but iā€™m failing miserablyā€¦ Extended OpenAI Conversation doesnā€™t seem to like my func-definition.

Hereā€™s what i build:

- spec:
    name: get_weather_forecast
    description: Get weather forecast
    parameters:
      type: object
      properties:
        time_period_type:
          type: string
          description: >-
          This argument is mandatory and must always be provided no matter what! Use 'daily' in case the request is for full days, examples: today, next Thursday, coming weekend. Use 'hourly' in case the request is for a part of a day, examples: at 8 hours this evening, next morning, this evening, from 3 to 5
        time_period_length:
          type: number
          description: >-
          This argument is mandatory and must always be provided no matter what! The length of the time period. This will be the number of days in case the 'time_period_type' is 'daily' and the number of hours in case 'time_period_type' is 'hourly'. Examples: 1 (day) if the request is for the forecast of today, 2 (days) if the request is for the weather this weekend, 6 (hours) if the request is for the weather periods like night, morning, afternoon and evening
        start_date:
          type: string
          description: >-
          This argument is mandatory and must always be provided no matter what! Provide the date on which the requested forecast starts. This date should be in the format YYYY-MM-DD. Always provide a date, so in case request is the forecast for today, provide today's date. When the forecast is for the night, do not use today's dat but always use the next day, unless the current time is still before 05:00:00
        start_time:
          type: string
          description: >-This argument is mandatory and must always be provided no matter what! Provide the time on which the requested forecast starts. This time should be in the format HH:MM:SS in 24 hour format (for example 17:00:00). Always provide a time. In case a full day is requested, use 00:00:00, in case the current hour is requested, use the start of the hour. Night starts at 00:00:00, morning starts at 06:00:00, afternoon starts at 12:00:00 and evening starts at 18:00:00
      required:
      - time_period_type
      - time_period_length
      - start_date
      - start_time
  function:
    type: script
    sequence:
    - service: script.llm_weather_forecast
      data:
        time_period_type: {{time_period_type}}
        start_date: {{start_date}}
        start_time: {{start_time}}
        time_period_length: {{time_period_length}}

Maybe someone here can help? Do you recommend sticking with ā€œOpenAI Conversationā€ instead of ā€œExtended OpenAI Conversationā€?

I never used Extended OpenAI conversation. I only tested it with core LLM integrations like Open AI Conversation and Google Generative AI

Thank you for creating these scripts! Iā€™ve been using the weather and music one. Is it possible to control spotify through home assistant with voice instead of music assistant? My wife prefers to use the spotify app. Music assistant ā€˜hijacksā€™ spotify so the music canā€™t be controlled through the spotify app anymore.