Enhanced Music Assistant voice control with global music search (without immediately playing)

Hi everyone,

I created a script today to ask assist things like:

“Search a few albums from Daft Punk to select from.”
“Show me some playlists with breakfast in the name.”
“Is there a song called She Moves from Graham Candy available?”

And as you’re talking to an LLM, that can process the result before replying, my kids tested:
“Search for Minions albums, but we want funny music and not movie soundtracks or audiobooks.”

Which only replied with a few albums that aren’t the OST or audiobook of one of the movies (they were in the result of the script, but filtered out by the LLM).

Together with the Full LLM Music Assistant Blueprint from @TheFes collection ( GitHub - TheFes/ha-blueprints: Home Assistant Blueprints for (voice) commands ), you can then tell assist to play the result e.g. in the living room.

This is already way cooler than Alexas music control, and this was one of the points where she was better than custom voice assistants for a looong time. :sunglasses:

Another example:
“List me 10 songs from Lilly Wood and the Prick”
And then tell assist to play number 1,3,8 from the list in the kitchen (which again calls the exisiting Music Assistant script linked above, with the correct parameters to start the playback with the first song and enqueue the rest).

It’s more or less a thin wrapper around the music assistant “search” action, that filters out unneeded field:

(This is a simple script, shared with the LLM assistant. Not a intent_script.)

alias: Music Library Search
icon: mdi:search-web
description: >
  Use this tool to search for Radio Stations or available music in the streaming
  services we subscribed. Attention: if you search for songs of an artist, the
  media_return_type is track, as you search for tracks. media_return_type artist
  is only for searching artists, not songs of artists.
fields:
  media_return_type:
    name: Media Return Type
    description: >-
      Choose the right category that you want to be returned (Not the type you
      provide as search_query). If you search for multiple artists, choose
      artist. If you search for tracks from an artist, choose track.
    required: true
    selector:
      select:
        options:
          - track
          - artist
          - album
          - playlist
          - radio
  search_query:
    name: Search String
    description: >-
      Could be the artist name, a specific track name, both combined with a dash
      between, a radio station name, a playlist name
    required: true
    selector:
      text: {}
  limit:
    name: Limit
    description: >-
      Set the maximum results you want to get back. If you only need a single
      match set limit to 5, which is the minimum value when calling this tool.
      If the user requested a longer list, set the limit to something larger
      than 5 as needed.
    required: true
    selector:
      number:
        min: 0
        max: 50
        mode: slider
sequence:
  - data:
      name: "MUSIC LIBRARY SEARCH: "
      message: "{{ media_return_type, search_query,limit }}"
      entity_id: "{{ this.entity_id }}"
    action: logbook.log
  - data:
      config_entry_id: 01JH0XS4ZEMCFN1DTCH87B3G50
      media_type:
        - "{{ media_return_type }}"
      name: "{{ search_query }}"
      limit: "{{ limit }}"
      artist: ""
      album: ""
      library_only: false
    response_variable: raw_result
    action: music_assistant.search
  - variables:
      simplified_result: >
        {%- set ns = namespace(
              tracks   = [],
              albums   = [],
              artists  = [],
              radio    = [],
              playlists= []
           ) %}

        {# -------- TRACKS -------- #} {%- for t in raw_result.tracks |
        default([]) %}
          {%- set track_artist = (t.artists | default([]))[0].name
                                 if (t.artists | default([]))|length > 0
                                 else none %}
          {%- set album_name = t.album.name
                               if t.album is defined and t.album.name is defined
                               else none %}
          {%- set album_artist = (t.album.artists | default([]))[0].name
                                 if t.album is defined
                                    and (t.album.artists | default([]))|length > 0
                                 else none %}
          {%- set ns.tracks = ns.tracks + [{
                'name':   t.name,
                'artist': track_artist,
                'album': {
                    'name':   album_name,
                    'artist': album_artist
                }
          }] %}
        {%- endfor %}

        {# -------- ALBUMS -------- #} {%- for a in raw_result.albums |
        default([]) %}
          {%- set album_artist = (a.artists | default([]))[0].name
                                 if (a.artists | default([]))|length > 0
                                 else none %}
          {%- set ns.albums = ns.albums + [{
                'name':   a.name,
                'artist': album_artist
          }] %}
        {%- endfor %}

        {# -------- ARTISTS -------- #} {%- for ar in raw_result.artists |
        default([]) %}
          {%- set ns.artists = ns.artists + [{'name': ar.name}] %}
        {%- endfor %}

        {# -------- RADIO -------- #} {%- for r in raw_result.radio |
        default([]) %}
          {%- set ns.radio = ns.radio + [{'name': r.name}] %}
        {%- endfor %}

        {# -------- PLAYLISTS -------- #} {%- for p in raw_result.playlists |
        default([]) %}
          {%- set ns.playlists = ns.playlists + [{'name': p.name}] %}
        {%- endfor %}

        {{ {
          'tracks':    ns.tracks,
          'albums':    ns.albums,
          'artists':   ns.artists,
          'radio':     ns.radio,
          'playlists': ns.playlists
        } }}
  - stop: ""
    response_variable: simplified_result

:red_circle: You might have to set the config_entry_id to the one of your Music Assistant instance. :red_circle:

In the script is the current stable Music Assistant version identifier.
If you use the beta, try this one: 01JH0XS4ZEMCFN1DTCH87B3G50
In case it changed over the time look it up here:
Developer Tools → Actions → Music Assistant Search → Select the Music Assistant instance in the dropdown → Switch to YAML view and copy the needed id.

Some notes on the description and my prompt:
I changed the description of the existing Music Assistant Play Script / Blueprint linked from TheFes repo above, that it is to PLAY music (I removed the part where it stated “search and play”)
And I added that there is another script (this new one here) to search music.

Then in my prompt I have this section:

There are 2 tools to start media playback.
Always prefer “<NameOfTheOtherScriptOfTheFesRepo>” over HassMediaSearchAndPlay, as the latter one isn’t working currently.
If you only search for music, but won’t play it, use “<NameOfThisScript>”

This also solves the problem, that HA has it’s own “Play” tool now, which seems to be preferred by Assist.
That’s why I told it, that it wouldn’t work atm, so it doesn’t touch it anymore. :wink:

One last note:
I do all my automations in Node-RED.
So also my LLM Scripts use normally Nored.Trigger to call a flow and an event to hand the response back to the script.

But as this worked so wonderful, I thought it should be shared.
As I never touched Jinja beside some VERY simple templates, I asked ChatGPT O3 to convert the logic.
It works, but I don’t know if it’s best practise.
Fell free to post improvements if needed. :stuck_out_tongue:

4 Likes

Well done @Thyraz

Note. I had two copies of MA installed (production and beta) I had to go into your script and select the other instance (it autopicked the last one I installed - the beta) then it fired right up.

1 Like

Ah, ok. I thought this ID would be different for every installation.

Looks like it’s an identifier for the specific add-on. I use the Music Assistant beta as my default installation.

Can you post the ID for the stable version here?
I would replace it in the script and add the beta identifier in the text below for beta users so they don’t have to look it up on their own.

edit: replaced the beta ID with the default one in the start-post.
Thanks Nathan. :slight_smile:

I found this post searching for errors with HassMediaSearchAndPlay after installing the blueprint you mention. When you say: “Then in my prompt I have this section:”, are you referring to the main prompt for the Voice Assistant? I’m using the “OpenAI Conversation Agent” with the default prompt:

You are a voice assistant for Home Assistant.
Answer questions about the world truthfully.
Answer in plain text. Keep it simple and to the point.

Am I meant to chuck this bit:

There are 2 tools to start media playback.
Always prefer “<NameOfTheOtherScriptOfTheFesRepo>” over HassMediaSearchAndPlay, as the latter one isn’t working currently.
If you only search for music, but won’t play it, use “<NameOfThisScript>”

into that prompt and how do I find the right values for NameOfTheOtherScriptOfTheFesRepo and NameOfThisScript?

Thanks!

1 Like

Basically yes. Go read Friday’s Party to understand why the default prompt will not get you far.

1 Like

When you open the scripts section in HA you see the list of scripts.
Here you see the names you gave the scripts.
You can also click on then them and then choose rename in the “3 dots” menu on the top right to give it a different name.

These names are also provided to the LLM, so it knows what you’re talking about when you use them.

1 Like

Thank you both!

I fleshed out my prompt a bit, but I actually figured out that my issue was rather dumb: I had not Exposed the script to the LLM in the Voice Assistant settings. I’m new to all this and some of the steps that are listed are not obvious how to do at all.

For a beautiful moment, I had it working! I could say “Play Metallica in the Living Room”, and it did. In my debugging, I realized I hadn’t actually added my speakers to areas, and the script was invoked with “area: living_room”, where there was nothing.

Then I tried to pause the music, and that did not work. So I had this issue where after I set up Music Assistant and added the SONOS provider and the Home Assistant MediaPlayers provider (should you add both?), I saw the speakers multiple times and I had errors on the form of “you have duplicate speakers” from the LLM, so I removed things until I only had one speaker show, and that’s when it worked.

Now, in order to give the LLM access to pause / unpause music, I foolishly tried to re-add things (don’t remember the order, but I think I added back the HA provider) and now nothing works at all.

I’m pretty sure that the issue is that MA doesn’t know about the HA areas. When I click “run” on the LLM script and scroll down to the “Area” field, the dropdown shows zero items:

Sorry for the essay (and the off-topic question, might start a new thread), just so excited to get this working and I feel like I’m so close!

Any hints?

Update 1: disregard, a good ol’ reboot (of both MA and HA) seems to have mostly fixed it (one area isn’t showing up still).

Update 2: it’s working on 3 of 5 of my SONOS speakers. One area / room is not showing up at all and one speaker isn’t working even though it is showing up (I renamed this area from Bedroom to Master Bedroom and then back again, so maybe it got into a weird state). I’ll just try to kind of turn things off and on again until it starts working I guess.

Since I already took this thread off-topic, can I ask two more follow-up questions:

  1. The only way I can seem to get it to work is if I have both the SONOS provider and the HA provider in MA, but that leaves a bit of a mess with doubled up speakers. Is this just how it is and the solution is to hide them in my dashboard?

  2. The speech-to-text I have is quite awful. I am using faster-whisper. Before I discovered Home Assistant, I set up a simple thing in Python using just OpenAI’s APIs which used the transcribe endpoint for the stt and it worked so well. Have y’all figured out how to use OpenAI’s tts in HA?

I disabled the Sonos integration since I use Music Assistant.
Not sure if there are any features that only work with the Sonos integration, but all features I need are also working with the Music Assistant players.

The Music Assistant players are added to the areas in HA, and this works for me.

About SST: I use Nabu Casa Cloud, so haven’t tested anything else so far.
There seems to be an HACS integration that allows to use the OpenAI SST models in Assist:

And as we’re not transcribing hours of text here, it should be pretty cheap.
Might also try that at some time to see how it compares to Nabu Casa Cloud.

And also the OpenAI TTS integration, as these voices and the customizability is really cool.

Thank you Thyraz!

I ended up also disabling the SONOS integration and that seems to be fine now. I think the issue with the areas not showing was because I hadn’t added the MA integration to HA.

I also ended up signing up for Home Assistant Cloud and the STT has been working very well so far. It might be cheaper to use OpenAI’s, but for $5 a month, it’s not worth the hassle.

My current endeavor is to change the wake word for my Home Assistant Voice Preview Edition units, as I think “OK Nabu” is kind of weird and I find the three options all bad and limited. I tried out Porcupine before I found HA and it worked really well, but in the config panel for the device, it doesn’t seem to be an option at all to use another wake word provider:

Any idea how I can override this?

You have to compile and provide your own
Then you customize the VPE build:

Yep, currently I use the Alexa one also explained in the thread Nathan linked in a post further down the thread:

Not really happy to use the Amazon wake word, but it was easy to add and works best for me so far.

1 Like