Google Generative AI - Unable to Control Speakers

Configuration Voice Assistant
1

I have just started playing around with the voice control functionality of HA, with the intent of moving away from Alexa for device control in my home. Right now I’m just talking to it via Assist in the browser, but have a HA Voice PE on the way.

I got some basic functionality working, “turn on living room lights 2 and 3 and turn on the bedroom fan” does exactly what I expect.

However, if I ask it to control my speakers (Sonos) it gets awkward. I am trying “Play Spotify on the Living Room Speaker” or “Play BBC Radio One on the Living Room Speaker” (2 things that work with Alexa just fine) and I get the following. This seems to be a response from Gemini.

image
image404×507 14.7 KB

I have also tried to tell the Assistant to Prefer Handling Commands Locally, which doesn’t help.

Has anyone had much luck recreating this behavior that Alexa can accomplish?

2

If you have not provided any context beyond the defaults and have not provided it a tool to ‘play Spotify’ then it has literally no clue what you asked it to do.

Yes I can get mine to literally do my pantry inventory for me if I want to and in the next week or two it’s going to be reading the chem levels on my hot tub BUUUUUUT it takes lots of care and customization. It is not magic. It’s context + tools. If you’re interested in the (reeeeeeeally) long version go find and read my Friday’s Party post.

TL;dr:
You gotta have a tool for that and tell the AI how to use it.