I have several Google Chromecasts and a Google Home Mini connected to Hass.io running on a RPi3. Everything works great!
On the larger Google Home you can “Press and hold down on the top of the device” to “start your request”. The Google Home Mini is only voice activated. I find this problematic when playing music (the Google Home Mini can’t hear my voice over the music).
I would like to make an automation that sends a command to the Google Home Mini to put it in “listening mode”. I could then use a Xiaomi Zigbee button or something similar as a trigger.
Privacy is sort of a big concern for these smart speaker manufacturers, so Google very deliberately doesn’t let you control Google Home from software. Google Home Mini used to have a tap control, but this was disabled because it wasn’t always working properly.
Having said that, my Google Home Mini usually can hear my voice through music if I speak a little louder than usual.
Keep in mind that placement of the device is important. If it’s in direct line of sight of one of your speakers, it might have more trouble hearing you.
I used to have mine (too) close to the TV and it wouldn’t hear me through a talk show or something similar. Placing it just a meter or two further away did miracles for me.
Not the answer you were looking for, probably, but maybe it helps.
How are you playing music? You could look to pause it using the button, rather than trying to control the Google Home. For example, if you’re using Spotify you could try the Home Assistant Spotify component.