Trigger Google Home Routine/Automation from Home Assistant reliably?

I have created a helper input_boolean which is exposed to Google Home. When this button is ON the routine/automation in Google Home will run. But this is not reliable at all. First off, when it does work, it is very slow. The button triggered from Home Assistant will be detected instantly in Google Home, but the routine is very slow to run and will not run at all times.
If I activate the routine (without button) directly in Google Home it works flawlessly.

Is there any other option to run a routine/automation which in Google Home from Home Assistant?

I use Home Assistant for all my other automations, but this specific automation (play music from Youtube Music on specific speaker groups) does not work from Home Assistant, without using things like ytube_music_player. And starting music from this integration will result in me not being able to control the music from my Youtube Music app on my phone without reconnecting to the speaker groups through the app.
Starting playing from Google Home will create a seamless handover.

Thank you!