Not possible with Google Home and Home Assistant. Google Home does not offer a way for third parties to create groups and link/unlink speakers on the fly. Only possible withing Google Home. So only solution is what you’re already doing: create all combinations of groups within Google Home and them cast from Home Assistant to a pre-defined group (which acts like a single media player in Home Assistant).
This is one of the reason I ditched music from Chromecast all together. Also I got crazy with Spotcast, since Spotify keeps changing its API and this plugin breaks randomly and needs to be fixed by the dev. And the 60min token expiration just ruins the whole point of music automation (which the creator of this plugin can’t do anything about because of how shitty Spotify operates). Sometimes Chromecat group speakers became unavailable as well in my setup, only a HA restart fixed it.
I have 6 Google speakers I strictly use for voice and notifications now. I use Logitech Media Server for all my music now, which is 100x better and is like Sonos (but better imho). Full Spotify, Tidal, Deezer, Radio, YouTube support and on the fly syncing/unsyncing of multiroom speakers. Also has native integration in Home Assistant. This is their forum if your interested: https://forums.slimdevices.com There are multiple threads about Logitech Media Server in HA forums as well if you’re interested.