You’re not alone, there’re a few threads about this.
My Denon AVR with Spotify Connect seems to stay in the source list, but I have the same issue with all my Chromecasts (even after specifying aliases for them).
I gave up and ended up using assistant relay to start Spotify on desired Chromecast / Chromecast group.