Relays are binary only in modelling software - they are analogue mechanical systems, with all the contact bounce, delay, and make-before-break / break-before-make imperfections that can bite.
Amplifier outputs are usually push-pull drivers (i.e. not 0V and +/-40V), originally driven by matching transformers, these days by arrays of MOSFETs. This means you need to switch two wires, not one.
Typical “cheap” relay boards use $1 copies of Omron relays with one SPCO contact. To switch one speaker to two amplifiers therefore requires two SPCO relays as both terminals could be (say) +40V or -40V.
You might be lucky and your amp might use something like a dedicated TDA chip array with short and open circuit protection, but one day the relays might switch at the peak of a drum solo causing an impedance mis-match, and a surge, and an escape of magic smoke.
Making hardware development accessible is great, however the downside is less experience in designing resilient systems - e.g. you could use two relays per channel, driven by two GPIOs. Reboot the ESP with a bad choice of pins, and insufficient decoupling, and find one pin high; one pin low, leading to potentially two dead amps.
A better way is still two relays, but with one GPIO guaranteeing in hardware that even if the microcontroller gets in an odd state, the outputs will always be in a safe state. Of course, one DPCO relay would also work but that still leaves what happens to the amp when the load does from 4R to open circuit.
I suspect manual switcher designs assume the user will turn the volume down before switching - something harder to arrange with automation.
An even better way would be to mute both amps, switch at low volume, then unmute, but that’s a lot more hardware and wiring inputs and outputs to the same relay board is asking for feedback paths. Two 4x relay boards, with one set connecting (say) 1k mute resistors to the amp inputs?
Of course, an AV receiver typically does this internally with input and speaker switching - perhaps an old one with an RS232 or IR interface might be easier? (kind of shifts the problem to one of getting the latest 5k HDMI signal and 88kbps DAC into an old receiver amp though!)