No need for any RPi or amplifier(s).
I will explain my setup a bit. Hopefully it clarifies things a bit.
I run VLC player and Mopidy player om my HAOS. VLC player streams to the audio output of the HA PC and Mopidy to a USB audio device hooked up to the same HA PC. Both outputs are connected to the XAP800.
Via mopidy I stream for example internet radio to only the rooms where movement is. If I have an announcement to make I use VLC player and do that to any room i like. I can however also have music from my NAS in one room and other music in any other or all other rooms.
The XAP800 can lower or raise the volume on each and any channel.
But there is a catch here. These outputs are about as loud as a kitchen radio (but HiFi quality). For me max volume it to loud in a room to have a normal conversation. If you need more volume then just add a 5W amp (only $2-3t a piece on AliExpress) at the output of the XAP800 and connect the speaker wire to that… I do that for my workshop.
For me this audio system is intended to have nice background music in every room, multiple audio sources and announcement options but not at concert sound levels.
You control the unit via a serial connection. (USB to Serial will do.)
Just have a look at the ClearOne XAP800 manual for the specs
They go for around $80,- on ebay.
The newer model ClearOne Converge Pro 840T is a bit more expressive on ebay but that one has included a hefty amplifiers output on one channel and the phone unit is also included.