I figured out what the problem was. All three of my speakers were running the S1 OS firmware. The Play 3 and the Symfonisk both support the S2 OS so I updated them both and now everything works as advertised. I hope this can help someone else if the future. The documentation should be updated to note this.
BTW is anyone using the Chimes TTS HACs integration for announcements? It works great with the main feature being combining a sound effect with TTS before sending it to the media device. The only thing I am surprised it doesn’t support is sending to multiple speaker entities at once.