Override default speaker for Nest mini media player

Hi,
I use an external bluetooth speaker as default speaker for my nest mini for music.
Unfortunately there is a 2 second gap in audio in the beginning when I start to stream music.
This is a problem for TTS notifications. It skips first words of a notification.

I searched but couldnt find a way to force TTS media clips to be played using nest mini internal speaker, which works without any gap.

Is there any way to override default speaker selection?