Assistant conversation stops / listening stops on Android after just one answer

Hey, I just went the way to use tasker and the hotword-plugin to speak to assist, because I want to implement this via some Sonoff-NSPanel Pro with Android.
The hotword runs not too bad with my phone, but there is a major issue:

Whenever I say something - which is correctly recognized and processed (I’m using GoogleAI for the actual processing), the mic turns completely off.
I have to tap it again to be able to speak, but this way it is not a conversation at all - but just basically a single command that I can say.

Any idea how this happens and what i can do against it?