Google Assistant - state reporting slow down

Hello,

I am running home assistant on Rpi3. I am using zigbee2mqtt and node red for most of my applications.
For example, pressing a zigbee switch, node red processing and the final action takes place under half a second usually.

However once I have added google_assistant integration all response slowed down considerably.
When I press the button, lights turn on/off in about 4-5 seconds.

I have found out that setting

report_state: false

resolves that problem.

Is there some synchronous communication happening to google assistant that is slowing things down.
Anyone else experienced something similar?
The lights I am switching are exposed to google assistant.

Any suggestions on how to keep the state reporting while having somewhat reasonable responses?