Philips Hue dimmer switch v2 not responsive

Hi! I’m new to Home assistant and I’ve been testing Hue lights in one of the rooms to be sure everything works fine before I transfer the rest of the Hue stuff from Hue hub to Home assistant. I have started by adding one Hue light bulb and Hue dimmer switch v2 via Zigbee. I have setup a simple ON/OFF automation as you can see here:

The thing is that the switch is very often unresponsive. It takes a few presses to the on/off button on the Hue switch to actually get the light to turn/on. Once the light is on, the switch stays responsive but after some time it will become unresponsive again. It’s like something went to standby mode and then it takes a few nudges to wake up again. Does anyone have any idea what might be the issue?
I run HA on Raspberry PI 4B 8GB and I use SkyConnect.

I now have the same problem as you are describing, did you find a fix for it?