Use Zigbee Dimmer Switch in combination with physcial dimmer

I installed a Girier 2 Gang/CH Zigbee Dimmer Switch Module via Z2M to control the kitchen and living room.
Using HA everthing works fine and turning the lights on & off with the physical switches in the room itself is also no problem.
But it’s not possible to dim the bulbs by using the knob on the physical dimmer, it just doesn’t do anything.
Does someone know the reason or the solution for this problem?