Hardware Switch on GPIO no longer working after deep sleep

Hi, I hope you can help me with a kind of strange behavior:
I have a XIAO ESP32C with a hardware switch attached. I want it to work as a “deep sleep on off switch”. This is going to GPIO02.

The state of the switch is displayed and sent back to HA through a binary_sensor.
The switch is also used to wake up the ESP. See config below.

The switch is working normal and displays correct results in the log when turning on/off.

But the deep sleep is initiated turning the switch on and off does not wake the ESP.

And even more strange: After the ESP wakes up on its own through the sleep timer, the Hardware switch state is not working anymore. Whether I turn it on or off is no longer displayed in the log. As if events on the gpio are no longer recognized.

When power cycle the device (by removing the battery for some seconds) the switch works normal again.

Is my config wrong somehow or is this expected behavior?

Best regards,
Janis

binary_sensor:
  - platform: gpio
    id: auto_sleep_activated_hw_switch
    name: "Auto Sleep On/Off HW Switch"
    pin:
      allow_other_uses: true
      number: GPIO02
      inverted: true
      mode:
        input: true
        pullup: true

deep_sleep:
  id: deep_sleep_1
  wakeup_pin:
    number: GPIO2
    inverted: false
    allow_other_uses: true
    mode: 
      input: true
      pullup: true
  wakeup_pin_mode: KEEP_AWAKE