ESP32 not waking up from deep sleep when on battery

I’ve got an M5StickC Plus2. I’m trying to get it to wake up from deep sleep. It does so as expected when powered by USB, but it does not wake up when powered from the internal or an external battery.

I’ve found this topic about deep sleep on the M5StickC Plus2 that mentions doing something to GPIO4, and on the pinout it’s got a special HOLDGPIO4 line that doesn’t seem to go anywhere - but I don’t know how that translates to settings on the ESPHome deep_sleep component.

My spec is nothing special:

esphome:
  name: m5stick

esp32:
  board: m5stick-c
...
deep_sleep:
  id: deep_sleep_1
  sleep_duration: 10s
  run_duration: 40s

Can anyone give any pointers to what may be going on, and how I can get the wake up to work on battery power?