TL;DR: It’s a well known problem, more mains-powered devices help, update the firmware.
The 5-button remote (and other IKEA devices) has just received firmware update 0x24040005 / v24.4.5 which reduces polling and increases battery life. ZHA can automatically install such updates (no idea about Z2M), but note if you can update, a pre-release Zigbee quirks file (likely until HASS 2023-03) is needed.
