Ikea Tradfri 5-button remote - ZHA - missing events

I am using an Ikea Tradfri 5 button remote with ZHA and a GoControl HUSBZB zigbee controller.

Overall, my setup is very reliable, however I have one particular issue with my Ikea Tradfri 5-button remote. I currently have the left and right buttons set up to loop through a list of scenes. As long as I push the buttons very slowly, as in one press every 2-3 seconds, it works reliably. However, if I push any faster than that it starts to miss button presses.

Once it begins missing button presses, I generally have to wait 5-10 seconds before it will start detecting button presses reliably again.

I have verified that its not a software/automation issue by monitoring the zha_events as they come in in developer tools. I thought it could also be interference, but I am less that 10 feet line of sight to the zigbee router that its connected to.

Any suggestions on this?