Controlling Xbox one

Right now, I’m using a legacy integration and addin locally control my Xbox one with a simple IR remote. I know that the official integration exists and tried it, but since it’s going over the internet, which is btw unnecessary, it’s laggy and simply unusable.
But lately my Xbox just doesn’t connect to the old smart glass rest server and needs to be rebooted to connect.
My Xbox is actually just used for Netflix, YouTube and Kodi, rarely gaming, so it’s more or less a streaming box.
Do you have any other idea how to control it or fix my issues? I thought about an esp talking via USB Host Module to the Xbox as “keyboard”, but for that I have to write my own c++ code, which isn’t my strength.