I think it is kernel issues personally. However, if it was standalone, if you get it working you can leave the OS alone - not really an option with HA OS.
Yes, probably it is a kernel issue. But what I still don’t understand is why Bluetooth works fine on a Raspberry with HA OS for most people, while some, like you, have issues. It is the same hardware, same OS. If it wouldn’t work at all on HA OS, github would be filled with issues, but that’s not the case. On the other hand, it’s not only you, from time to time there is someone who is having these unexplainable issues. Very frustrating, I agree.