ADB debug mode RSA changes on every reboot?

I have ADB enabled on my Android media player so that I can interact with it from Home Assistant. My issue is that sometimes when my Home Assistent server reboots, I get a popup in my android device that wants we to accept that the RSA key (of the HA server I guess) has been updated and I need to re approve it.

Is there a way to make this key not change randomly every now and then so I dont get the popup?