Ok, I figured it out, I needed to use the Private BLE Device integration.
If I understand, this is because the Mac address is rotating and you need an IRK.
The instructions I followed to get one was : GUIDE - Getting the IRK for iPhone and Apple Watch without MacOS (Maybe also other devices?)
you can ignore the parts that are Apple specific. I skipped all those without problem. The super short version is to flash an esp32 with some custom firmware that will let you phone pair via Bluetooth, and when it pairs, it will spit out the IRK into the serial output. You can then use the IRK from there to add your device via Private BLE device integration.