When changing a binary_sensor (ping) I get a new entity causing issues in alarms and automations

Hi,
due to a network change I needed to change a binary_sensor (ping) we have been running:

- platform: ping
  host: 192.168.16.4
  name: 'VPN Mörby'
  count: 2
  scan_interval: 180

This has been available in automations as well as in alerts. After the network change I changed the IP-address and was surprised to see that the alert didn’t go away but the sensor instead got unavailable. Checking my entities I see that in fact now have two:


Do I need to update all my alerts and automations or is there a better way to solve this mess?
What should I have done instead of simply changing the IP-address to avoid this in the first place?

Thanks,
/Mattias

delete binary_sensor.vpn_morby and rename binary_sensor.vpn_morby_2 to binary_sensor.vpn_morby

Can I do that from the UI? Checkboxes are disabled and HA complains of missing unique id but binary_sensor doesn’t support unique_id setting so I can’t assign any.