Zigbee devices re-autojoin after removal

Configuration Zigbee
1

This happens almost immediately via ZHA if they are powered on at the time of removal. (If removed via ZHA)

2

Check if ‘permit join’ (or the ZHA equivalent) is turned on. Turn this off before removing.

3

I did not find any similar option in ZHA.