Now that I actually think about it this kinda makes sense. HA is essentially a collection of docker containers. Each add-on runs in it own docker behind the scenes. Supervisor, core and observer and other ‘main’ parts are also running in their own docker containers.
When you ‘restart home assistant’ I think it only restarts the core docker container. Not the esphome container or anything else.
I would imagine if you ‘Reboot System’ it would also clear out phantom devices as that is essentially a power cycle event.