This happened again, and again when I was away from home.
Did some more research (i.e. HA on Rasperry Pi 4 keeps hanging up) and found out it might be connected to me running HA on a Raspberry Pi with a SSD connected via USB.
Turns out this setup does not always deliver enough power to the SSD and then HA (or whatever else would be running on the Pi) is failing in weird ways - even when using the official RPi 3A power supply.
Still kinda funny that this always happened when I wasn’t home 
So I reactivated an old Intel Nuc, turned its TDP down and moved my HA setup over.
Now
- power consumption is roughly the same
- speed is much faster than on the RPi
- HA is rock solid - let’s see when I go away for a while again…
- HA is running in a VM and I have other stuff running in separate VMs now - finally concerns are separated and HA is solely used for home automation
- I have a RPi available for tinkering