Upgrade 2023.11 caused outage

Hello,

After last nights update to 2023.11 the system became unresponsive and both IP and WebUI are not accessible. When checking the network interface on the switch side we see that the link is up, protocol is up but only traffic from the network to the Raspberry is visible and no communication back. Not a single packet is being send by the Raspberry.

I do have backups, but not outside of the SD-Card. So re-install and restore is currently not an option.

When the HA> command prompt is visible the wireless and ethernet adapter are showing no IP details.

When working on assigning an IP through the command line we ran the following command:

net update end0 --ipv4-address 192.168.0.122/23 --ipv4-gateway 192.168.1.254 --ipv4-method static --ipv4-nameserver 8.8.8.8
net reload

This resulted in a message that the file not available for writing and an out of diskspace issue was ongoing.

After imaging the original SD card (64GB) and transferring the image to a 256GB SD-Card should allow HA to expand the volumes automatically resolving the out of disk space issue. The new SD-Card booted fine, whereas the issue persisted.

Next we deleted a few backups using the command line ‘backups delete xxxxx’ to ‘free up some space’ which also didn’t help.

Next we checked any pending update tasks / jobs, new updates and refreshed the update repo just in case. Nothing.

Who can help me understand:

  1. Why / how to set the IP address so i can get back into the UI?
  2. How can i extract the backups that do exist on disk?
  3. What could be possibly be wrong?

We have eyes on the system (it was running headless for more then 2 years without issues) and acces to the HA> command interface.

Any suggestions?

(the undervoltage is new, initially it was running on an USB-A to USB-c adapter, which we now moved to an USB-C - USB-C adapter with more power. Message sometimes still shows)

Why did the system crash?

  • Bluetooth devices are passing by our home on a daily basis, but in sheer numbers! There is a fery nearby and hence thousands pass by all carying smart- and mobile devices. Each of them get picked up by HASS which end up in a bluetooth cache. This cache seems to have no restrictions and can cause an overflow condition where there is no more diskspace.

How to solve this is written in: "no host internet connection" - "can't edit network settings" - "No space left" · Issue #2752 · home-assistant/operating-system · GitHub

in short:

run the following command to clear out the cache and regain control over the system:

find /var/lib/bluetooth/*/cache -mindepth 1 -type f -atime +7  -exec rm {} \;

post which changes to the network can be made.