How to analyze crashes

Installation Home Assistant OS
1

Hello,

I’m running HA OS on a RPI 4B. It’s installed in a SSD. A lot of times when upgrade to a major version (like from 2023.8.x to 2023.9.0) HA crashes. The UI, Observer and addons (e. g. Adguard) become unavailable. I usually have to unplug the RPI from power for it to come back. The update is then successful. Yesterday, I switched on a light via a widget on my tablet. Somewhere in between there and turning the light back off (it didn’t) HA crashed again. It hasn’t come up since last night. Neither the UI nor Observer are working.

My question is what I can to do analyze the crash afterwards to know what caused it. Also, is there a way to access the host OS to see if that may still be running? I cannot connect a screen to it. I guess SSH won’t work of the SSH addon is unavailable.

Greetings

2

You can enable a debug SSH shell on the underlying OS on port 22222, see SSH access to the host.

In the OS shell you can get the system logs of the previous boot using journalctl -b -1.

I also recommend to enable the System Monitor integration to monitor memory and CPU usage.

3

Took me a while to get this running, but now I have access to the OS’s SSH. I ran the command and it ends at october 14th, 15:57:15, right around when I updated Home Assistant OS from 10.5 to 11, which apparently didn’t work. HA didn’t come back online, so I pulled the plug now and restarted it. HA OS is still on version 10.5. I don’t really wanna post the file as it has over 500.000 lines.

These are the lines from the last second:

Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.012 DEBUG (Thread-8) [pychromecast.socket_client] [KD-65XF9005(192.168.178.30):8009] connection listener: a57d1f30 (CastStatusListener) ConnectionStatus(status='FAILED', address=NetworkAddress(address='192.168.178.30', port=8009))
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.013 DEBUG (Thread-8) [homeassistant.components.cast.media_player] [media_player.kd_65xf9005 KD-65XF9005] Received cast device connection status: FAILED
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.013 DEBUG (Thread-8) [pychromecast.socket_client] [KD-65XF9005(192.168.178.30):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.178.30', 8009)), retrying in 300.0s
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.013 DEBUG (Thread-8) [pychromecast.socket_client] [KD-65XF9005(192.168.178.30):8009] Not connected, sleeping for 5.0s. Services: {ServiceInfo(type='mdns', data='BRAVIA-4K-GB-ATV3-194bb77d1a3523f67123da3094aec7d1._googlecast._tcp.local.'), Se
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.600 DEBUG (MainThread) [async_upnp_client.traffic.ssdp] Received packet from ('192.168.178.57', 50275): b'M-SEARCH * HTTP/1.1\r\nHOST: 239.255.255.250:1900\r\nMAN: "ssdp:discover"\r\nST: upnp:rootdevice\r\nMX: 5\r\n\r\n'
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.602 DEBUG (MainThread) [async_upnp_client.traffic.ssdp] Received packet from ('192.168.178.57', 50275): b'M-SEARCH * HTTP/1.1\r\nHOST: 239.255.255.250:1900\r\nMAN: "ssdp:discover"\r\nST: upnp:rootdevice\r\nMX: 5\r\n\r\n'
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.603 DEBUG (MainThread) [async_upnp_client.server] Received M-SEARCH from: ('192.168.178.57', 50275), headers: {'HOST': '239.255.255.250:1900', 'MAN': '"ssdp:discover"', 'ST': 'upnp:rootdevice', 'MX': '5', '_host': '192.168.178.57', '_timesta
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.603 DEBUG (MainThread) [async_upnp_client.server] Deferring response for 5 seconds
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.603 DEBUG (MainThread) [async_upnp_client.server] Sending SSDP packet, transport: None, socket: <socket.socket fd=72, family=2, type=2, proto=0, laddr=('0.0.0.0', 56719)>, target: ('192.168.178.57', 50275)
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.604 DEBUG (MainThread) [async_upnp_client.traffic.ssdp] Sending SSDP packets, target: ('192.168.178.57', 50275), data: [b'HTTP/1.1 200 OK\r\nCACHE-CONTROL:max-age=1800\r\nDATE:Sat, 14 Oct 2023 13:57:19 GMT\r\nSERVER:async-upnp-client/0.36.1
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.769 DEBUG (MainThread) [async_upnp_client.server] Sending SSDP packet, transport: None, socket: <socket.socket fd=72, family=2, type=2, proto=0, laddr=('0.0.0.0', 56719)>, target: ('192.168.178.57', 50275)
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.769 DEBUG (MainThread) [async_upnp_client.traffic.ssdp] Sending SSDP packets, target: ('192.168.178.57', 50275), data: [b'HTTP/1.1 200 OK\r\nCACHE-CONTROL:max-age=1800\r\nDATE:Sat, 14 Oct 2023 13:57:19 GMT\r\nSERVER:async-upnp-client/0.36.1
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.884 DEBUG (MainThread) [async_upnp_client.server] Announcing
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.885 DEBUG (MainThread) [async_upnp_client.server] Sending advertisement, NTS: ssdp:alive, NT: UUID:66995FCA-65B5-46C6-8815-CA510E6643A4, USN: UUID:66995FCA-65B5-46C6-8815-CA510E6643A4
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.885 DEBUG (MainThread) [async_upnp_client.ssdp] Sending SSDP packet, transport: <_SelectorDatagramTransport fd=77 read=polling write=<idle, bufsize=0>>, socket: <asyncio.TransportSocket fd=77, family=2, type=2, proto=0, laddr=('0.0.0.0', 344
Oct 14 13:57:19 homeassistant homeassistant[505]: 2023-10-14 15:57:19.885 DEBUG (MainThread) [async_upnp_client.traffic.ssdp] Sending SSDP packet, target: ('239.255.255.250', 1900), data: b'NOTIFY * HTTP/1.1\r\nNTS:ssdp:alive\r\nHOST:239.255.255.250:1900\r\nCACHE-CONTROL:max-age=1800\r\nSERVER:async-upnp-clien

Doesn’t seem to have anything strange in it. I can’t see any errors in the log of the past few minutes either. Any suggestions?