I have been experiencing the same issues lately, but like you I’m unable to go home and check the physical hardware, the only thing I can do is asking my relatives to manually reset the RPi. After that everything comes back online like nothing happened.
It crashed Friday, Saturday, Monday and just less than two hours ago apparently, since the UI is unreachable and Wireguard stopped working. The timing is completely random and by looking just at the hardware info there is no sign of any bad process (CPU usage, temp, RAM and disk are all OK right before it stops recording the data).
This weekend I’ll be going home to see what is happening, does anyone know any way I can debug these errors? (I have a mini HDMI cable I can use to connect a monitor to the headless RPi, but I have no Idea of what to do from that point on).
This is my setup in case someone else is experiencing similar issues:
Hardware:
RPi 4 - 4GB
External SSD Conbee II
USB connection to the UPS
EDIT: I don’t know if the hardware behaves well in cold tempertaures, my RPi is running in a room which has no heating, so the temperature can get to around 8-11°C (CPU temp is around 30-35°C)
Here is how you get to the logs post-crash to see what’s up
Cold is way better than heat unless there’s condensation involved.
Pis usually fail because of storage issues (worn out SD card, etc.) or voltage (bad /failing power supply) considering you’re already on an SSD. I’d look power first. - bring a spare known good power adapter when you visit your box…)
Sooo, I tried SSH’ing in the RPi this morning after someone restarted it for me and I got a look at the system journal like you suggested.
After checking the latest crashes (apparently there were multiple), this is what I found:
Nov 29 12:42:27 homeassistant addon_a0d7b954_wireguard[584]: allowed ips: 172.27.69.99/32
Nov 29 12:42:27 homeassistant addon_a0d7b954_wireguard[584]: persistent keepalive: every 25 seconds
Nov 29 12:42:38 homeassistant addon_core_configurator[584]: INFO:2023-11-29 13:42:38,793:hass_configurator.configurator:127.0.0.1 - "GET / HTTP/1.1" 200 -
Nov 29 12:42:48 homeassistant systemd[1]: run-docker-runtime\x2drunc-moby-d907ccf591e0173a61dcbaa7d7d1c3ae71e3f46a43a6e597f7209537bc87b9e7-runc.8XrdA7.mount: Deactivated successfully.
Nov 29 12:42:48 homeassistant hassio_dns[584]: [INFO] 172.30.32.1:43168 - 39883 "A IN wlan0.local.hass.io. udp 37 false 512" NXDOMAIN qr,aa,rd 37 0.000464843s
-- Boot 679b3ee1f1794b4db750dcbf04daa5dd --
Apr 04 10:55:32 homeassistant systemd-timesyncd[549]: System clock time unset or jumped backwards, restoring from recorded timestamp: Wed 2023-11-29 12:42:59 UTC
Nov 29 17:50:51 homeassistant systemd-journald[134]: Oldest entry in /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal is older than the configured file retention duration (1month), suggesting rotation.
Nov 29 17:50:51 homeassistant systemd-journald[134]: /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal: Journal header limits reached or header out-of-date, rotating.
Nov 29 12:42:59 homeassistant systemd-resolved[445]: Clock change detected. Flushing caches.
Nov 29 17:50:51 homeassistant systemd-time-wait-sync[548]: adjtime state 5 status 40 time Wed 2023-11-29 12:42:59.732333 UTC
Nov 29 17:50:51 homeassistant systemd-time-wait-sync[548]: adjtime state 0 status 2000 time Wed 2023-11-29 17:50:51.536679 UTC
Nov 29 12:42:59 homeassistant systemd[1]: Started Network Time Synchronization.
Nov 29 12:42:59 homeassistant systemd[1]: Reached target System Time Set.
Nov 29 17:50:51 homeassistant systemd-resolved[445]: Clock change detected. Flushing caches.
Nov 29 17:50:51 homeassistant systemd-timesyncd[549]: Contacted time server 162.159.200.123:123 (time.cloudflare.com).
Nov 29 17:50:51 homeassistant systemd-timesyncd[549]: Initial clock synchronization to Wed 2023-11-29 17:50:51.536533 UTC.
Nov 29 17:50:51 homeassistant systemd[1]: Finished Wait Until Kernel Time Synchronized.
Nov 29 17:50:51 homeassistant systemd[1]: Reached target System Time Synchronized.
Nov 29 17:50:51 homeassistant systemd[1]: Started Discard unused blocks once a week.
Nov 29 17:50:51 homeassistant systemd[1]: Started Remove Bluetooth cache entries.
Nov 29 17:50:51 homeassistant systemd[1]: Reached target Timer Units.
Nov 29 17:50:51 homeassistant systemd[1]: Starting HassOS AppArmor...
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]:
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: peer: REDACTED
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: allowed ips: 172.27.69.3/32
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: persistent keepalive: every 25 seconds
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]:
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: peer: REDACTED
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: allowed ips: 172.27.69.99/32
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: persistent keepalive: every 25 seconds
Nov 29 18:10:52 homeassistant addon_core_configurator[560]: INFO:2023-11-29 19:10:52,761:hass_configurator.configurator:127.0.0.1 - "GET / HTTP/1.1" 200 -
-- Boot 9be89bbf284b4615962f2635136688c3 --
Apr 04 10:55:30 homeassistant systemd-timesyncd[546]: System clock time unset or jumped backwards, restoring from recorded timestamp: Wed 2023-11-29 18:10:21 UTC
Nov 29 18:10:21 homeassistant systemd-journald[134]: Oldest entry in /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal is older than the configured file retention duration (1month), suggesting rotation.
Nov 29 18:10:21 homeassistant systemd-journald[134]: /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal: Journal header limits reached or header out-of-date, rotating.
Nov 29 18:10:21 homeassistant systemd-time-wait-sync[545]: adjtime state 5 status 40 time Wed 2023-11-29 18:10:21.456318 UTC
Nov 29 18:10:21 homeassistant systemd-resolved[446]: Clock change detected. Flushing caches.
Nov 29 18:10:21 homeassistant systemd[1]: Started Network Time Synchronization.
Nov 29 18:10:21 homeassistant systemd[1]: Reached target System Time Set.
Nov 29 18:10:22 homeassistant kernel: bcmgenet fd580000.ethernet end0: Link is Down
Nov 29 18:10:23 homeassistant bluetoothd[525]: Path / reserved for Adv Monitor app :1.14
Nov 29 18:10:23 homeassistant bthelper[549]: hci0 new_settings: ssp br/edr le secure-conn
Nov 29 18:10:23 homeassistant bthelper[549]: [63B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: Changing power off succeeded
Nov 29 18:10:23 homeassistant bthelper[549]: [54B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: [50B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: [54B blob data]
Nov 29 18:10:23 homeassistant bluetoothd[525]: Adv Monitor app :1.14 disconnected from D-Bus
Nov 29 18:10:23 homeassistant bluetoothd[525]: Path / reserved for Adv Monitor app :1.15
Nov 29 18:10:23 homeassistant bthelper[524]: [63B blob data]
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: allowed ips: 172.27.69.99/32
Nov 29 18:10:51 homeassistant addon_a0d7b954_wireguard[560]: persistent keepalive: every 25 seconds
Nov 29 18:10:52 homeassistant addon_core_configurator[560]: INFO:2023-11-29 19:10:52,761:hass_configurator.configurator:127.0.0.1 - "GET / HTTP/1.1" 200 -
-- Boot 9be89bbf284b4615962f2635136688c3 --
Apr 04 10:55:30 homeassistant systemd-timesyncd[546]: System clock time unset or jumped backwards, restoring from recorded timestamp: Wed 2023-11-29 18:10:21 UTC
Nov 29 18:10:21 homeassistant systemd-journald[134]: Oldest entry in /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal is older than the configured file retention duration (1month), suggesting rotation.
Nov 29 18:10:21 homeassistant systemd-journald[134]: /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal: Journal header limits reached or header out-of-date, rotating.
Nov 29 18:10:21 homeassistant systemd-time-wait-sync[545]: adjtime state 5 status 40 time Wed 2023-11-29 18:10:21.456318 UTC
Nov 29 18:10:21 homeassistant systemd-resolved[446]: Clock change detected. Flushing caches.
Nov 29 18:10:21 homeassistant systemd[1]: Started Network Time Synchronization.
Nov 29 18:10:21 homeassistant systemd[1]: Reached target System Time Set.
Nov 29 18:10:22 homeassistant kernel: bcmgenet fd580000.ethernet end0: Link is Down
Nov 29 18:10:23 homeassistant bluetoothd[525]: Path / reserved for Adv Monitor app :1.14
Nov 29 18:10:23 homeassistant bthelper[549]: hci0 new_settings: ssp br/edr le secure-conn
Nov 29 18:10:23 homeassistant bthelper[549]: [63B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: Changing power off succeeded
Nov 29 18:10:23 homeassistant bthelper[549]: [54B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: [50B blob data]
Nov 29 18:10:23 homeassistant bthelper[549]: [54B blob data]
Nov 29 18:10:23 homeassistant bluetoothd[525]: Adv Monitor app :1.14 disconnected from D-Bus
Nov 29 18:10:23 homeassistant bluetoothd[525]: Path / reserved for Adv Monitor app :1.15
Nov 29 18:10:23 homeassistant bthelper[524]: [63B blob data]
Nov 29 18:10:23 homeassistant bthelper[524]: AdvertisementMonitor path registered
Nov 29 18:10:23 homeassistant bluetoothd[525]: Adv Monitor app :1.15 disconnected from D-Bus
Nov 29 18:10:25 homeassistant NetworkManager[452]: <info> [1701281425.2762] device (end0): carrier: link connected
Nov 29 18:10:25 homeassistant kernel: bcmgenet fd580000.ethernet end0: Link is Up - 100Mbps/Half - flow control off
Nov 29 18:10:26 homeassistant kernel: bcmgenet fd580000.ethernet end0: Link is Down
Nov 29 18:10:31 homeassistant systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Nov 29 18:10:32 homeassistant NetworkManager[452]: <info> [1701281432.3017] device (end0): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Nov 29 18:10:32 homeassistant systemd-resolved[446]: end0: Bus client set default route setting: no
Nov 29 18:10:32 homeassistant systemd-resolved[446]: end0: Bus client set MulticastDNS setting: no
Nov 29 18:10:32 homeassistant systemd-resolved[446]: end0: Bus client reset DNS server list.
Nov 29 18:10:32 homeassistant NetworkManager[452]: <info> [1701281432.3440] manager: NetworkManager state is now CONNECTED_LOCAL
Nov 29 18:10:32 homeassistant NetworkManager[452]: <info> [1701281432.3478] manager: NetworkManager state is now DISCONNECTED
I’m no expert, but these freezes do look like a dying power supply to me…
Okay, I finally replaced the power supply… and it crashed less than 24h later during the night.
No connection from SSH, no output to an external monitor (I could see the CLI after a restart), just a strange loop of the activity lights (the blinking happens multiple times, you can see only one long pulse because of the framerate).
When I come back this evening I will check the journals again.
I have the same setup, RPi4 in a Argon One case with a Transcend SSD 256Gb and an original power supply, running since april without any issues.
I had crashes before, but that was because of Pi-hole running on the same RPi, I had too many upstream DNS servers active in Pi-hole and was overloading the network leading to crashes.
I checked the journal and this is what happened this time:
Dec 03 04:52:49 homeassistant addon_a0d7b954_wireguard[552]: persistent keepalive: every 25 seconds
Dec 03 04:52:53 homeassistant systemd[1]: run-docker-runtime\x2drunc-moby-789cf4f036aabcee80ba880c499a2da317a9bd0c11f68b0e4b8d1a6510411e34-runc.Yl3SMQ.mount: Deactivated successfully.
Dec 03 04:52:53 homeassistant systemd[1]: run-docker-runtime\x2drunc-moby-dd588a41fbb194732aaafb6dba7a574aa65cbf8b8f87248db3b8e1bec5a8ec67-runc.yKxPGx.mount: Deactivated successfully.
Dec 03 04:52:53 homeassistant hassio_dns[552]: [INFO] 172.30.32.1:38852 - 61180 "A IN wlan0.local.hass.io. udp 37 false 512" NXDOMAIN qr,aa,rd 37 0.000430194s
Dec 03 04:53:02 homeassistant systemd[1]: run-docker-runtime\x2drunc-moby-9a0aeae14fb260a9c874dc833ae0f41cdf3a0fd352d33d8dfb319b4864229f76-runc.4TvU4N.mount: Deactivated successfully.
Dec 03 04:53:02 homeassistant addon_core_configurator[552]: INFO:2023-12-03 05:53:02,094:hass_configurator.configurator:127.0.0.1 - "GET / HTTP/1.1" 200 -
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: [05:53:20] INFO: Requesting current status from WireGuard...
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: interface: wg0
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: public key: REDACTED
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: private key: (hidden)
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: listening port: 51820
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]:
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: peer: REDACTED
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: allowed ips: 172.27.69.2/32
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: persistent keepalive: every 25 seconds
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]:
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: peer: REDACTED
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: allowed ips: 172.27.69.3/32
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: persistent keepalive: every 25 seconds
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]:
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: peer: REDACTED
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: allowed ips: 172.27.69.99/32
Dec 03 04:53:20 homeassistant addon_a0d7b954_wireguard[552]: persistent keepalive: every 25 seconds
Dec 03 04:53:25 homeassistant hassio_supervisor[552]: 23-12-03 05:53:25 INFO (MainThread) [supervisor.api.middleware.security] /network/interface/default/info access from core_duckdns
Dec 03 04:53:26 homeassistant addon_core_duckdns[552]: [05:53:26] INFO: OK
Dec 03 04:53:26 homeassistant addon_core_duckdns[552]:
Dec 03 04:53:26 homeassistant addon_core_duckdns[552]: REDACTED
Dec 03 04:53:26 homeassistant addon_core_duckdns[552]: NOCHANGE
Dec 03 04:53:27 homeassistant addon_core_duckdns[552]: [05:53:27] INFO: OK
Dec 03 04:53:27 homeassistant addon_core_duckdns[552]: REDACTED
Dec 03 04:53:27 homeassistant addon_core_duckdns[552]:
Dec 03 04:53:27 homeassistant addon_core_duckdns[552]: NOCHANGE
Dec 03 04:53:27 homeassistant hassio_supervisor[552]: 23-12-03 05:53:27 INFO (MainThread) [supervisor.homeassistant.secrets] Request secret DuckDNS_Domain
Dec 03 04:53:27 homeassistant hassio_supervisor[552]: 23-12-03 05:53:27 INFO (MainThread) [supervisor.homeassistant.secrets] Request secret DuckDNS_VPN_Domain
Dec 03 04:53:27 homeassistant hassio_supervisor[552]: 23-12-03 05:53:27 INFO (MainThread) [supervisor.homeassistant.secrets] Request secret DuckDNS_Token
Dec 03 04:53:27 homeassistant hassio_dns[552]: [INFO] 172.30.32.1:35023 - 60901 "A IN wlan0.local.hass.io. udp 37 false 512" NXDOMAIN qr,aa,rd 37 0.000347011s
Dec 03 04:53:32 homeassistant addon_core_configurator[552]: INFO:2023-12-03 05:53:32,318:hass_configurator.configurator:127.0.0.1 - "GET / HTTP/1.1" 200 -
-- Boot 788b24d6ef574dc18379551754612bbb --
Apr 04 10:55:33 homeassistant systemd-timesyncd[550]: System clock time unset or jumped backwards, restoring from recorded timestamp: Sun 2023-12-03 04:54:01 UTC
Dec 03 12:49:54 homeassistant systemd-journald[135]: Oldest entry in /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal is older than the configured file retention duration (1month), suggesting rotation.
Dec 03 12:49:54 homeassistant systemd-journald[135]: /var/log/journal/6c2208e0ffb14c36991568c05da0a1d2/system.journal: Journal header limits reached or header out-of-date, rotating.
Dec 03 12:49:54 homeassistant systemd-time-wait-sync[549]: adjtime state 5 status 40 time Sun 2023-12-03 04:54:01.072439 UTC
Dec 03 12:49:54 homeassistant systemd-time-wait-sync[549]: adjtime state 0 status 2000 time Sun 2023-12-03 12:49:54.630305 UTC
Dec 03 04:54:01 homeassistant systemd-resolved[441]: Clock change detected. Flushing caches.
Dec 03 04:54:01 homeassistant systemd[1]: Started Network Time Synchronization.
Dec 03 04:54:01 homeassistant systemd[1]: Reached target System Time Set.
Dec 03 12:49:54 homeassistant systemd-timesyncd[550]: Contacted time server 162.159.200.123:123 (time.cloudflare.com).
Dec 03 12:49:54 homeassistant systemd-resolved[441]: Clock change detected. Flushing caches.
Dec 03 12:49:54 homeassistant systemd-timesyncd[550]: Initial clock synchronization to Sun 2023-12-03 12:49:54.630180 UTC.
Dec 03 12:49:54 homeassistant systemd[1]: Finished Wait Until Kernel Time Synchronized.
Dec 03 12:49:54 homeassistant systemd[1]: Reached target System Time Synchronized.
Dec 03 12:49:54 homeassistant systemd[1]: Started Discard unused blocks once a week.
Dec 03 12:49:54 homeassistant systemd[1]: Started Remove Bluetooth cache entries.
Dec 03 12:49:54 homeassistant systemd[1]: Reached target Timer Units.
If you look at this log together with the logs from before, this line appeared before every single crash:
After updating to the latest version of Home Assistant OS the system hasn’t crashed even once, so maybe there’s still some hope.
The thing that looks strange to me is the complete lack of any useful information in the journals.
Since all of the people experiencing problems run on external storage, one possibility that came up on my mind might be that somehow the external ssd gets disconnected from the system. That would explain the half crashes (what is loaded into memory still works for a while) and the complete absence of logs after a reboot.
The only way to check this would be by leaving a monitor connected to the system at all times to check if the RPi generates any errors which don’t get saved on the ssd.
it’s not very consistent these random crashes, I have no issues at all.
I’m not convinced it’s just due to the SSD, it has to be a combination of factors if I had to guess.
to rule things out, I have nothing else plugged in to my Rpi, only the SSD, no memory card or other sticks.
and besides HA it runs PiHole and VNC.
Had the same issues with random crashes, googled around a bit without finding much. Finally made an account here and found this thread.
The same thing got rid of the crashes for me, simply change from USB3 port to USB2 port. Thanks!
I have the “half crashes” too…I can turn on and off lights… But the second I ask to see history, or do a reboot from the developers tab… the system hangs… and all automations stop…
I’ve been thinking about upgrading my hardware to a NUC or something similar because I’ve been having these issues persistently as well. It’s been becoming quite an annoyance.
I have a watchdog running and it does the job for the most part, but sometimes I still need to physically restart the Pi. There is another separate issue that I’m currently trying to resolve where some zigbee lights get turned on when HA starts and it’s a bit problematic if we’re away from home for a long time and the watchdog has rebooted HA, but HA crashed immediately after and without physically rebooting the pi, I have no way of turning those lights off until we’re back home. We have of course resorted to turning off all those lights when we leave, but it’s not ideal when we have them running presence simulation when we’re away.
I’m hoping a hardware upgrade would resolve the issue, but I thought I’d ask here if anyone can confirm.
I switched to a dell optiplex 3050 usff pc. I bought a used one off ebay for less than $100. Much faster, more storage, more memory, reliable power supply … Just an all around a better experience than the pi for about the same price.
I have a similar unresolved problem. Periodically, both network interfaces fail and control of automations stops at this time.
Core2024.3.3
Supervisor2024.03.0
Operating System12.1
Інтерфейс20240307.0
RPI4, RAM2Gb, boot from SD-card
Mar 23 10:14:37 rpi4.lan systemd[1]: run-docker-runtime\x2drunc-moby-0520e488e292ea3c81119493b9af5f51f71802ab78199ab713626182c60507f3-runc.PCBrdW.mount: Deactivated successfully.
Mar 23 10:14:54 rpi4.lan systemd[1]: run-docker-runtime\x2drunc-moby-77ae6b61fe18c48d7d39987065cca82a99971fadd29e3fd6b0d672053d6f9a2c-runc.TZyO4m.mount: Deactivated successfully.
Mar 23 10:14:56 rpi4.lan kernel: bcmgenet fd580000.ethernet end0: Link is Down
Mar 23 10:15:02 rpi4.lan NetworkManager[468]: <info> [1711188902.3608] device (end0): state change: activated -> unavailable (reason 'carrier-changed', sys-iface-state: 'managed')
Mar 23 10:15:02 rpi4.lan systemd-resolved[466]: end0: Bus client set default route setting: no
Mar 23 10:15:02 rpi4.lan systemd-resolved[466]: end0: Bus client set MulticastDNS setting: no
Mar 23 10:15:02 rpi4.lan systemd-resolved[466]: end0: Bus client reset DNS server list.
Mar 23 10:15:02 rpi4.lan NetworkManager[468]: <info> [1711188902.4328] policy: set 'Supervisor wlan0' (wlan0) as default for IPv4 routing and DNS
Mar 23 10:15:02 rpi4.lan systemd[1]: Starting Network Manager Script Dispatcher Service...
Mar 23 10:15:02 rpi4.lan systemd[1]: Started Network Manager Script Dispatcher Service.
