How to make Homeassistant more Crash Proof

Shadowtm · March 7, 2022, 11:27am

Hello Guys,

I have the following problem.
My Home Assistant Install on my RPI3 crashes from time to time, can be months, can be hours.
The problem with this is, that the SSH Server becomes unresponsive and thus I can not shutdown the Pi gracefully, often resulting in a destroyed file system and therefore a reinstall.

This is less than optimal and I don’t understand why this keeps happening.
Even if there is a memory leek in some addon or whatever, why does it keep crashing my ssh Server and what can I do to stop this since I don’t want to reinstall a few times a year.

Thanks in Advance.

RogTP · March 7, 2022, 12:56pm

Are you trying to SSH in to the Pi using the HA SSH add-on, or (I assume you are using Hassos) at the is level? If you can get it at os level you might be able to run ‘top’ to see what is killing it. Or at least do a sudo reboot.

Shadowtm · March 7, 2022, 1:08pm

I have it instaleld as the full OS Version on the Pi (HA Image directly put onto SD Card)
I am using the Terminal & SSH extension and when this happens everything is dead. Webinterface is dead, ssh is dead. I mean you could maybe plug a monitor and keyboard in, but the pi is in a remote room.

Is there some way to make “hardware soft shutdown” button so it won’t kill my entire setup every time this happens ?

Btw this has persistet through multiple reinstalls, though they all have the same backup loaded.

tmjpugh · March 7, 2022, 1:32pm

Why is it crashing? This is what she be resolved. Would be helpful for you to determine if Only HA crashes or entire OS crashes. Hook up a monitor and look or some other method to figure this out. No flashing green led on Pi PWB is sign OS down I believe.

SD Card failure or less then great Power Supply may cause this. Do you use logging? Could sd card be filling up and crashing system due to disk full.

I’ve only seen crash when disk full, disk broke or power supply fail. All but disk broke issue we’re caused by my security cam videos or other service using up all resources.

Make backups and figure out cause of crash

Shadowtm · March 8, 2022, 10:45am

Thanks for your thoughts, I am using the original Raspberrypi Power Supply and a 32 GB Sundisk Ultra SD Card.
Do you have any hints how I could find out if anything is damaged besides looking at the remaining space.

Shadowtm · March 9, 2022, 1:13pm

To add, I think even my Backup is corrupted.
I’m gonna start from Scratch and just copy over my automations and config.

Shadowtm · March 10, 2022, 1:12pm

So I have setup a completely new Home Assistant Instance and I get this:
grafik

Also all my Zigbee Devices are still connected for some reason ?

Is this a Bug in 7.4 ?
Should I stay on 7.0 for a while ?

Edit: also my Core Log is littered with a shit ton of Warnings and Errors, mostly about ZigBee Devices.

How can I reset my ZigBee Network ?
Why did it keep the network over the reinstall anyway ?

Shadowtm · March 10, 2022, 9:40pm

Ok after resetting everything, it still hangs and crashes after e.G. Add On Installs.
Is it maybe that the pi 3 just isn’t good enough?
Especially the 1 GB Ram seems too low.

tmjpugh · March 11, 2022, 3:13am

Replace sd card
Verify power supply, maybe replace it also

Sd cards on pi notoriously break and result in random crashes. Hard to tell if it’s the problem but cheap to replace or easy to temporarily replace to verify

Bad or “not quite up to spec” power supplies cause issues as well

CentralCommand · March 11, 2022, 4:04am

Your zigbee devices are talking to the coordinator (i.e. the USB stick). Home assistant is just listening to what the USB tells it.

So reinstall of home assistant doesn’t change anything there. All your zigbee are still paired with the coordinator and keep right on talking to it. And the new install of HA is still just listening. This isn’t a bug and has always been the case, it’s not a 7.4 thing.

As for how to reset your network, depends on what integration you’re using. Since you’re seeing errors in your core logs I assume you’re using either zha or deconz and not z2m. With zha easiest way to completely disconnect is to change the channel. I’m not sure what option there is for deconz since I don’t use it and don’t see anything in the doc. Maybe there’s some advice around here on the forum or could ask in the zigbee channel of the HA discord.

Although you may want to unpair some devices individually first. Otherwise the devices will still think they’re paired which may make them difficult to pair again (like lightbulbs for example).

Also how much is “a shitton”? Enough to fill up your disk space with logs?

Shadowtm · March 11, 2022, 10:06am

I have an original Raspberrypi Power Supply, the Power Supply Check Addon also reports OK without any problems:
grafik

I also tested the SD Card with f3read and f3write on my macbook and it said the sd card is healthy.

Shadowtm · March 11, 2022, 10:09am

As for Errors, this is the Core log since last reboot (has been more Errors before)

2022-03-10 22:49:22 DEBUG (MainThread) [zigpy_znp.api] Sending request: SYS.Ping.Req()
2022-03-10 22:49:22 DEBUG (MainThread) [zigpy_znp.api] Received command: SYS.Ping.Rsp(Capabilities=<MTCapabilities.APP_CNF|GP|UTIL|ZDO|AF|SYS: 1625>)
2022-03-10 22:53:02 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-10 23:22:17 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>')]
2022-03-10 23:22:17 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>')]
2022-03-10 23:22:38 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-10 23:22:38 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-10 23:41:16 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-10 23:41:16 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-10 23:41:34 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-10 23:42:50 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-10 23:45:36 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-10 23:56:14 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-10 23:56:14 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 00:26:30 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 00:27:33 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 01:00:34 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 01:00:34 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 01:07:00 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 01:07:00 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 02:23:23 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 02:23:23 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 02:32:38 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/zigpy/topology.py", line 52, in scan_loop
    await self.scan()
  File "/usr/local/lib/python3.9/site-packages/zigpy/topology.py", line 63, in scan
    await self._scan_task
  File "/usr/local/lib/python3.9/site-packages/zigpy/topology.py", line 86, in _scan
    await device.neighbors.scan()
  File "/usr/local/lib/python3.9/site-packages/zigpy/neighbor.py", line 94, in scan
    return await self._scan()
  File "/usr/local/lib/python3.9/site-packages/zigpy/neighbor.py", line 105, in _scan
    status, rsp = await self._device.zdo.Mgmt_Lqi_req(idx, tries=3, delay=1)
  File "/usr/local/lib/python3.9/site-packages/zigpy/util.py", line 110, in retry
    r = await func()
  File "/usr/local/lib/python3.9/site-packages/zigpy/device.py", line 265, in request
    result, msg = await self._application.request(
  File "/usr/local/lib/python3.9/site-packages/zigpy_znp/zigbee/application.py", line 439, in request
    return await self._send_request(
  File "/usr/local/lib/python3.9/site-packages/zigpy_znp/zigbee/application.py", line 1401, in _send_request
    response = await self._send_request_raw(
  File "/usr/local/lib/python3.9/site-packages/zigpy_znp/zigbee/application.py", line 1279, in _send_request_raw
    response = await self._znp.request(
AttributeError: 'NoneType' object has no attribute 'request'
2022-03-11 02:42:02 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 02:42:02 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 02:56:56 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 02:56:56 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 03:24:30 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.MAC_NO_ACK: 233>')]
2022-03-11 03:24:31 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>')]
2022-03-11 04:01:16 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 04:01:16 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 04:07:43 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 04:07:43 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 05:24:09 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 05:24:09 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 05:42:48 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 05:42:48 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 05:57:14 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 05:57:14 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 06:22:20 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 06:23:29 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 06:35:59 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 06:37:04 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 06:42:09 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 06:53:31 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 06:55:46 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 06:57:33 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 07:00:32 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:01:38 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:01:59 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 07:01:59 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 07:04:38 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:06:49 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:08:27 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 07:08:27 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 07:21:31 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:23:43 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:25:22 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0006]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>')]
2022-03-11 07:25:22 WARNING (MainThread) [homeassistant.components.zha.core.channels.base] [0x6DF4:11:0x0008]: async_initialize: all attempts have failed: [DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>'), DeliveryError('Request failed after 5 attempts: <Status.NWK_NO_ROUTE: 205>')]
2022-03-11 07:57:32 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 07:58:51 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 08:24:56 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 08:24:56 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 08:43:30 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 08:43:30 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 08:54:08 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 08:54:15 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x6246
2022-03-11 08:57:51 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 08:57:51 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x568C
2022-03-11 09:09:19 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 09:10:24 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 10:02:41 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 10:02:41 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2B1
2022-03-11 10:09:07 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 10:09:07 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0x59A8
2022-03-11 10:57:01 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 10:59:09 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD
2022-03-11 11:04:03 WARNING (MainThread) [zigpy_znp.zigbee.application] Received an AF message from an unknown device: 0xF2CD

I suspect it to be a problem with ram since it especially occurs when I try to install additional Addons (already have adguard, Vaultwarden, Letsencrypt and VS Code Server installed and running).

Maybe I’ll try to install it onto an my old Thinkpad to see if it is more stable and treat myself to a rpi 4 with 4 or 8 gigabytes of ram once those are obtainable and affordable again.

Oh and btw I have ZHA with a SONOFF USB Zigbee Dongle.