Hi all,
For the last couple of weeks I’ve experienced that AppDaemon crashes completely and hangs, seems like everything stops and so far the only thing I’ve been able to do is restart the AD plugin and everything returns to normal. It does not happen daily, maybe more like weekly.
Not sure if this is an AD-issue, if it is one or several of my apps or if it is a Home Assistant issue. But it is really annoying, because it makes my system unreliable and unstable.
I’ve just updated everything today, but was on HA v110.3, Hassos 3.13, AD 0.2.5 when this happened. I’m on Rpi3. A bit hard to give exact log info, but this is what happens in AD log when it happens:
2020-06-06 09:34:57.259916 INFO AppDaemon: --------------------------------------------------
2020-06-06 09:34:57.261254 INFO AppDaemon: Threads
2020-06-06 09:34:57.262315 INFO AppDaemon: --------------------------------------------------
2020-06-06 09:34:57.265195 INFO AppDaemon: Currently busy threads: 0
2020-06-06 09:34:57.267564 INFO AppDaemon: Most used threads: 3 at 2020-06-04 07:05:01+02:00
2020-06-06 09:34:57.270415 INFO AppDaemon: Last activity: 2020-06-04T07:05:01+02:00
2020-06-06 09:34:57.273238 INFO AppDaemon: Total Q Entries: 409
2020-06-06 09:34:57.275270 INFO AppDaemon: --------------------------------------------------
2020-06-06 09:34:57.278314 INFO AppDaemon: thread-0 - qsize: 0 | current callback: idle | since 2020-06-06T09:05:00+02:00, | alive: True, | pinned apps: ['Garbage']
2020-06-06 09:34:57.280501 INFO AppDaemon: thread-1 - qsize: 0 | current callback: idle | since 2020-06-06T09:18:21+02:00, | alive: True, | pinned apps: ['RSS']
2020-06-06 09:34:57.283109 INFO AppDaemon: thread-2 - qsize: 204 | current callback: idle | since 2020-06-06T06:15:08+02:00, | alive: True, | pinned apps: ['HeatPump']
2020-06-06 09:34:57.285532 INFO AppDaemon: thread-3 - qsize: 0 | current callback: idle | since 2020-06-06T09:18:20+02:00, | alive: True, | pinned apps: ['Yr']
2020-06-06 09:34:57.290411 INFO AppDaemon: thread-4 - qsize: 205 | current callback: idle | since 2020-06-06T06:15:13+02:00, | alive: True, | pinned apps: ['AlarmClock']
2020-06-06 09:34:57.292129 INFO AppDaemon: --------------------------------------------------
2020-06-06 09:34:57.294597 CRITICAL AppDaemon: Thread thread-2 has died
2020-06-06 09:34:57.296758 CRITICAL AppDaemon: Pinned apps were: ['HeatPump']
2020-06-06 09:34:57.298434 CRITICAL AppDaemon: Thread will be restarted
2020-06-06 09:34:57.300516 INFO AppDaemon: Adding thread 2
and so it continues telling me thread by thread that the they died, trying again and so forth. It’s filling up my log with these messages until I restart the AD plugin. I’ve got 5 apps running and they all seem to be doing fine before everything crashes at the same time, that makes me believe it is not related to a specific bug in my apps since they’re all different. It may very well be though, and I have seen some “random” concurrent.futures._base.TimeoutError errors every now and then.
I’ve also experienced that HA seems to having issues, but I didn’t find anything specific in the logs except warnings and errors from “all” of my stuff in there at a specific time, but then it seems to recover. I noticed the CPU- and memory sensors were also showing a sudden peak in load, but not sure if it is a cause or an effect. At least twice when this happened it was more or less the same time around 06:15 in the morning or so (and my system or apps are not setup to do anything particular at that time). Not sure if this is relevant, but… SD-card issues came to my mind, but isn’t it funny it seems to be working fine after a restart then?
Anyone that could guide me in the right direction for further troubleshooting or have had similar issues?
Thanks in advance.