AppDaemon apps freezing

It seems like my appdaemon instance froze. It’s running in a docker using the latest tag, seems to be from 2019-05-12, but I have to admit I am not sure which version I am running.

I noticed all my light apps stopped responding (no light changes, no logs). However, I get log messages from dashboard and another app. Hence, it seems like something killed all my light apps (I run multiple instances of the same app). This has happened at least once before. I have not restarted AppDaemon, so that I can do any debugging if you want me to.

Here are excerpts from the last logs of the last “smartlight” instance, in addition to the app that is still “running”:

2019-10-10 19:04:11.876415 INFO function [get_dashboard] finished in 12 ms
2019-10-10 19:14:11.923473 INFO function [get_dashboard] finished in 10 ms
2019-10-10 19:20:49.713631 INFO storage_laundry_smart_light: ACTIVE_SUSTAINED - TIMEOUT -> IDLE
2019-10-10 19:20:49.749411 INFO storage_laundry_smart_light: Enabling deferred override check
2019-10-10 19:20:49.750609 INFO storage_laundry_smart_light: Setting {'light.storage_roof_light': 0.0, 'light.laundry_roof_light': 0.0}
2019-10-10 19:21:49.002992 INFO storage_laundry_smart_light: Disabling deferred override check
2019-10-10 19:24:12.017401 INFO function [get_dashboard] finished in 11 ms
2019-10-10 19:34:12.120881 INFO function [get_dashboard] finished in 10 ms
2019-10-10 19:44:12.117963 INFO function [get_dashboard] finished in 12 ms
2019-10-10 19:54:12.221371 INFO function [get_dashboard] finished in 12 ms
2019-10-10 20:00:00.024363 INFO housekeeper: zwave.bathroom_thermostat has low battery: 0%
2019-10-10 20:00:00.058284 INFO housekeeper: zwave.z_wave_me_zme_uzb1_usb_stick has not been updated in 4 days, 8:43:39.180862
2019-10-10 20:00:00.077170 INFO housekeeper: zwave.living_room_blinds_2 has not been updated in 4 days, 8:39:58.684152
2019-10-10 20:00:00.099797 INFO housekeeper: binary_sensor.kitchen_leakage_1 is unavailable
2019-10-10 20:00:00.105420 INFO housekeeper: binary_sensor.bathroom_leakage is unavailable
2019-10-10 20:00:00.110899 INFO housekeeper: light.garden_light is unavailable
2019-10-10 20:04:12.294474 INFO function [get_dashboard] finished in 32 ms
2019-10-10 20:14:12.339339 INFO function [get_dashboard] finished in 10 ms
...
2019-10-14 19:30:20.954400 INFO function [get_dashboard] finished in 10 ms
2019-10-14 19:40:21.046237 INFO function [get_dashboard] finished in 10 ms
2019-10-14 19:50:21.117155 INFO function [get_dashboard] finished in 11 ms
2019-10-14 20:00:00.041406 INFO housekeeper: zwave.bathroom_thermostat has low battery: 0%
2019-10-14 20:00:00.059056 INFO housekeeper: zwave.z_wave_me_zme_uzb1_usb_stick has not been updated in 8 days, 8:43:39.181687
2019-10-14 20:00:00.066255 INFO housekeeper: zwave.living_room_blinds_2 has not been updated in 8 days, 8:39:58.673233
2019-10-14 20:00:00.086526 INFO housekeeper: binary_sensor.kitchen_leakage_1 is unavailable
2019-10-14 20:00:00.092004 INFO housekeeper: binary_sensor.bathroom_leakage is unavailable
2019-10-14 20:00:00.097142 INFO housekeeper: light.garden_light is unavailable
2019-10-14 20:00:21.175455 INFO function [get_dashboard] finished in 11 ms
2019-10-14 20:10:21.214910 INFO function [get_dashboard] finished in 11 ms
2019-10-14 20:20:21.297927 INFO function [get_dashboard] finished in 10 ms

appdaemon.yaml:

log:
  logfile: STDOUT
  errorfile: STDERR
appdaemon:
  threads: 10
  plugins:
    HASS:
      type: hass
      ha_url: http://hass.iot:8123
      token: !secret appdaemon_token
hadashboard:
  dash_url: http://0.0.0.0:5050

SHA of docker image:

263281461a80e308468554c2413f8e2debbca0805d1f1b3bd9462fd951311ee7

It looks like a lot of your devices are unavailable according to your “housekeeper” app. Are you sure that the devices were available when appdaemon froze? Can you show the code of an app that froze?