For the second time in the last week or so, I’ve woken up to see my Home Assistant is no longer running. I’m running “Home Assistant Supervised” on a NUC clone, on (latest as I post) 115.6, with Supervisor 245.
Rather than just a reboot which got it working last time I thought I’d first make some effort to work out why it did it! I can still ssh into the box and I can see all the hassio_* containers are up and running, but home assistant itself and all the addons have died:
root@zbox:~# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
1a632a08ccc9 homeassistant/amd64-hassio-multicast:3 "/init" 11 hours ago Up 11 hours hassio_multicast
199b3d59d750 homeassistant/amd64-hassio-cli:26 "/init /bin/bash -c …" 11 hours ago Up 11 hours hassio_cli
beb5cab91229 homeassistant/amd64-hassio-audio:17 "/init" 11 hours ago Up 11 hours hassio_audio
fe7053a9f2fa homeassistant/amd64-hassio-dns:9 "/init coredns -conf…" 11 hours ago Up 11 hours hassio_dns
5c81a637ba5b hassioaddons/log-viewer-amd64:0.8.1 "/init" 39 hours ago Exited (0) 11 hours ago addon_a0d7b954_logviewer
8e7d59162259 homeassistant/amd64-addon-configurator:5.0.0 "/init" 39 hours ago Exited (0) 11 hours ago addon_core_configurator
b869801fdb40 esphome/esphome-hassio-amd64:1.14.5 "/init" 39 hours ago Exited (129) 11 hours ago addon_15ef4d2f_esphome
66374ad3583b hassioaddons/vscode-amd64:2.7.1 "/init" 39 hours ago Exited (0) 11 hours ago addon_a0d7b954_vscode
d87878e5aa79 hassioaddons/portainer-amd64:1.2.2 "/init" 39 hours ago Exited (0) 11 hours ago addon_a0d7b954_portainer
7c9a5e938871 hassioaddons/grafana-amd64:5.3.0 "/init" 39 hours ago Exited (129) 11 hours ago addon_a0d7b954_grafana
f5973c66ecd0 hassioaddons/influxdb-amd64:3.7.6 "/init" 39 hours ago Exited (0) 11 hours ago addon_a0d7b954_influxdb
f46d447ce251 hassioaddons/glances-amd64:0.8.0 "/init" 39 hours ago Exited (0) 11 hours ago addon_a0d7b954_glances
6eb15ac5e24e homeassistant/qemux86-64-homeassistant:0.115.6 "/init" 46 hours ago Exited (0) 11 hours ago homeassistant
38579fe29ad3 homeassistant/amd64-hassio-supervisor "/init" 2 weeks ago Up 11 hours hassio_supervisor
9571021e706c homeassistant/amd64-hassio-observer:3 "/init" 2 weeks ago Up 11 hours 0.0.0.0:4357->80/tcp hassio_observer
6cc2a33384e0 homeassistant/amd64-addon-check_config:3.3.0 "/init" 4 months ago Exited (0) 4 months ago addon_core_check_config
I can see the web page for the observer, but it simply tells me that the Supervisor is connected.
This is what the end of the home assistant log looks like:
2020-10-03 00:46:19 ERROR (SyncWorker_17) [homeassistant.components.influxdb] InfluxDB database is not accessible due to '404: {"error":"database not found: \"home_assistant\""}
'. Please check that the database, username and password are correct and that the specified user has the correct permissions set. Retrying in 60 seconds.
2020-10-03 00:47:19 ERROR (SyncWorker_1) [homeassistant.components.influxdb] InfluxDB database is not accessible due to '404: {"error":"database not found: \"home_assistant\""}
'. Please check that the database, username and password are correct and that the specified user has the correct permissions set. Retrying in 60 seconds.
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
2020-10-03 00:48:04 WARNING (MainThread) [homeassistant.components.onvif] Failed to fetch ONVIF PullPoint subscription messages for '9c:14:63:87:cf:ad'
2020-10-03 00:48:04 WARNING (MainThread) [homeassistant.components.onvif] Failed to fetch ONVIF PullPoint subscription messages for '18:68:cb:28:bb:d7'
2020-10-03 00:48:06 ERROR (MainThread) [homeassistant] Error doing job: Future exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
TypeError: stop() takes 1 positional argument but 2 were given
Update for binary_sensor.ping_8_8_8_8 fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 469, in async_device_update
await self.async_update() # type: ignore
File "/usr/src/homeassistant/homeassistant/components/ping/binary_sensor.py", line 117, in async_update
await self._ping.async_update()
File "/usr/src/homeassistant/homeassistant/components/ping/binary_sensor.py", line 138, in async_update
data = await self.hass.async_add_executor_job(
File "/usr/src/homeassistant/homeassistant/core.py", line 350, in async_add_executor_job
task = self.loop.run_in_executor(None, target, *args)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 783, in run_in_executor
executor.submit(func, *args), loop=self)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 179, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Update for media_player.squeezebox_radio fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 469, in async_device_update
await self.async_update() # type: ignore
File "/usr/src/homeassistant/homeassistant/components/squeezebox/media_player.py", line 311, in async_update
await self._player.async_update()
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/player.py", line 255, in async_update
response = await self.async_query("status", "-", "1", f"tags:{tags}")
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/player.py", line 246, in async_query
return await self._lms.async_query(*parameters, player=self._id)
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/server.py", line 156, in async_query
response = await self.session.post(url, data=data, auth=auth)
File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 357, in _request
raise RuntimeError('Session is closed')
RuntimeError: Session is closed
Update for media_player.squeezebox_touch fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 469, in async_device_update
await self.async_update() # type: ignore
File "/usr/src/homeassistant/homeassistant/components/squeezebox/media_player.py", line 311, in async_update
await self._player.async_update()
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/player.py", line 255, in async_update
response = await self.async_query("status", "-", "1", f"tags:{tags}")
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/player.py", line 246, in async_query
return await self._lms.async_query(*parameters, player=self._id)
File "/usr/local/lib/python3.8/site-packages/pysqueezebox/server.py", line 156, in async_query
response = await self.session.post(url, data=data, auth=auth)
File "/usr/local/lib/python3.8/site-packages/aiohttp/client.py", line 357, in _request
raise RuntimeError('Session is closed')
RuntimeError: Session is closed
Update for binary_sensor.ping_8_8_8_8 fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 469, in async_device_update
await self.async_update() # type: ignore
File "/usr/src/homeassistant/homeassistant/components/ping/binary_sensor.py", line 117, in async_update
await self._ping.async_update()
File "/usr/src/homeassistant/homeassistant/components/ping/binary_sensor.py", line 138, in async_update
data = await self.hass.async_add_executor_job(
File "/usr/src/homeassistant/homeassistant/core.py", line 350, in async_add_executor_job
task = self.loop.run_in_executor(None, target, *args)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 783, in run_in_executor
executor.submit(func, *args), loop=self)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 179, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Update for switch.l_subwoofer_248 fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 471, in async_device_update
await self.hass.async_add_executor_job(self.update) # type: ignore
File "/usr/src/homeassistant/homeassistant/core.py", line 350, in async_add_executor_job
task = self.loop.run_in_executor(None, target, *args)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 783, in run_in_executor
executor.submit(func, *args), loop=self)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 179, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
Update for light.h_lamp_79 fails
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 278, in async_update_ha_state
await self.async_device_update()
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 471, in async_device_update
await self.hass.async_add_executor_job(self.update) # type: ignore
File "/usr/src/homeassistant/homeassistant/core.py", line 350, in async_add_executor_job
task = self.loop.run_in_executor(None, target, *args)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 783, in run_in_executor
executor.submit(func, *args), loop=self)
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 179, in submit
raise RuntimeError('cannot schedule new futures after shutdown')
RuntimeError: cannot schedule new futures after shutdown
s6-svscanctl: fatal: unable to control /var/run/s6/services: supervisor not listening
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
root@zbox:~#
I see that “supervisor not listening” error is mentioned here:
Could that be my issue?
There were lots of those InfluxDB errors before this. I’d been having issues getting connection to the InfluxDB addon working yesterday. Had abandoned it to have another look today.
Here’s the log from the Supervisor, from before until after the crash:
20-10-02 23:48:04 INFO (MainThread) [supervisor.misc.scheduler] Shutdown scheduled tasks
20-10-02 23:48:04 INFO (MainThread) [supervisor.api] Stop API on 172.30.32.2
20-10-02 23:48:04 INFO (MainThread) [supervisor.misc.hwmon] Stop Supervisor hardware monitor
20-10-02 23:48:04 INFO (MainThread) [supervisor.core] Supervisor is down
[cont-finish.d] executing container finish scripts...
20-10-02 23:48:04 INFO (MainThread) [__main__] Close Supervisor
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing...
[23:48:09] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[23:48:09] INFO: Start local supervisor watchdog...
20-10-02 23:48:10 INFO (MainThread) [supervisor.bootstrap] Use the old homeassistant repository for machine extraction
20-10-02 23:48:10 INFO (MainThread) [__main__] Initialize Supervisor setup
20-10-02 23:48:10 INFO (MainThread) [supervisor.bootstrap] Initialize Supervisor Sentry
20-10-02 23:48:10 INFO (MainThread) [supervisor.bootstrap] Setup coresys for machine: qemux86-64
20-10-02 23:48:10 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attach to Supervisor homeassistant/amd64-hassio-supervisor with version 245
20-10-02 23:48:10 ERROR (MainThread) [supervisor.docker] Docker logging driver json-file is not supported!
20-10-02 23:48:10 INFO (MainThread) [__main__] Setup Supervisor
20-10-02 23:48:16 WARNING (MainThread) [supervisor.dbus.rauc] Host has no rauc support. OTA updates have been disabled.
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.info] Update local host information
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.services] Update service information
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.network] Update local network information
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.apparmor] Load AppArmor Profiles: {'hassio-supervisor'}
20-10-02 23:48:16 INFO (MainThread) [supervisor.host.services] Reload local service hassio-apparmor.service
20-10-02 23:48:16 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-dns with version 9
20-10-02 23:48:16 INFO (MainThread) [supervisor.plugins.dns] Start CoreDNS plugin
20-10-02 23:48:16 INFO (SyncWorker_0) [supervisor.docker.interface] Clean hassio_dns application
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.dns] Start DNS homeassistant/amd64-hassio-dns with version 9 - 172.30.32.3
20-10-02 23:48:17 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-audio with version 17
20-10-02 23:48:17 INFO (MainThread) [supervisor.plugins.audio] Start Audio plugin
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.interface] Clean hassio_audio application
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.audio] Start Audio homeassistant/amd64-hassio-audio with version 17 - 172.30.32.4
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-cli with version 26
20-10-02 23:48:17 INFO (MainThread) [supervisor.plugins.cli] Start cli plugin
20-10-02 23:48:17 INFO (SyncWorker_0) [supervisor.docker.interface] Clean hassio_cli application
20-10-02 23:48:18 INFO (SyncWorker_0) [supervisor.docker.cli] Start CLI homeassistant/amd64-hassio-cli with version 26 - 172.30.32.5
20-10-02 23:48:18 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-observer with version 3
20-10-02 23:48:18 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-hassio-multicast with version 3
20-10-02 23:48:18 INFO (MainThread) [supervisor.plugins.multicast] Start Multicast plugin
20-10-02 23:48:18 INFO (SyncWorker_0) [supervisor.docker.interface] Clean hassio_multicast application
20-10-02 23:48:18 INFO (SyncWorker_0) [supervisor.docker.multicast] Start Multicast homeassistant/amd64-hassio-multicast with version 3 - Host
20-10-02 23:48:18 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-10-02 23:48:19 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/qemux86-64-homeassistant with version 0.115.6
20-10-02 23:48:19 INFO (MainThread) [supervisor.homeassistant.secrets] Load Home Assistant secrets: 1
20-10-02 23:48:19 INFO (MainThread) [supervisor.hassos] No Home Assistant Operating System found
20-10-02 23:48:19 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/core repository
20-10-02 23:48:19 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/git/a0d7b954 repository
20-10-02 23:48:19 INFO (MainThread) [supervisor.store.git] Load add-on /data/addons/git/15ef4d2f repository
20-10-02 23:48:19 INFO (MainThread) [supervisor.store] Load add-ons from store: 69 all - 69 new - 0 remove
20-10-02 23:48:19 INFO (MainThread) [supervisor.addons] Found 10 installed add-ons
20-10-02 23:48:19 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to hassioaddons/influxdb-amd64 with version 3.7.6
20-10-02 23:48:19 INFO (SyncWorker_4) [supervisor.docker.interface] Attach to hassioaddons/glances-amd64 with version 0.8.0
20-10-02 23:48:19 INFO (SyncWorker_2) [supervisor.docker.interface] Attach to hassioaddons/portainer-amd64 with version 1.2.2
20-10-02 23:48:19 INFO (SyncWorker_1) [supervisor.docker.interface] Attach to hassioaddons/vscode-amd64 with version 2.7.1
20-10-02 23:48:19 INFO (SyncWorker_3) [supervisor.docker.interface] Attach to hassioaddons/log-viewer-amd64 with version 0.8.1
20-10-02 23:48:19 INFO (SyncWorker_5) [supervisor.docker.interface] Attach to esphome/esphome-hassio-amd64 with version 1.14.5
20-10-02 23:48:19 INFO (SyncWorker_7) [supervisor.docker.interface] Attach to hassioaddons/grafana-amd64 with version 5.3.0
20-10-02 23:48:19 INFO (SyncWorker_0) [supervisor.docker.interface] Attach to homeassistant/amd64-addon-configurator with version 5.0.0
20-10-02 23:48:19 INFO (SyncWorker_4) [supervisor.docker.interface] Attach to homeassistant/amd64-addon-check_config with version 3.3.0
20-10-02 23:48:19 INFO (SyncWorker_6) [supervisor.docker.interface] Attach to homeassistant/amd64-addon-samba with version 9.2.0
20-10-02 23:48:19 INFO (MainThread) [supervisor.snapshots] Found 0 snapshot files
20-10-02 23:48:19 INFO (MainThread) [supervisor.discovery] Load 0 messages
20-10-02 23:48:19 INFO (MainThread) [supervisor.ingress] Load 10 ingress session
20-10-02 23:48:19 INFO (MainThread) [__main__] Run Supervisor
20-10-02 23:48:19 INFO (MainThread) [supervisor.api] Start API on 172.30.32.2
20-10-02 23:48:19 CRITICAL (MainThread) [supervisor.core] System running in a unsupported environment!
20-10-02 23:48:19 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.INITIALIZE' start 0 add-ons
20-10-02 23:48:19 INFO (MainThread) [supervisor.core] Supervisor reboot detected
20-10-02 23:48:19 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled
20-10-02 23:48:19 INFO (MainThread) [supervisor.misc.hwmon] Started Supervisor hardware monitor
20-10-02 23:48:19 INFO (MainThread) [supervisor.core] Supervisor is up and running
20-10-02 23:48:19 INFO (MainThread) [supervisor.host.info] Update local host information
20-10-02 23:48:19 INFO (MainThread) [supervisor.host.services] Update service information
20-10-02 23:48:19 INFO (MainThread) [supervisor.host.network] Update local network information
20-10-03 01:48:19 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-10-03 01:54:59 INFO (MainThread) [supervisor.host.info] Update local host information
20-10-03 01:54:59 INFO (MainThread) [supervisor.host.services] Update service information
20-10-03 01:55:00 INFO (MainThread) [supervisor.host.network] Update local network information
20-10-03 01:55:00 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
20-10-03 02:48:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
20-10-03 02:48:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/hassio-addons repository
20-10-03 02:48:19 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/hassio repository
20-10-03 02:48:21 INFO (MainThread) [supervisor.store] Load add-ons from store: 69 all - 0 new - 0 remove
20-10-03 03:48:19 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-10-03 04:01:40 INFO (MainThread) [supervisor.host.info] Update local host information
20-10-03 04:01:40 INFO (MainThread) [supervisor.host.services] Update service information
20-10-03 04:01:40 INFO (MainThread) [supervisor.host.network] Update local network information
20-10-03 04:01:40 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
20-10-03 05:48:19 INFO (MainThread) [supervisor.updater] Fetch update data from https://version.home-assistant.io/stable.json
20-10-03 05:48:21 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/esphome/hassio repository
20-10-03 05:48:21 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/home-assistant/hassio-addons repository
20-10-03 05:48:21 INFO (MainThread) [supervisor.store.git] Update add-on https://github.com/hassio-addons/repository repository
20-10-03 05:48:22 INFO (MainThread) [supervisor.store] Load add-ons from store: 69 all - 0 new - 0 remove
20-10-03 06:08:20 INFO (MainThread) [supervisor.host.info] Update local host information
20-10-03 06:08:21 INFO (MainThread) [supervisor.host.services] Update service information
20-10-03 06:08:21 INFO (MainThread) [supervisor.host.network] Update local network information
20-10-03 06:08:21 INFO (MainThread) [supervisor.host.sound] Update PulseAudio information
I think the restart at the beginning was from me - I did a few yesterday while battling with the InfluxDB connection attempt. I’ve left all the empty lines in - is this normal?
Nothing too much in the observer log:
2020/10/01 16:19:19 Start webserver on http://0.0.0.0:80
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2020/10/01 20:56:11 Start webserver on http://0.0.0.0:80
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2020/10/03 00:48:08 Start webserver on http://0.0.0.0:80
Continuing in a follow-up post as I can’t post any more in one go…