I noticed the same behavior on my test system. I thought it was because I installed the 2021.3 beta, but it seems it has to to with the supervisor version.
This PITA with Docker’s updates and Supervisor’s (strictly automatic) updates has to end. EVERY time the same thing.
I updated yesterday to latest Docker-ce, had the exact same behavior (supervisor starts only manually) and I almost ruined all my installation troubleshooting. This has to be red-flagged in the HA alarms immediately.
Reboot also takes almost 3-4 minutes, whereas it only took 30-45 seconds. Systemmd cannot stop the supervisor service effectively and it almost hangs.
I’ve somehow narrowed the problem to the following:
During boot (I’m not completely sure):
hassio-apparmor[343]: Warning: unable to find a suitable fs in /proc/mounts, is it mounted?
hassio-apparmor[343]: Use --subdomainfs to override.
hassio-apparmor[343]: [Error]: Can't load profile /usr/share/hassio/apparmor/hassio-supervisor
During shutdown/reboot (I’m positive-look at the timeline):
**Feb 27 19:40:50** systemd[1]: hassio-supervisor.service: Main process exited, code=killed, status=15/TERM
**Feb 27 19:42:20** systemd[1]: hassio-supervisor.service: State 'stop-final-sigterm' timed out. Killing.
Feb 27 19:42:20 systemd[1]: hassio-supervisor.service: Killing process 1450 (docker) with signal SIGKILL.
Feb 27 19:42:20 systemd[1]: hassio-supervisor.service: Failed with result 'timeout'.
Is it confirmed that reverting to the previous docker-ce version is a temporary solution? If so, can someone confirm the cmd in the terminal?
I’ve also tried with the beta supervisor (2021.02.12). It shows the exact same behaviour for me (not starting automatically upon restart, very slow shutdown).
OP’s link in installing Debian points to this dynamic URL here which is now showing 10.8. Since the guide insists we download 10.7, it should probably be updated. Note that I couldn’t find where to see older versions on that site, but I did find them here on a mirror site.
Docker 20.10.4
Debian 10.8
VMware machine on my PC
Installing succsesfuly with manual.
# curl -Lo installer.sh https://raw.githubusercontent.com/home-assistant/supervised-installer/master/installer.sh
# bash installer.sh
But after reboot container hassio_supervisor is Exited.
If I try
sudo docker start hassio_supervisor
thats works OK.
How can I fix autostart hassio_supervisor container after reboot?
The guide insists you use Debian 10, Debian 10.7 was the available version at the time of last update to the guide.
Use 10.8, or the available version at the time.
If also you experience a huge amount of errors logged by hassio_audio while it attempts to start, then try this solution .
Quickest way to confirm you have this problem is to see if hassio_audio returns a valid date:
# the date in hassio_audio is not set:
docker exec -it hassio_audio date
Sun - 40447 20480:16:-1227370880 -1225953501
# error in hassio_audio logs:
Assertion 'clock_gettime(CLOCK_REALTIME, &ts) == 0' failed
In addition, hassio_audio may cause audio to fail on the host device, and it can also cause an increase in CPU usage, up to 16% on my RPi 4.
Because it is not supported to run Home Assistant without hassio_audio, even when we don’t use it at all, one solution to work around these problems is to suspend PulseAudio running in the hassio_audio container when it is idle, and this can be done by loading the PulseAudio “module-suspend-on-idle” module. The shell script described below runs as service and will automatically load the module-suspend-on-idle module when the hassio_audio container is (re)started.
1 Like
How often are all you guys rebooting the host to even see this as a problem? If you reboot the host all the time I would have to ask why?
Got it, my mistake. Thanks for the heads up.
1 Like
When I updated to latest docker (which I’m not doing lightheartedly again), home assistant container stopped and was not restarted by the watchdog as usual. So I tried with a reboot of the host, at which point I found out that supervisor no longer starts upon boot. Since I also had not restarted the host for a long time, I wasn’t sure if this update or a recent update of the supervisor (usual suspects) or something else was the culprit.I almost destroyed my installation then troubleshooting. Thank God I have portainer, which helps a lot in these situations.
Anyway, back to docker 20.10.3 now and everything works. Lesson to self, if something works do not update. Especially docker (and supervisor, but this is not an option-I can’t understand why).
Only 1 previous docker I remember being a problem… but you do need to keep debian up-to-date and docker will auto-update as part of that.
When I get a docker update I manually restart the supervisor and yesterday I guess I did notice the supervisor and homeassistant container didn’t start but I used portainer (not the addon… yet another reason to use the real portainer to troubleshoot stuff like this). So I did manually start HA and supervisor containers and then in HA I have a script that restarts all my addons so I never rebooted the host anyway… I didn’t think twice about the ha and supervisor not starting when I restarted the service… just thought I was impatient and restarted them manually.
If you are running a supervised install you should have the skills to work this out without too many dramas IMO.
My gut feel is that this is a supervisor issue more than a docker issue…
1 Like
Make that two failed updates of docker in a couple of months then. Supervisor is also my first gut feeling in these situations, but I tried with two versions and only when I downgraded docker everything worked.
I also run portainer independently from ha.
IMHO restarts are occasionally needed for general stability but in this case to also emulate loss of power/wifi for our installation. Imagine you were away (without having prepared for that) and there were a power loss, you’d think everything worked and ha would never have started, since supervisor would be essentially dead.
So,no, no dramas, just stating the facts with two very recent docker updates and various supervisor updates in the past.
This time is different to the previous time though. Last time, because I never restart the host I didn’t need to roll back docker either FWIW. It seems to me the supervisor service isn’t restarting or starting and even if it started, it’s not running the containers but you can manually start them and all is good. So I suspect it will be a supervisor update to fix it but we will see.
I think supervisor is not starting at all this time. Folks are beginning to notice in other threads too, how can we add this to the HA alarms for people to know?
If you ever decide to restart, check your journalctl logs, mine were flooded with docker and various errors, which went away when I downgraded docker. This happens with both latest stable and beta supervisor.
Already reported as a issue ( hassio-supervisor fails to start and attach to an not running hassio_supervisor · Issue #2642 · home-assistant/supervisor (github.com)) and it seems to be pointing to a upstream issue on docker, other have reported issues with latest version in other projects
docker 20.10.4 on linux and windows breaks docker-compose run -T · Issue #42093 · moby/moby (github.com)
2 Likes
totaly flabberghasted right now.
a few weeks ago i installed a debian 10 server , and also installed hassio-supervised.
since a reboot and some network card issues my home assistant doesnt work anymore.
network issues have been fixed for the native site on the server, but i’m not familair with docker containers, so troubleshooting this is a pain in the …
who can help me out here. because the things i searched for, dont bring up hassio.
Downgrading docker did the trick for me. Remember to unhold the the packages after the issues has been fixed in HA. Otherwise docker won’t be upgraded in the future.
sudo apt remove docker-ce docker-ce-rootless-extras docker-ce-cli
sudo apt autoremove --purge -y
sudo reboot
sudo apt install docker-ce=5:20.10.3~3-0~debian-buster docker-ce-rootless-extras=5:20.10.3~3-0~debian-buster docker-ce-cli=5:20.10.3~3-0~debian-buster
sudo apt-mark hold docker-ce docker-ce-rootless-extras docker-ce-cli
sudo reboot
4 Likes
Seems the docker issue has identified it’s only the docker-cli that is a problem.
I was unable to update supervisor this morning and then supervisor wont auto restart and when it then starts says my system is in an unhealthy state so I’m going to try reverting cli only.
Reverting docker-cli and supervisor immediately self updated, unhealthy state gone again and HA updates. Transcript of session:
david@debian:~$ sudo apt install docker-ce-cli=5:20.10.3~3-0~debian-buster
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be DOWNGRADED:
docker-ce-cli
0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 0 not upgraded.
Need to get 41.4 MB of archives.
After this operation, 3,072 B disk space will be freed.
Do you want to continue? [Y/n] Y
Get:1 https://download.docker.com/linux/debian buster/stable amd64 docker-ce-cli amd64 5:20.10.3~3-0~debian-buster [41.4 MB]
Fetched 41.4 MB in 12s (3,523 kB/s)
dpkg: warning: downgrading docker-ce-cli from 5:20.10.4~3-0~debian-buster to 5:20.10.3~3-0~debian-buster
(Reading database ... 91925 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a20.10.3~3-0~debian-buster_amd64.deb ...
Unpacking docker-ce-cli (5:20.10.3~3-0~debian-buster) over (5:20.10.4~3-0~debian-buster) ...
Setting up docker-ce-cli (5:20.10.3~3-0~debian-buster) ...
Processing triggers for man-db (2.8.5-2) ...
Scanning processes...
Scanning candidates...
Scanning linux images...
Running kernel seems to be up-to-date.
Failed to check for processor microcode upgrades.
Restarting services...
systemctl restart hassio-supervisor.service
No containers need to be restarted.
User sessions running outdated binaries:
david @ user manager service: systemd[22601]
david@debian:~$ sudo apt-mark hold docker-ce-cli
docker-ce-cli set on hold.
That worked for me, too. I just ran sudo apt install docker-ce-cli=5:20.10.3~3-0~debian-buster and then restarted for good measure, and it’s all back to normal.
I’m still on docker 20.10.3.
Does the new supervisor fix this?
I’m on the latest I can see 2021.02.11