Unhealthy state

Petrica · December 9, 2020, 9:35am

Yeap, done that with both Windows and Ubuntu host machines. However, it is not an option to administer the host from a corporate laptop that doesn’t allow installation of any software or the use of remote desktop software, or from a tiny phone screen (thus the web based solution).

DavidFW1960 · December 9, 2020, 10:57am

Yes Watchtower won’t work. I use Docker-compose with a cronjob running every 12 hours.

jo-me · December 10, 2020, 8:05pm

I’m fine with having an “unsupported” OS.

But I’m not fine with imposing restrictions on the host system for features that are irrelevant for supervised installations anyway. Like “observer”, “dns”, “network”…

And not fine with forced spontaneous updates outside of the normal release cycle.

The supervisor should differentiate between running on the Home Assistant OS (or supported installations) and other installations and adapt its feature-set and checks accordingly. Supervised installs don’t have those strict security requirements or necessity to change host network stuff via UI.

I’m pretty sure that it is not really necessary to use exactly docker v19.x to start/update/stop containers. But v18 is unsupported and v20 as well… I do understand that the devs want a certain version to test against and not 5. Great, but you could still build it in a way that other versions (that only appear on rogue supervised installations) are not automatically blocked overnight or use the common feature set wherever possible. Same goes for network manager.

And about the “supported” supervised installation: Having a supported OS with restrictions on what to install alongside it defeats the purpose of a supervised installation and a docker-based application in general. I will certainly not switch to Debian only because of that. Those requirements might change again over night and tomorrow you find yourself being forced to use a different Debian version or some kind of custom HA Debian fork.

And what if other software behaved like that? Imagine if Nextcloud’s docker version only worken on Arch hosts, or Plex’s container only on Redhat hosts.

Too bad that the great improvements to HA core in the last releases are somewhat overshadowed by this stuff.

DavidFW1960 · December 10, 2020, 8:21pm

No one is forcing you to use supervised there are many installation methods you can use. If you want supervisor, these are the requirements.

nickrout · December 10, 2020, 8:34pm

If you want a supervisor and your own choice of linux to run on the same machine, do as has been suggested in this thread and elsewhere. Install proxmox+hassos+otherdistro+whateveryouwant.

LongDono · December 11, 2020, 6:24pm

Sigh. Disable the supervisor feature as alluded to by ludeeus (thanks):

/usr/share/hassio/homeassistant$ ha jobs options --ignore-conditions healthy

No one wants to told they have to change the OS they used for years or decades (especially those with more than just HA on their home server). Or to be told their system is unhealthy and HA rendered broken as a… protection. Yes, the decision was made. Did everyone agree with it? No. Doesn’t mean it needs to be mentioned constantly like we voted on it. Likewise, we can’t stick our heads in the sand either. If we all just did as told the first time (no more supervisor), supervised would already be gone. It’s okay to have a different opinion and to find workarounds for issues to support our unsupported systems. That is our option. It’s just not supported by the developers, hence why we are here in the community to find a fix. Use the fix, continue on, or move on. Sorry, had to be said.

Stay safe, everyone.

PS: You’ll still get the unhealthy state error message, but you’ll be able to upgrade, etc. You’ll also have this in your logs:

20-12-11 18:26:33 CRITICAL (MainThread) [supervisor.jobs] The following job conditions are ignored and will make the system unstable when they occur: {<JobCondition.HEALTHY: 'healthy'>}

PPS: I updated Ubuntu and restarted the OS just now and HA is no longer unhealthy. I spent quite a lot of time looking at other install methods and frustrating myself in the meantime. Sorry if I overreacted, covid times are hard on us all.

March 2020: I built a completely new server/desktop with ProxMox, 64gb ECC RAM, etc. It runs Ubuntu as a VM, Windows 10 as a VM, and of course, Home Assistant as an appliance. I have passed through the GPU so I can play games in Windows 10 with this system, and it replaced two separate tower PCs (and lowered power consumption). I saw a like on this post when I logged in and want to acknowledge that, as time goes on, it should be time to investigate your options and move on. I thought ProxMox was a horrible setup due to some misconceptions I had learned about it, probably from older or incorrect websites. Now I swear by ProxMox, especially on newer hardware.

dereitz · December 15, 2020, 2:55am

I was seeing this error as well and fixed it by replacing my /usr/sbin/hassio-supervisor with the new one from https://raw.githubusercontent.com/home-assistant/supervised-installer/master/files/hassio-supervisor. You’ll need to update the CONFIG_FILE parameter and then systemctl stop hassio-supervisor.service / systemctl start hassio-supervisor.service.

Once I had that error cleared, now I’m seeing the error about network-manager being too old since I’m running Ubuntu 18.04… Will have to figure that out next.

123 · December 15, 2020, 10:37pm

Two choices:

Install a newer version from an unofficial repository.
Upgrade to Ubuntu 20.04.

avd706 · December 15, 2020, 11:03pm

~$ apt policy network-manager
network-manager:
  Installed: 1.22.10-1ubuntu2.2
  Candidate: 1.22.10-1ubuntu2.2
  Version table:
 *** 1.22.10-1ubuntu2.2 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     1.22.10-1ubuntu1 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

123 · December 15, 2020, 11:08pm

I can post the same screenshot for you from my system. What was the intent there?

avd706 · December 15, 2020, 11:12pm

upgrade your ubuntu to 20.04

123 · December 15, 2020, 11:20pm

If you look at the link in my previous post you’ll see that I upgraded to 20.04 nine days ago.

dereitz · December 16, 2020, 3:41am

Yes, thanks so much for the info! I had gone with option #1 and after a few other issues, I was able to finally upgrade to 2020.12. Cheers!

Moonshine · December 16, 2020, 7:19pm

Was going to backup and try this route, as any OS upgrade or migration right now would trigger a cascade of events for my setup. Shouldn’t have clicked that update Supervisor button, but things have worked so well for sooo long.

I have two questions though…

I assumed that ha command was inside the HomeAssistant container, or potentially Supervisor container or on the host itself, but I don’t see it anywhere in my install. What generally provides that command?
ludeeus mentioned it could also be accomplished by “creating a file in the data dir for the Supervisor”. I’d actually prefer that route, but am not sure what “a file” refers to name/location wise or the syntax involved. Would someone have clues on that?

Thanks!

jo-me · December 16, 2020, 7:38pm

This worked on one machine but not on another.

I have a Pi3 running Volumio with HA supervised installed on top. Worked fine for years, now everything unhealthy.
Tried with creating jobs.json directly and via CLI, but the supervisor still reports unhealthy…

The jobs.json file was updated by the cli command and now looks like this:

[19:27:34] INFO: Starting local supervisor watchdog...,
20-12-16 19:27:38 INFO (MainThread) [__main__] Initializing Supervisor setup,
20-12-16 19:27:39 INFO (MainThread) [supervisor.bootstrap] Initializing Supervisor Sentry,
20-12-16 19:27:39 INFO (MainThread) [supervisor.bootstrap] Seting up coresys for machine: raspberrypi3,
20-12-16 19:27:39 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attaching to Supervisor homeassistant/armv7-hassio-supervisor with version 2020.12.7,
20-12-16 19:27:39 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.INITIALIZE,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.docker_configuration] Docker storage driver devicemapper is not supported!,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.docker_configuration] Docker logging driver json-file is not supported!,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] The configuration of Docker is not supported (more-info: https://www.home-assistant.io/more-info/unsupported/docker_configuration),
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] Docker version '18.06.3-ce' is not supported by the Supervisor! (more-info: https://www.home-assistant.io/more-info/unsupported/docker_version),
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] Found unsupported job conditions settings. (more-info: https://www.home-assistant.io/more-info/unsupported/job_conditions),
20-12-16 19:27:39 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete,
20-12-16 19:27:39 INFO (MainThread) [__main__] Setting up Supervisor,
20-12-16 19:27:39 INFO (MainThread) [supervisor.api] Starting API on 172.30.32.2,
20-12-16 19:27:40 ERROR (MainThread) [supervisor.dbus.network] Version '0.9.10.0' of NetworkManager is not supported!,
20-12-16 19:27:45 WARNING (MainThread) [supervisor.dbus.rauc] Host has no rauc support. OTA updates have been disabled.,
20-12-16 19:27:45 INFO (MainThread) [supervisor.host.info] Updating local host information,
20-12-16 19:27:45 INFO (MainThread) [supervisor.host.services] Updating service information,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host] Host information reload completed,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.apparmor] Loading AppArmor Profiles: {'hassio-supervisor'},
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.services] Reloading local service hassio-apparmor.service,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-dns with version 2020.11.0,
20-12-16 19:27:46 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-audio with version 17,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-cli with version 2020.11.1,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-observer with version 2020.10.1,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-multicast with version 3,
20-12-16 19:27:46 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json,
20-12-16 19:27:47 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 1 Home Assistant secrets,
20-12-16 19:27:47 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/raspberrypi3-homeassistant with version 0.118.4,
20-12-16 19:27:47 INFO (MainThread) [supervisor.hassos] No Home Assistant Operating System found,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/a0d7b954 repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/core repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/15ef4d2f repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/d4868bb1 repository,
20-12-16 19:27:54 INFO (MainThread) [supervisor.store] Loading add-ons from store: 70 all - 70 new - 0 remove,
20-12-16 19:27:54 INFO (MainThread) [supervisor.addons] Found 6 installed add-ons,
20-12-16 19:27:54 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-check_config with version 3.6.0,
20-12-16 19:27:54 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to hassioaddons/log-viewer-armv7 with version 0.9.1,
20-12-16 19:27:54 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to hassioaddons/ide-armv7 with version 2.2.0,
20-12-16 19:27:54 INFO (SyncWorker_4) [supervisor.docker.interface] Attaching to hassioaddons/pi-hole-armv7 with version v3.0.2,
20-12-16 19:27:54 INFO (SyncWorker_5) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-rpc_shutdown with version 2.2,
20-12-16 19:27:54 INFO (MainThread) [supervisor.snapshots] Found 3 snapshot files,
20-12-16 19:27:54 INFO (MainThread) [supervisor.discovery] Loaded 0 messages,
20-12-16 19:27:54 INFO (MainThread) [supervisor.ingress] Loaded 0 ingress sessions,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.SETUP,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.check] System checks complete,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP,
20-12-16 19:27:55 WARNING (MainThread) [supervisor.resolution.evaluations.base] NetworkManager is not correctly configured (more-info: https://www.home-assistant.io/more-info/unsupported/network_manager),
20-12-16 19:27:55 WARNING (MainThread) [supervisor.resolution.evaluations.base] Detected unsupported OS: Raspbian GNU/Linux 8 (jessie) (more-info: https://www.home-assistant.io/more-info/unsupported/os),
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete,
20-12-16 19:27:55 INFO (MainThread) [__main__] Running Supervisor,
20-12-16 19:27:55 WARNING (MainThread) [supervisor.core] System running in a unsupported environment!,
20-12-16 19:27:55 CRITICAL (MainThread) [supervisor.core] System running in a unhealthy state and need manual intervention!,
20-12-16 19:27:55 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.INITIALIZE' starting 0 add-ons,
20-12-16 19:27:55 INFO (MainThread) [supervisor.core] Supervisor reboot detected,
20-12-16 19:27:55 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled,
20-12-16 19:27:55 INFO (MainThread) [supervisor.misc.hwmon] Started Supervisor hardware monitor,
20-12-16 19:27:55 INFO (MainThread) [supervisor.core] Supervisor is up and running,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host.info] Updating local host information,
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING,
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host.services] Updating service information,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host] Host information reload completed,

The only error in the supervisor log is about network manager:

20-12-16 19:27:40 ERROR (MainThread) [supervisor.dbus.network] Version '0.9.10.0' of NetworkManager is not supported!,
...
...
20-12-16 19:27:55 CRITICAL (MainThread) [supervisor.core] System running in a unhealthy state and need manual intervention!,

Any ideas why the job condition is not working?

Moonshine · December 16, 2020, 8:14pm

I spend nearly an hour looking for answer and then find them 5 mins after post the questions… For anyone else who’s curious…

This is in the hassio_cli container, duh. For example:

docker exec hassio_cli ha host info

This would be creating a jobs.json file in Supervisor’s data directory with {"ignore_conditions": ["healthy"]}

Anyway, FWIW this did allow me to update HA. Obviously it would be better to meet all the guidelines, but sometimes life feels too short.

One question I still have is regarding running other “unsupported” docker containers. Like the following warning:

20-12-16 19:51:52 WARNING (MainThread) [supervisor.resolution.evaluations.base] Found images: {'watchtower'} which are not supported, remove these from the host! (more-info: https://www.home-assistant.io/more-info/unsupported/container)

I understand having another project like Watchtower doing updates would be BAD, but does anyone know what is actually triggering this warning? I’m using a different, custom Docker image of Watchtower that only targets 2 totally unrelated containers explicitly, and the container is not named watchtower. So I’m wondering what it’s keying off?

Thanks…

nickrout · December 16, 2020, 8:27pm

There is a link showing in @jo-me post. If you click on it it should show what is causing the error. (I don’t have the error, so I cannot click on it).

jo-me · December 16, 2020, 8:31pm

Sure, I guess it is complaining about Docker 18.x which is not supported, but why it is “Unhealthy” beause of this is not clear and neither why the job condition is not working as advertised.

The Supervisor will be marked as unhealthy if any of these requirements are not met:

jo-me · December 16, 2020, 8:33pm

Just look in the supervisor sources. It has list of image names it compares against:

nickrout · December 16, 2020, 8:41pm

Unhealthy because you have below the minimum supported version (19.03.0).

I am not sure what “job condition” you are referring to, nor what “advertised” refer so.