Unhealthy state

jo-me · December 16, 2020, 7:38pm

This worked on one machine but not on another.

I have a Pi3 running Volumio with HA supervised installed on top. Worked fine for years, now everything unhealthy.
Tried with creating jobs.json directly and via CLI, but the supervisor still reports unhealthy…

The jobs.json file was updated by the cli command and now looks like this:

[19:27:34] INFO: Starting local supervisor watchdog...,
20-12-16 19:27:38 INFO (MainThread) [__main__] Initializing Supervisor setup,
20-12-16 19:27:39 INFO (MainThread) [supervisor.bootstrap] Initializing Supervisor Sentry,
20-12-16 19:27:39 INFO (MainThread) [supervisor.bootstrap] Seting up coresys for machine: raspberrypi3,
20-12-16 19:27:39 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attaching to Supervisor homeassistant/armv7-hassio-supervisor with version 2020.12.7,
20-12-16 19:27:39 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.INITIALIZE,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.docker_configuration] Docker storage driver devicemapper is not supported!,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.docker_configuration] Docker logging driver json-file is not supported!,
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] The configuration of Docker is not supported (more-info: https://www.home-assistant.io/more-info/unsupported/docker_configuration),
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] Docker version '18.06.3-ce' is not supported by the Supervisor! (more-info: https://www.home-assistant.io/more-info/unsupported/docker_version),
20-12-16 19:27:39 WARNING (MainThread) [supervisor.resolution.evaluations.base] Found unsupported job conditions settings. (more-info: https://www.home-assistant.io/more-info/unsupported/job_conditions),
20-12-16 19:27:39 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete,
20-12-16 19:27:39 INFO (MainThread) [__main__] Setting up Supervisor,
20-12-16 19:27:39 INFO (MainThread) [supervisor.api] Starting API on 172.30.32.2,
20-12-16 19:27:40 ERROR (MainThread) [supervisor.dbus.network] Version '0.9.10.0' of NetworkManager is not supported!,
20-12-16 19:27:45 WARNING (MainThread) [supervisor.dbus.rauc] Host has no rauc support. OTA updates have been disabled.,
20-12-16 19:27:45 INFO (MainThread) [supervisor.host.info] Updating local host information,
20-12-16 19:27:45 INFO (MainThread) [supervisor.host.services] Updating service information,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host] Host information reload completed,
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.apparmor] Loading AppArmor Profiles: {'hassio-supervisor'},
20-12-16 19:27:46 INFO (MainThread) [supervisor.host.services] Reloading local service hassio-apparmor.service,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-dns with version 2020.11.0,
20-12-16 19:27:46 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-audio with version 17,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-cli with version 2020.11.1,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-observer with version 2020.10.1,
20-12-16 19:27:46 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armhf-hassio-multicast with version 3,
20-12-16 19:27:46 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json,
20-12-16 19:27:47 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 1 Home Assistant secrets,
20-12-16 19:27:47 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/raspberrypi3-homeassistant with version 0.118.4,
20-12-16 19:27:47 INFO (MainThread) [supervisor.hassos] No Home Assistant Operating System found,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/a0d7b954 repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/core repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/15ef4d2f repository,
20-12-16 19:27:48 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/d4868bb1 repository,
20-12-16 19:27:54 INFO (MainThread) [supervisor.store] Loading add-ons from store: 70 all - 70 new - 0 remove,
20-12-16 19:27:54 INFO (MainThread) [supervisor.addons] Found 6 installed add-ons,
20-12-16 19:27:54 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-check_config with version 3.6.0,
20-12-16 19:27:54 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to hassioaddons/log-viewer-armv7 with version 0.9.1,
20-12-16 19:27:54 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to hassioaddons/ide-armv7 with version 2.2.0,
20-12-16 19:27:54 INFO (SyncWorker_4) [supervisor.docker.interface] Attaching to hassioaddons/pi-hole-armv7 with version v3.0.2,
20-12-16 19:27:54 INFO (SyncWorker_5) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-rpc_shutdown with version 2.2,
20-12-16 19:27:54 INFO (MainThread) [supervisor.snapshots] Found 3 snapshot files,
20-12-16 19:27:54 INFO (MainThread) [supervisor.discovery] Loaded 0 messages,
20-12-16 19:27:54 INFO (MainThread) [supervisor.ingress] Loaded 0 ingress sessions,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state CoreState.SETUP,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.check] System checks complete,
20-12-16 19:27:54 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.SETUP,
20-12-16 19:27:55 WARNING (MainThread) [supervisor.resolution.evaluations.base] NetworkManager is not correctly configured (more-info: https://www.home-assistant.io/more-info/unsupported/network_manager),
20-12-16 19:27:55 WARNING (MainThread) [supervisor.resolution.evaluations.base] Detected unsupported OS: Raspbian GNU/Linux 8 (jessie) (more-info: https://www.home-assistant.io/more-info/unsupported/os),
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete,
20-12-16 19:27:55 INFO (MainThread) [__main__] Running Supervisor,
20-12-16 19:27:55 WARNING (MainThread) [supervisor.core] System running in a unsupported environment!,
20-12-16 19:27:55 CRITICAL (MainThread) [supervisor.core] System running in a unhealthy state and need manual intervention!,
20-12-16 19:27:55 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.INITIALIZE' starting 0 add-ons,
20-12-16 19:27:55 INFO (MainThread) [supervisor.core] Supervisor reboot detected,
20-12-16 19:27:55 INFO (MainThread) [supervisor.misc.tasks] All core tasks are scheduled,
20-12-16 19:27:55 INFO (MainThread) [supervisor.misc.hwmon] Started Supervisor hardware monitor,
20-12-16 19:27:55 INFO (MainThread) [supervisor.core] Supervisor is up and running,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host.info] Updating local host information,
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING,
20-12-16 19:27:55 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host.services] Updating service information,
20-12-16 19:27:55 INFO (MainThread) [supervisor.host] Host information reload completed,

The only error in the supervisor log is about network manager:

20-12-16 19:27:40 ERROR (MainThread) [supervisor.dbus.network] Version '0.9.10.0' of NetworkManager is not supported!,
...
...
20-12-16 19:27:55 CRITICAL (MainThread) [supervisor.core] System running in a unhealthy state and need manual intervention!,

Any ideas why the job condition is not working?

Moonshine · December 16, 2020, 8:14pm

I spend nearly an hour looking for answer and then find them 5 mins after post the questions… For anyone else who’s curious…

This is in the hassio_cli container, duh. For example:

docker exec hassio_cli ha host info

This would be creating a jobs.json file in Supervisor’s data directory with {"ignore_conditions": ["healthy"]}

Anyway, FWIW this did allow me to update HA. Obviously it would be better to meet all the guidelines, but sometimes life feels too short.

One question I still have is regarding running other “unsupported” docker containers. Like the following warning:

20-12-16 19:51:52 WARNING (MainThread) [supervisor.resolution.evaluations.base] Found images: {'watchtower'} which are not supported, remove these from the host! (more-info: https://www.home-assistant.io/more-info/unsupported/container)

I understand having another project like Watchtower doing updates would be BAD, but does anyone know what is actually triggering this warning? I’m using a different, custom Docker image of Watchtower that only targets 2 totally unrelated containers explicitly, and the container is not named watchtower. So I’m wondering what it’s keying off?

Thanks…

nickrout · December 16, 2020, 8:27pm

There is a link showing in @jo-me post. If you click on it it should show what is causing the error. (I don’t have the error, so I cannot click on it).

jo-me · December 16, 2020, 8:31pm

Sure, I guess it is complaining about Docker 18.x which is not supported, but why it is “Unhealthy” beause of this is not clear and neither why the job condition is not working as advertised.

The Supervisor will be marked as unhealthy if any of these requirements are not met:

jo-me · December 16, 2020, 8:33pm

Just look in the supervisor sources. It has list of image names it compares against:

nickrout · December 16, 2020, 8:41pm

Unhealthy because you have below the minimum supported version (19.03.0).

I am not sure what “job condition” you are referring to, nor what “advertised” refer so.

Moonshine · December 16, 2020, 8:44pm

Thanks! yes that seems a little broad, basically any image found with “watchtower” as part of its name would trigger. Well, I got rid of the image for now, but the container is still running. And things are “healthy” now.

I don’t believe the jobs.json override is meant to make your system show “healthy”, rather it’s to overlook the system being “unhealthy” as a condition for updating.

Iike you’re seeing, all of yours seem to be related to Docker. Those would have to be resolved in order for you to see “healthy” AFAIK.

DavidFW1960 · December 16, 2020, 10:27pm

I switched to using docker-compose with a cronjob to fix the watchtower issue even though my watchtower wasn’t touching any HA container either. Works just as well.

JD_01 · January 16, 2021, 12:36pm

I’m now getting this Unhealthy / Unsupported error. I’m not sure why. I’m using a Raspberry Pi 3, installed Hassio on it years back. and have never had a problem before.

I’m really not sure what to do or why my setup is now apparently not suitable. Is Pi 3 no longer supported?

Many of the fixes recommended in the documentation for the various errors seem to be to update my HA version… Which I cannot do, as it does not let me.

nickrout · January 16, 2021, 9:27pm

Does the error not contain a link to some documentation?

JD_01 · January 16, 2021, 9:47pm

There are but, it doesn’t make much sense. It links me to these:

DBUS - solution is to start the daemon (how?) or restart (didn’t work).
Docker Config - Solution is upgrade (can’t) or modify a json file (how? I don’t know how to get to this).
Privileged - solution is to upgrade
Network Manager - not even sure what I’m meant to do here
OS - Solution to … reinstall? But again I’m on a Pi 3 and was using Hassio. Not a custom install by any means.
Systemd - Reinstall?

So apparently there are all these issues in my setup that were never present before. Without wishing to sound naive, I did not interfere with the setup I had (at the OS level), I just “let” Hass do whatever it needed to do. I now need to edit what I assume are core system files, when I have never interacted with these before (and not even sure how to do that?).

Or… Is my environment corrupted and I’m meant to reinstall back using the same hardware? But then if I restore using a snapshot, won’t I be reintroducing these errors? So I need to start fresh?

Marconius · January 16, 2021, 10:09pm

Just wanted to echo @JD_01’s issue from above. I’ve had my RPi 3 setup with Hassio for years and upgrading up to now has been just touching up any potential breaking changes and clicking the upgrade button. It’s just worked… mostly like an appliance.

I’m cool with trying to figure things out, but the whole point of Hassio was to get me to stop tinkering with it. Would be great if there was a thing we could do to turn it back to the state of being an appliance again. Most of the threads on this seem to be targeted at HA running in other VMs or docker environments. Any ideas for us RPi3 guys with Hassio? Thanks!

JD_01 · January 16, 2021, 10:12pm

Thank God someone else is in the same situation, I felt like I was going crazy and was remembering things wrong. Like you, only changes I’ve had to make after upgrades previously is minor config changes in the integrations. Was pleased to never concern myself with the intricacies of running my own VM / Docker environment etc.

nickrout · January 16, 2021, 10:56pm

In another thread you just told us your system reports that you are running

Installation Type: Home Assistant Supervised

kanga_who · January 16, 2021, 11:04pm

It is most likely that the underlying OS you are using is outdated. If you are using a very old version of Home Assistant OS, it’s probably using ResinOS which is no longer supported. If you are running Supervised, then you are probably using Raspberry Pi OS (formerly Raspbian) which is also no longer supported.

Take a snapshot, backup all your config and .yaml files, do a fresh install using the most most current OS from the HA website on a new SD card, then restore your config.

The entire process should take you less than an hour on a Pi3 and you’ll maintain the current working SD card you have as a backup.

rpr · January 17, 2021, 2:44am

I am running Supervisor/HA in docker on Ubuntu 20.04. The last two times my system has come up as ‘unhealthy’ there was a new docker package available. Once I did an apt ugrade and docker restarted, and the Supervisor and HA containers restarted, the ‘unhealthy’ message is gone and I can upgrade HA.

Hope that helps someone, it took me awhile to figure that one out.

JD_01 · January 17, 2021, 11:30am

OK, that sounds reasonable. I’ll give it a try.

Thanks for the direction.

JD_01 · January 17, 2021, 11:40am

Indeed, that’s what it reports in the Info section. But that is not what I did, at least not intentionally. And I don’t see how I could have run the system for 3 years without knowing it - doesn’t Supervised require a significant amount of “tinkering” with the OS?

yetanotherzero · January 17, 2021, 8:17pm

Once I did an apt ugrade and docker restarted, and the Supervisor and HA containers restarted, the ‘unhealthy’ message is gone and I can upgrade HA.

I can confirm as well this worked for me

JD_01 · January 19, 2021, 4:30pm

Hi, this worked by the way. Bit of a faff as restoring from a snapshot wasn’t working - so had to go and create the new instance, enable SAMBA, copy the snapshot across etc.

But overall it worked and I am back in a healthy state, able to upgrade.

Thanks for your help.