Update to 0.117 just bricked my system

Installation
1

UI will not load, no errors in the core log, and a non-specific API error in the supervisor logs. Tried a roll back from the command line which instantly return saying it has been successful when clearly it has failed.

~ $ ha core update --version 0.116.4
Processing... Done.

Command completed successfully.
~ $ ha core info
arch: armv7
audio_input: null
audio_output: null
boot: true
image: homeassistant/raspberrypi3-homeassistant
ip_address: 172.30.32.1
last_version: 0.117.0
machine: raspberrypi3
port: 8123
ssl: true
update_available: false
version: 0.117.0
version_latest: 0.117.0
wait_boot: 600
watchdog: true
~ $

Totally stuck now, any suggestions ?

Core Logs

~ $ ha core logs
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[11:47:17] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2020-10-29 11:47:54 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for govee which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
2020-10-29 11:47:54 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for sonoff which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[14:41:10] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2020-10-29 14:41:48 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for sonoff which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
2020-10-29 14:41:48 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for govee which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[14:53:31] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
2020-10-29 14:54:05 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for govee which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
2020-10-29 14:54:05 WARNING (MainThread) [homeassistant.loader] You are using a custom integration for sonoff which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant.
~ $

Supervisor Logs

~ $ ha supervisor logs
20-10-29 14:47:39 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 14:48:37 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
20-10-29 14:48:55 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
[14:52:02] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[14:52:03] INFO: Starting local supervisor watchdog...
20-10-29 14:52:11 INFO (MainThread) [__main__] Initializing Supervisor setup
20-10-29 14:52:11 INFO (MainThread) [supervisor.bootstrap] Initializing Supervisor Sentry
20-10-29 14:52:11 INFO (SyncWorker_0) [supervisor.docker.supervisor] Attaching to Supervisor homeassistant/armv7-hassio-supervisor with version 2020.10.0
20-10-29 14:52:12 INFO (MainThread) [__main__] Setting up Supervisor
20-10-29 14:52:12 INFO (MainThread) [supervisor.api] Starting API on 172.30.32.2
20-10-29 14:52:13 INFO (MainThread) [supervisor.host.info] Updating local host information
20-10-29 14:52:13 INFO (MainThread) [supervisor.host.services] Updating service information
20-10-29 14:52:14 INFO (MainThread) [supervisor.host.network] Updating local network information
20-10-29 14:52:15 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
20-10-29 14:52:15 INFO (MainThread) [supervisor.host.apparmor] Loading AppArmor Profiles: {'hassio-supervisor'}
20-10-29 14:52:15 INFO (MainThread) [supervisor.host.services] Reloading local service hassos-apparmor.service
20-10-29 14:52:15 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-hassio-dns with version 2020.10.0
20-10-29 14:52:15 INFO (MainThread) [supervisor.plugins.dns] Starting CoreDNS plugin
20-10-29 14:52:15 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_dns application
20-10-29 14:52:18 INFO (SyncWorker_0) [supervisor.docker.dns] Starting DNS homeassistant/armv7-hassio-dns with version 2020.10.0 - 172.30.32.3
20-10-29 14:52:18 INFO (MainThread) [supervisor.plugins.dns] Updated /etc/resolv.conf
20-10-29 14:52:19 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-hassio-audio with version 17
20-10-29 14:52:19 INFO (MainThread) [supervisor.plugins.audio] Starting Audio plugin
20-10-29 14:52:19 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_audio application
20-10-29 14:52:31 INFO (SyncWorker_0) [supervisor.docker.audio] Starting Audio homeassistant/armv7-hassio-audio with version 17 - 172.30.32.4
20-10-29 14:52:31 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-hassio-cli with version 2020.10.0
20-10-29 14:52:31 INFO (MainThread) [supervisor.plugins.cli] Starting CLI plugin
20-10-29 14:52:31 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_cli application
20-10-29 14:52:47 INFO (SyncWorker_0) [supervisor.docker.cli] Starting CLI homeassistant/armv7-hassio-cli with version 2020.10.0 - 172.30.32.5
20-10-29 14:52:47 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-hassio-observer with version 2020.10.1
20-10-29 14:52:47 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-hassio-multicast with version 3
20-10-29 14:52:47 INFO (MainThread) [supervisor.plugins.multicast] Starting Multicast plugin
20-10-29 14:52:47 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning hassio_multicast application
20-10-29 14:52:49 INFO (SyncWorker_0) [supervisor.docker.multicast] Starting Multicast homeassistant/armv7-hassio-multicast with version 3 - Host
20-10-29 14:52:49 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
20-10-29 14:52:49 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/raspberrypi3-homeassistant with version 0.117.0
20-10-29 14:52:49 INFO (MainThread) [supervisor.homeassistant.secrets] Loaded 3 Home Assistant secrets
20-10-29 14:52:50 INFO (MainThread) [supervisor.hassos] Detect HassOS 4.15 / BootSlot A
20-10-29 14:52:52 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/a0d7b954 repository
20-10-29 14:52:52 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/44c60309 repository
20-10-29 14:52:52 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/core repository
20-10-29 14:52:52 INFO (MainThread) [supervisor.store.git] Loading add-on /data/addons/git/cebe7a76 repository
20-10-29 14:52:53 INFO (MainThread) [supervisor.store] Loading add-ons from store: 68 all - 68 new - 0 remove
20-10-29 14:52:53 INFO (MainThread) [supervisor.addons] Found 6 installed add-ons
20-10-29 14:52:53 INFO (SyncWorker_2) [supervisor.docker.interface] Attaching to hassioaddons/adguard-armv7 with version 2.5.1
20-10-29 14:52:53 INFO (SyncWorker_4) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-ssh with version 8.9.1
20-10-29 14:52:53 INFO (SyncWorker_0) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-duckdns with version 1.12.1
20-10-29 14:52:53 INFO (SyncWorker_1) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-check_config with version 3.6.0
20-10-29 14:52:53 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to homeassistant/armv7-addon-configurator with version 5.1.0
20-10-29 14:52:53 INFO (SyncWorker_5) [supervisor.docker.interface] Attaching to sabeechen/hassio-google-drive-backup-armv7 with version 3.12
20-10-29 14:52:53 INFO (MainThread) [supervisor.snapshots] Found 5 snapshot files
20-10-29 14:52:54 INFO (MainThread) [supervisor.discovery] Loaded 1 messages
20-10-29 14:52:54 INFO (MainThread) [supervisor.ingress] Loaded 0 ingress sessions
20-10-29 14:52:54 INFO (MainThread) [__main__] Running Supervisor
20-10-29 14:52:55 INFO (MainThread) [supervisor.hassos] Rauc: A - marked slot kernel.0 as good
20-10-29 14:52:55 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.INITIALIZE' starting 0 add-ons
20-10-29 14:52:55 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.SYSTEM' starting 0 add-ons
20-10-29 14:52:55 INFO (MainThread) [supervisor.addons] Phase 'AddonStartup.SERVICES' starting 3 add-ons
20-10-29 14:52:55 INFO (SyncWorker_5) [supervisor.docker.interface] Cleaning addon_core_ssh application
20-10-29 14:52:59 INFO (SyncWorker_5) [supervisor.docker.addon] Starting Docker add-on homeassistant/armv7-addon-ssh with version 8.9.1
20-10-29 14:52:59 INFO (SyncWorker_2) [supervisor.docker.interface] Cleaning addon_core_duckdns application
20-10-29 14:53:14 INFO (SyncWorker_2) [supervisor.docker.addon] Starting Docker add-on homeassistant/armv7-addon-duckdns with version 1.12.1
20-10-29 14:53:14 INFO (SyncWorker_0) [supervisor.docker.interface] Cleaning addon_a0d7b954_adguard application
20-10-29 14:53:24 INFO (SyncWorker_0) [supervisor.docker.addon] Starting Docker add-on hassioaddons/adguard-armv7 with version 2.5.1
20-10-29 14:53:29 INFO (SyncWorker_2) [supervisor.docker.interface] Starting homeassistant
20-10-29 14:53:29 INFO (MainThread) [supervisor.api.security] /host/info access from a0d7b954_adguard
20-10-29 14:53:30 INFO (MainThread) [supervisor.homeassistant.core] Wait until Home Assistant is ready
20-10-29 14:53:44 INFO (MainThread) [supervisor.api.security] /dns/info access from a0d7b954_adguard
20-10-29 14:54:13 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
20-10-29 14:55:41 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
20-10-29 14:55:56 INFO (MainThread) [supervisor.api.security] /core/logs access from core_ssh
20-10-29 14:56:02 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 14:56:28 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
20-10-29 14:57:09 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 14:57:46 INFO (MainThread) [supervisor.api.security] /core/update access from core_ssh
20-10-29 14:57:46 ERROR (MainThread) [supervisor.utils] Can't execute update while a task is in progress
20-10-29 14:58:16 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 14:59:23 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 14:59:49 INFO (MainThread) [supervisor.api.security] /core/update access from core_ssh
20-10-29 14:59:49 ERROR (MainThread) [supervisor.utils] Can't execute update while a task is in progress
20-10-29 14:59:55 INFO (MainThread) [supervisor.api.security] /core/update access from core_ssh
20-10-29 14:59:55 ERROR (MainThread) [supervisor.utils] Can't execute update while a task is in progress
20-10-29 15:00:14 INFO (MainThread) [supervisor.api.security] /core/info access from core_ssh
20-10-29 15:00:30 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 15:00:30 INFO (MainThread) [supervisor.api.security] /core/logs access from core_ssh
20-10-29 15:01:37 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 15:02:44 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 15:03:06 INFO (MainThread) [supervisor.api.security] /core/logs access from core_ssh
20-10-29 15:03:51 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config: 
20-10-29 15:04:18 INFO (MainThread) [supervisor.api.security] /supervisor/logs access from core_ssh
2

Same issue on rpi3 Home Assistant OS install.
Tried a few things then decided to go the nuclear option.

Starting from scratch and ran into the Bad Gateway issue when uploading a snapshot during onboarding.

Doh !

3

Also tried a snapshot restore…, it rebooted, then did NOTHING…, WTAF is going on ???

4

What are the contents of your configuration.yaml?

5

Exactly same brick in my case. I am still trying to solve it. Already tried:
-downgrading. Impossible because there is already some task execution and the update can’t be done. You will see that message in supervisor logs.
-repair supervisor. It finishes but no improvement.
-rebooting the whole system.
Same exact API errors as in the first post.

6

Nothing worked, so created a new Hassio install SD card, booted, which then automatically updates itself to 0.117 (probably not the best idea), then selected wipe and restore from Snapshot. Selected a full snapshot from a few days ago of around 44Mb, this has now been ‘restoring’ for 30 mins…, not hopefull that this is going to work, very poor feedback (i.e. NONE) when restoring…, how long am I supposed to leave this going before giving up and rebuilding from scratch ???

7

Left it restoring for a hour, nothing… what a cluster…

This is a Joke…, bring out a release that you cannot roll back or even restore from…, well done !

8

Crossing fingers to someone that can find a solution. I don’t want to start from scratch

9

There will be no fix, its properly broken…, no errors in the logs to track down so it would be impossible to find, never mind fix. I’m trying my second clean install as the system failed to restore a 0.116.4 snapshot file, so something is clearly badly incompatible between these releases. Going to try and get a generic 0.117 install running, then try and downgrade to 0.116.4 via the command line, then restore the snapshot, but I’m not hopeful…

10

Hi! I think I have solved it! I have just renamed configuration.yalm using IDE to configuration2.yalm. I have rebooted once so HA creates one new file. After a second reboot, I have access to the lovelace panel.
(you may need to backup secrets.yalm or others because this trick sims to format them).

11

Sucks that your having issues, tried to do some testing to see if I can find a issue.

Ok, I have spun up a couple of VMs to do some testing.

UI Restore from front end 0.117.0 to 0.117.0 works. (ie the second screen that appears after install)

Before creating a user rolling back from 0.117.0 to 0.116.0 via console worked, moving to 0.116.4 also worked.

Try a clean install, roll back to 0.116.X and restore your snap shot. You should be able to choose what to restrore. Start with the bare minimum. No add-ons, no custom components just the core HA.

Worst case, if you have a backup, you can un-package it and do a ‘manual’ restore dropping files back into the config folder. Its a .tar file which you will need to either use a linux machine or get the right utility on windows to un-package it.

12

Hi silvrr,

So what you describe is exactly what I’ve just done, Clean install of 0.117, manual command line rollback to 0.116.4, then a partial snapshot restore (everything apart from the OS) and we are back.

What a nightmare…

So the BUGS…

  1. Failed to update from 0.116.4 to 0.117, catastrophic failure, no errors in the logs (apart from the ‘Error on call https://172.30.32.1:8123/api/config:’ error with no info), no UI, nothing
  2. Failed to downgrade back to 0.116.4 from the command line (or any version)
  3. Failed to restore from full snapshot via the commandline
  4. Failed to restore from full snapshot with clean 0.117 install as part of the onboarding process

Not impressed…

13

Same issue here. Just upgraded from 0.116.4 to 0.117.0 on a Hassos VM.

Had to rename configuration.yaml to configuration2.yaml to get back into the UI. Currently restoring a snapshot which I hope works.

As you say, the total absence of any logs is going to make this difficult to diagnose.

14

Snapshot restore worked, fortunately.

Something I did notice, during the update to 0.117.0, the front end did start, I saw a message about a new front end being available and to reload, then the usual message about ha is starting up, then about 10 seconds later, Connection Lost…

15

I have the feeling that something inside the configuration file is breaking everything. I may try to find it copying the content block by block.

16

So you are up and running again on 0.116.4? Good!

Are you running any custom components (govee or sonoff perhaps)? Anything with a major breaking change in 0.117.0?

17

I’m running a few custom components. Neither govee nor sonoff. But I will try again tomorrow in a more controlled way; I plan to start by upgrading, replacing configuration.yaml and slowly adding everything back component-by-component.

18

Think I might know what this is…

config:

So, i don’t include the directive default_config: in my configuration.yaml because I don’t want to use the (usless) map: integration, because its not configurable to do anything useful, instead I create my own (configuarble) map card.

Because of this I have to ‘turn on’ every individual component/service HA uses (rather that have a directive which just turns one of them off… hello devs…???).

So, NOT listed in the breaking changes for 0.117 is the fact there is a new directive/component which is included by the default_config directive…, you guessed it… config:

The docs say this was introduced in version 0.39, but I haven’t seen it before . It would explain the ‘No UI’ bug…, just wondering if anyone else is like me, not using the default_config directive ?

Not brave enough to test my theory quite yet having spent the last 5 hours getting my system back up…

19

I’m not using default_config either.

I shall test your theory tomorrow by upgrading, creating the problem, adding config in then restarting and seeing what’s what.

20

Ah, I just checked my configuration.yaml and I already have config in there.