HA update failed, container not starting anymore

Configuration
1

Hey,

I’ve done an HassOS update through the webinterface last Friday on my RPi4 and since then my installation isn’t working anymore. The OS starts
and it gets an IP but I can’t login to the machine over SSH or over the webinterface.

So I hook up a monitor to the RPi4 to check. It seems that one of the containers wasn’t updated correctly so
the supervisor wouldn’t start, leaving me in the emergency console eventually.

I’ve tried to manually update the supervisor container by pulling from the Docker hub but didn’t work either.

What I now have in the logs is this:

– Journal begins at Fri 2021-08-06 14:43:07 UTC. –
Nov 07 16:32:10 homeassistant dockerd[482]: time=“2021-11-07T16:32:10.093745427Z” level=error msg=“error unmounting /mnt/data/docker/overlay2/80a2d56b773ad10ee7f172d9d6aac98856dd12624dde6f1a96b50111686ab3cc-init/merged: invalid argument” storage-driver=overlay2
Nov 07 16:32:10 homeassistant dockerd[482]: time=“2021-11-07T16:32:10.094878129Z” level=error msg=“Handler for POST /v1.41/containers/create returned error: error creating overlay mount to /mnt/data/docker/overlay2/80a2d56b773ad10ee7f172d9d6aac98856dd12624dde6f1a96b50111686ab3cc-init/merged: no such file or directory”
Nov 07 16:32:10 homeassistant hassos-supervisor[1833]: Error response from daemon: error creating overlay mount to /mnt/data/docker/overlay2/80a2d56b773ad10ee7f172d9d6aac98856dd12624dde6f1a96b50111686ab3cc-init/merged: no such file or directory
Nov 07 16:32:10 homeassistant kernel: overlayfs: failed to resolve ‘/mnt/data/docker/overlay2/l/XVJUUQB55DKV47WMMUXQ3OX5X2’: -2
Nov 07 16:32:10 homeassistant systemd[1]: hassos-supervisor.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 16:32:10 homeassistant systemd[1]: hassos-supervisor.service: Failed with result ‘exit-code’.
Nov 07 16:32:15 homeassistant systemd[1]: hassos-supervisor.service: Scheduled restart job, restart counter is at 7.
Nov 07 16:32:15 homeassistant systemd[1]: Stopped HassOS supervisor.
Nov 07 16:32:15 homeassistant systemd[1]: Starting HassOS supervisor…
Nov 07 16:32:17 homeassistant docker[1847]: Error response from daemon: No such container: hassio_supervisor
Nov 07 16:32:15 homeassistant systemd[1]: Started HassOS supervisor.
Nov 07 16:32:17 homeassistant hassos-supervisor[1864]: Error: No such object: hassio_supervisor
Nov 07 16:32:17 homeassistant hassos-supervisor[1854]: [WARNING] Supervisor container did not remove the startup marker file. Assuming container image or container corruption.
Nov 07 16:32:17 homeassistant hassos-supervisor[1879]: Error: No such container: hassio_supervisor
Nov 07 16:32:17 homeassistant hassos-supervisor[1896]: Error: No such image: sha256:ce2c6e24c65d179f23410bff7ef3743b18c5ef0206f6b66896c7b67b8b885a44
Nov 07 16:32:17 homeassistant hassos-supervisor[1896]: sha256:4a2518e745b825ffe9c2a281895379b3a3b41dc7b69751429d738ee961b3b710
Nov 07 16:32:17 homeassistant hassos-supervisor[1854]: [WARNING] Supervisor image missing, downloading a fresh one: 2021.10.8
Nov 07 16:32:23 homeassistant hassos-supervisor[1909]: 2021.10.8: Pulling from homeassistant/aarch64-hassio-supervisor
Nov 07 16:32:23 homeassistant hassos-supervisor[1909]: Digest: sha256:a97e123edd9edaed02bc0fe024aed810c971632e8bb58158e077a2bea7fb2959
Nov 07 16:32:23 homeassistant hassos-supervisor[1909]: Status: Image is up to date for homeassistant/aarch64-hassio-supervisor:2021.10.8
Nov 07 16:32:23 homeassistant hassos-supervisor[1909]: docker.io/homeassistant/aarch64-hassio-supervisor:2021.10.8
Nov 07 16:32:24 homeassistant dockerd[482]: time=“2021-11-07T16:32:24.611406647Z” level=error msg=“Handler for GET /v1.41/images/homeassistant/aarch64-hassio-supervisor/json returned error: readlink /mnt/data/docker/overlay2/l/XVJUUQB55DKV47WMMUXQ3OX5X2: no such file or directory”
Nov 07 16:32:24 homeassistant hassos-supervisor[1933]: Error response from daemon: readlink /mnt/data/docker/overlay2/l/XVJUUQB55DKV47WMMUXQ3OX5X2: no such file or directory
Nov 07 16:32:24 homeassistant hassos-supervisor[1854]: [INFO] Creating a new Supervisor container…
Nov 07 16:32:24 homeassistant dockerd[482]: time=“2021-11-07T16:32:24.706727093Z” level=error msg=“error unmounting /mnt/data/docker/overlay2/18a188ac5e8a366d7cb678db51e023e8a275b15be1a01a230f5cff2fc807f524-init/merged: invalid argument” storage-driver=overlay2
Nov 07 16:32:24 homeassistant dockerd[482]: time=“2021-11-07T16:32:24.707932008Z” level=error msg=“Handler for POST /v1.41/containers/create returned error: error creating overlay mount to /mnt/data/docker/overlay2/18a188ac5e8a366d7cb678db51e023e8a275b15be1a01a230f5cff2fc807f524-init/merged: no such file or directory”
Nov 07 16:32:24 homeassistant kernel: overlayfs: failed to resolve ‘/mnt/data/docker/overlay2/l/XVJUUQB55DKV47WMMUXQ3OX5X2’: -2
Nov 07 16:32:24 homeassistant hassos-supervisor[1941]: Error response from daemon: error creating overlay mount to /mnt/data/docker/overlay2/18a188ac5e8a366d7cb678db51e023e8a275b15be1a01a230f5cff2fc807f524-init/merged: no such file or directory
Nov 07 16:32:24 homeassistant systemd[1]: hassos-supervisor.service: Main process exited, code=exited, status=1/FAILURE
Nov 07 16:32:24 homeassistant systemd[1]: hassos-supervisor.service: Failed with result ‘exit-code’.
Nov 07 16:32:29 homeassistant systemd[1]: hassos-supervisor.service: Scheduled restart job, restart counter is at 8.
Nov 07 16:32:29 homeassistant systemd[1]: Stopped HassOS supervisor.
Nov 07 16:32:29 homeassistant systemd[1]: Starting HassOS supervisor…
Nov 07 16:32:29 homeassistant systemd[1]: Started HassOS supervisor.
Nov 07 16:32:30 homeassistant docker[1949]: Error response from daemon: No such container: hassio_supervisor

Every attempt to run the command line client just tells me that hassio_cli container isn’t running.

$ ha help
Error: no such container hassio_cli

What’s the prefered way to recover from this?

I’ve found a few (older) comments in the forum suggesting to download and rerun the installation scripts but those don’t seem to work
anymore as also the installation procedure has changed.

Any help would be much appreciated.

Cheers
Andre

2

Hey, just bringing this up again. Is there really nobody who can assists?

Thanks
Andre

3

Looks like your docker install (/mnt/data/docker/overlay2) was corrupted or cannot be mounted.
No magic trick here.

4

So only chance is fresh reinstall you reckon?

5

Or dig into the issue with your “data” mount point…

6

Hi Andre,

Did you finally find a way to fixing this? I think I might be in the same spot, but I don´t want a fresh install, many things would need to be done again because I can’t access Samba share.

Thanks in advance.

7

Just in case anyone runs into this issue, found a way to recover backup files following the indications in this thread:
https://community.home-assistant.io/t/home-assistant-on-sd-card-wont-start-but-sd-card-is-still-readable/244250/2?u=andrax92

That way you can recover all the files that were in home assistant before it stopped working, including backup files.

8

Sorry for not responding earlier. It turned out to be a faulty SD card in my case. Luckily I did backups of the config that I also managed to copy to a USB drive before trying to “repair” the Docker mount. This didn’t work in the end either (because of the faulty SD card) so I did a fresh install on a new card and then imported the full backup. This worked fine and after 30mins I had a working installation with the full config.