Supervisor fails to update Core 2022.4.1 to Core 2022.4.5 due to "Internal Server Error" and "read: connection refused"

123 · April 21, 2022, 3:30pm

This problem has been reported in the Core and Supervisor repos but still awaiting an official response. Perhaps someone in the community has some idea what’s the cause (and the remedy).

Here’s the Supervisor log after attempting to update from 2022.4.1 to 2022.4.5 (same result with 2022.4.6).

22-04-20 12:02:27 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2022.4.5
22-04-20 12:02:27 INFO (SyncWorker_3) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi3-homeassistant:2022.4.1 to ghcr.io/home-assistant/raspberrypi3-homeassistant:2022.4.5
22-04-20 12:02:27 INFO (SyncWorker_3) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi3-homeassistant with tag 2022.4.5.
22-04-20 12:02:27 ERROR (SyncWorker_3) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/raspberrypi3-homeassistant:2022.4.5: 500 Server Error for http+docker://localhost/v1.41/images/create?tag=2022.4.5&fromImage=ghcr.io%2Fhome-assistant%2Fraspberrypi3-homeassistant&platform=linux%2Farm64: Internal Server Error ("Get "https://ghcr.io/v2/": dial tcp: lookup ghcr.io on [::1]:53: read udp [::1]:43736->[::1]:53: read: connection refused")
22-04-20 12:14:34 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

I may be interpreting the error message incorrectly but it seems to suggest a DNS problem while performing a lookup.

Running on RPi3 with Home Assistant Supervised on Debian. Had no problem upgrading to 2022.4.1. DNS is local and hasn’t changed since the upgrade to 2022.4.1.

me@ha:~$ ha dns info
host: 172.30.32.3                                                                              
locals: []                                                                                     
servers:                                                                                   
    - dns://192.168.1.2                                                                            
update_available: false                                                                        
version: 2021.06.0                                                                             
version_latest: 2021.06.0

123 · April 22, 2022, 9:34pm

Thinking out loud:

When something goes wrong with Supervisor, it becomes evident that this part of the Home Assistant project is understaffed; response time for reported Issues is days, weeks, months, or never (see current backlog of unanswered/unresolved Supervisor Issues).

At the moment, my system is functional, healthy, and supported but updates fail so it’s stuck at 2022.4.1. Although the problem was reported six days ago (by someone else) in the Core repo and two days ago (by me) in the Supervisor repo, there’s been no reply let alone a solution (nor in the community).

It’s times like this when I wonder if I should migrate to Home Assistant Container and dispense entirely with the under-supported Supervisor component. If this bug remains unresolved well into the May release of Home Assistant, I may have to give that migration serious consideration.

Tamsy · April 23, 2022, 3:24am

Have a look here.

It seems like adding:

140.82.121.34 ghcr.io
185.199.108.154 pkg-containers.githubusercontent.com

to /etc/hosts works as a workaround.

Addendum: @aceindy added also:

34.203.135.183  registry-1.docker.io

and it seems to work.

123 · April 23, 2022, 2:48pm

Thank you for confirming it’s a DNS issue. It helped me focus my investigation.

I confirmed the DNS issue is limited to the two machines hosting Home Assistant Supervised on Debian 11. All other computers on the LAN (Ubuntu, Windows, Android, iOS, etc) have no problem with DNS (so it’s not a LAN-wide issue).
The two affected machines were not even able to resolve deb.debian.org so even a basic apt update was impossible.
I ran sudo nmtui and discovered the wired connection’s network profile no longer had an entry in the DNS servers field.
I set it to my local DNS address, restarted the machine, and that fixed the DNS problem (able to resolve deb.debian.org again).
I successfully upgraded one machine from 2022.4.1 to 2022.4.6.

NOTE

The default wired connection profile’s name on Debian is usually something like “Wired connection 1”. However, here the sole connection profile’s name is “Supervisor eth0” suggesting this is the handiwork of Supervisor. Notice the DNS servers field is empty. It’s definitely not something I ever erased and explains why the machine’s DNS lookups failed.

Tamsy · April 24, 2022, 1:38am

I have cross-checked your settings with the settings on my side:

DNS servers field is indeed empty but this is expected since I have configured DNS through /etc/resolv.conf like:

domain mylandomain.com
search mylandomain.com.
nameserver 192.168.xxx.xxx

where nameserver 192.168.xxx.xxx points to my local DNS server inside the LAN which does all DNS resolving.

Furthermore I have set IPv4 configuration to "Automatic"

auto eth0
iface eth0 inet dhcp

insdide /etc/network/interfaces.d/eth0

because I usually assign static IP-addresses to all LAN devices through the router only (which obviously also works as a dhcp server) thus there is little need to set static ip-address with the LAN devices themselves anymore.

As you can see there is also no need to add anything to /etc/hosts:

127.0.0.1       localhost
::1             localhost ip6-localhost ip6-loopback
ff02::1         ip6-allnodes
ff02::2         ip6-allrouters

123 · April 24, 2022, 3:30am

In my case, /etc/resolv.conf is basically empty except for a single comment:

# Generated by NetworkManager

and /etc/hosts contains the usual IPv4 related stuff plus the localhost address with the machine’s hostname.

It all seems fairly normal given that I am using NetworkManager to manually assign a static IPv4 address and hostname (and IPv6 is disabled). What was abnormal was the absence of a manually assigned DNS server address; something was responsible for its removal (and I know I didn’t do it). I suspect a recent upgrade of Supervisor may have mishandled the situation where IPv4 networking is manually configured but I can’t prove it.

Tamsy · April 24, 2022, 5:31am

Using NetworkManager here too for the RPI running HA.

Exactly the reason why I prefer to use the routers “Address Reservation” contrary to assigning static IP addresses to each and every device on the LAN is the possibility to manage all IP addresses within the LAN using one central point instead of taking the hassle of having to (re)configure each and every device if needed. I regard it as simply more comfy this way but other user’s preferences may vary.

However, it is a different story when working with full blown servers directly facing the internet.

For readers having NetworkManager intalled who want to dig deeper into the matter, run:

sudo nmcli connection show --active

to check the configured network connections of your HA server.

The output of the above command looks like i.e.:

NAME             UUID                                  TYPE      DEVICE 
Supervisor eth0  08a46ff4-59d5-6310-aad1-6feccb2a2468  ethernet  eth0

Where “Supervisor eth0” is the active profile name.

To check on the assigned DNS-Server of the active connection simply run:

sudo nmcli connection show "Supervisor eth0" | grep DNS

The output looks like i.e.:

IP4.DNS[1]:                             192.168.178.xx

where 192.168.178.xx is the active DNS Server IP address in my case.

Running:

sudo nmcli connection show "Supervisor eth0"

shows ALL parameters of the active profile.

Now if you are experiencing trouble with a missing DNS server like the OP did, instead of adding IP addresses/domain names to /etc/hosts (see RVital’s workaround, which does not really solve the issue (IP addresses can change at any time) you can run:

sudo nmcli connection modify "Supervisor eth0" ipv4.dns "local-dns-servers-ip-address"

Lets assume you want to use Google’s DNS servers instead of your local DNS server for name resolution:

For IPv4:
sudo nmcli connection modify "Supervisor eth0" ipv4.dns "8.8.8.8 8.8.4.4"

For IPv6:
sudo nmcli connection modify "Supervisor eth0" ipv6.dns "2001:4860:4860::8888 2001:4860:4860::8844"

Don’t forget to activate the new settings:

sudo nmcli connection up "Supervisor eth0"

The output shoud look similar to:

Connection successfully activated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/5)

With the above the problem of missing DNS server address entries as Taras was experiencing it with some of his installations should be solved.

(Thanks @123 for pointing out the problem )