Unable to pull images from ghcr.io

I’m getting the following error in my supervisor logs when trying to pull images from the github container registry. It unfortunately seems to be intermittent, and does not affect any other systems on my network. This is the error that I get my logs.

22-08-08 12:52:11 ERROR (SyncWorker_3) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.8.2: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")

Because it’s not a 403, it doesn’t appear to be an authentication issue, but even adding my credentials for the registry does not resolve this issue. I’m guessing that there’s something cached that may have gotten corrupted, but I’ve already done a complete reinstallation of my home assistant OS installation. Does anyone have any ideas?

Error 500 is usually a server error, which means an error occured in the code that is executed on the server Itself and the request can therefore not be fulfilled.

Since the 500 is originating from http+docker://localhost/v1.41/auth, wouldn’t this more of a Docker issue than a GitHub issue?

That is a bit tricky to decode.
You need someone with more insight than me to answer that. ,:thinking:

If i remember, this error is due to dns not resolving on IP6 or something….(should use IP4)
A, found it again:

“It’s always DNS”

I’ve disabled IPv6, hoping that would do something, but it hasn’t fixed anything. AAAA records do resolve on my network, but my ISP doesn’t have native transport (that I know of), so I use a hurricane-electric tunnel (which doesn’t always work).

I’m curious what sort of cache or settings I can check on the HAOS host itself.

see if “ha resolution info” gives an error


It’s definitely having issues with DNS. I removed any firewall/NAT rules for DNS. However, I’m getting a new error now.
'AddonManager.install' blocked from execution, system is not healthy

Try settings → System → Repairs → Three dots in upper right corner → System Information → Scroll down to Home Assistant Supervisor

It should state unhealthy there and usually have a link to what the steps to repair is.

That leads me to this page, where the recommendation is to run ha supervisor update. However, if I try to run this in the shell, I get the same 500 error as before.

The host has unfettered access to DNS on the internet, and ghcr.io resolves both A and AAAA records on my network. DHCP is handing out the gateway address, which is just a unifi box pointed at NextDNS (and the only filter I’m using is osid, but there’s no blocks in the logs that seem relevant). I did try pointing the host directly at 1.1.1.1 (settings, network, v4, static), and it still has been giving the same error. If it’s DNS, I’m hitting a wall. It’s got to be at the host level. ¯\_(ツ)_/¯

Here’s the error in the core logs. Theres a bit of additional info.

Logger: homeassistant.components.websocket_api.http.connection
Source: components/hassio/update.py:256
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 6:42:24 PM (3 occurrences)
Last logged: 8:08:34 PM

[140573741188432] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
[140573758068848] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
[140573658932864] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 254, in async_install
    await async_update_supervisor(self.hass)
  File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 45, in _wrapper
    raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 199, in handle_call_service
    await hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 1713, in async_call
    task.result()
  File "/usr/src/homeassistant/homeassistant/core.py", line 1750, in _execute_service
    await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
  File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
    await service.entity_service_call(
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 676, in entity_service_call
    future.result()  # pop exception if have
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 930, in async_request_call
    await coro
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 713, in _handle_entity_call
    await result
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 152, in async_install
    await entity.async_install_with_progress(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 411, in async_install_with_progress
    await self.async_install(version, backup)
  File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 256, in async_install
    raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")

And the supervisor logs.

22-08-16 20:08:34 ERROR (SyncWorker_6) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
22-08-16 20:08:34 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")

I am at a loss here.
Only thing I can think of is a corrupted docker setup or maybe an outdated host system.

I’m thinking it’s a corrupted Docker setup as well. What bothers me is that it has apparently made its way into my backups. My main Home Assistant instance is a fresh install that I restored from a backup a little over a week ago.

This is also why I prefer YAML over UI configuration, and it really concerns me to see so many integrations moving over to a UI-only configuration flow. If I wanna take my back up out of the equation, I lose a shit load of configuration that was only possible through the UI.

Docker setups are not in yaml anyway.
It is setup at the creation of the container and you could manage it with add-ons like portainer earlier, but I think it was discovered that portainer trashed the network settings of the containers and was then removed as an add-on.

Sorry, I was mainly referring to HA configuration. If the issue has somehow ‘corrupted’ my backups, there’s a lot of HA configuration that I don’t have captured anywhere else but a JSON file somewhere in .storage, and it’s a little worrying. This is a fresh install that was restored from a backup. I know Docker is its own thing entirely. However, I try not to touch Docker or the underlying OS as much as possible, and let the supervisor handle everything. I remember when Portainer got pulled and it’s when I started writing my own add-ons instead.

I pulled the logs for docker from journalctl and there doesn’t seem to be any more information there that wasn’t already in the HA logs. Maybe I just need to start combing my backups for configurations that I need to keep and start fresh.

Make a new installation on a new media and test that you can pull the image without anything else install.
If you can do that then the error must be in the HA setup.

Well, I removed all of my container registry credentials from the supervisor, and it seems to be working now. I’ve never had an issue with my credentials. I’m curious if something changed in a recent version and I just missed it. Regardless, this seems to have corrected the issue. I will hold off on marking this as an answer for a few more days until I’m certain that it’s fixed.

Had the error have been a 403, I would have immediately known it was the credentials, but the 500 still makes me think it could have been something else.

1 Like

I am running into the same issue, how did you remove your credentials?
I’d like to give it a go too.

Alright so I found that you removed the registries from the Add-On store that you had added yourself (I have none).
I also found out that after retrying the install/update a few dozen times it ends up going through (this happens of most updates it seems).
Very odd.