I’m getting the following error in my supervisor logs when trying to pull images from the github container registry. It unfortunately seems to be intermittent, and does not affect any other systems on my network. This is the error that I get my logs.
22-08-08 12:52:11 ERROR (SyncWorker_3) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/generic-x86-64-homeassistant:2022.8.2: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
Because it’s not a 403, it doesn’t appear to be an authentication issue, but even adding my credentials for the registry does not resolve this issue. I’m guessing that there’s something cached that may have gotten corrupted, but I’ve already done a complete reinstallation of my home assistant OS installation. Does anyone have any ideas?
Error 500 is usually a server error, which means an error occured in the code that is executed on the server Itself and the request can therefore not be fulfilled.
I’ve disabled IPv6, hoping that would do something, but it hasn’t fixed anything. AAAA records do resolve on my network, but my ISP doesn’t have native transport (that I know of), so I use a hurricane-electric tunnel (which doesn’t always work).
I’m curious what sort of cache or settings I can check on the HAOS host itself.
It’s definitely having issues with DNS. I removed any firewall/NAT rules for DNS. However, I’m getting a new error now. 'AddonManager.install' blocked from execution, system is not healthy
That leads me to this page, where the recommendation is to run ha supervisor update. However, if I try to run this in the shell, I get the same 500 error as before.
The host has unfettered access to DNS on the internet, and ghcr.io resolves both A and AAAA records on my network. DHCP is handing out the gateway address, which is just a unifi box pointed at NextDNS (and the only filter I’m using is osid, but there’s no blocks in the logs that seem relevant). I did try pointing the host directly at 1.1.1.1 (settings, network, v4, static), and it still has been giving the same error. If it’s DNS, I’m hitting a wall. It’s got to be at the host level. ¯\_(ツ)_/¯
Here’s the error in the core logs. Theres a bit of additional info.
Logger: homeassistant.components.websocket_api.http.connection
Source: components/hassio/update.py:256
Integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 6:42:24 PM (3 occurrences)
Last logged: 8:08:34 PM
[140573741188432] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
[140573758068848] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
[140573658932864] Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 254, in async_install
await async_update_supervisor(self.hass)
File "/usr/src/homeassistant/homeassistant/components/hassio/handler.py", line 45, in _wrapper
raise HassioAPIError(data["message"])
homeassistant.components.hassio.handler.HassioAPIError: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 199, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1713, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1750, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
await service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 676, in entity_service_call
future.result() # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 930, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 713, in _handle_entity_call
await result
File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 152, in async_install
await entity.async_install_with_progress(version, backup)
File "/usr/src/homeassistant/homeassistant/components/update/__init__.py", line 411, in async_install_with_progress
await self.async_install(version, backup)
File "/usr/src/homeassistant/homeassistant/components/hassio/update.py", line 256, in async_install
raise HomeAssistantError(
homeassistant.exceptions.HomeAssistantError: Error updating Home Assistant Supervisor: Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
And the supervisor logs.
22-08-16 20:08:34 ERROR (SyncWorker_6) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
22-08-16 20:08:34 ERROR (MainThread) [supervisor.supervisor] Update of Supervisor failed: Can't install ghcr.io/home-assistant/amd64-hassio-supervisor:2022.08.3: 500 Server Error for http+docker://localhost/v1.41/auth: Internal Server Error ("Get "https://ghcr.io/v2/": denied: denied")
I’m thinking it’s a corrupted Docker setup as well. What bothers me is that it has apparently made its way into my backups. My main Home Assistant instance is a fresh install that I restored from a backup a little over a week ago.
This is also why I prefer YAML over UI configuration, and it really concerns me to see so many integrations moving over to a UI-only configuration flow. If I wanna take my back up out of the equation, I lose a shit load of configuration that was only possible through the UI.
Docker setups are not in yaml anyway.
It is setup at the creation of the container and you could manage it with add-ons like portainer earlier, but I think it was discovered that portainer trashed the network settings of the containers and was then removed as an add-on.
Sorry, I was mainly referring to HA configuration. If the issue has somehow ‘corrupted’ my backups, there’s a lot of HA configuration that I don’t have captured anywhere else but a JSON file somewhere in .storage, and it’s a little worrying. This is a fresh install that was restored from a backup. I know Docker is its own thing entirely. However, I try not to touch Docker or the underlying OS as much as possible, and let the supervisor handle everything. I remember when Portainer got pulled and it’s when I started writing my own add-ons instead.
I pulled the logs for docker from journalctl and there doesn’t seem to be any more information there that wasn’t already in the HA logs. Maybe I just need to start combing my backups for configurations that I need to keep and start fresh.
Make a new installation on a new media and test that you can pull the image without anything else install.
If you can do that then the error must be in the HA setup.
Well, I removed all of my container registry credentials from the supervisor, and it seems to be working now. I’ve never had an issue with my credentials. I’m curious if something changed in a recent version and I just missed it. Regardless, this seems to have corrected the issue. I will hold off on marking this as an answer for a few more days until I’m certain that it’s fixed.
Had the error have been a 403, I would have immediately known it was the credentials, but the 500 still makes me think it could have been something else.
Alright so I found that you removed the registries from the Add-On store that you had added yourself (I have none).
I also found out that after retrying the install/update a few dozen times it ends up going through (this happens of most updates it seems).
Very odd.
