Cant update core or HAOS, cant reinstall either?

I have a raspberry pi 4 running HA 2023.1.7, HAOS 9.5 running off a 512GB SSD with 97% free space. Its running fine, but I can no longer update HA or HAOS ; whenever I try, I get this in the logs:

23-02-17 15:23:12 INFO (MainThread) [supervisor.homeassistant.core] Updating Home Assistant to version 2023.2.5
23-02-17 15:23:12 INFO (SyncWorker_6) [supervisor.docker.interface] Updating image ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.1.7 to ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.2.5
23-02-17 15:23:12 INFO (SyncWorker_6) [supervisor.docker.interface] Downloading docker image ghcr.io/home-assistant/raspberrypi4-64-homeassistant with tag 2023.2.5.
23-02-17 15:24:34 ERROR (SyncWorker_6) [supervisor.docker.interface] Can't install ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.2.5: 404 Client Error for http+docker://localhost/v1.41/images/ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.2.5/json: Not Found ("no such image: ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.2.5: No such image: ghcr.io/home-assistant/raspberrypi4-64-homeassistant:2023.2.5")
23-02-17 15:24:34 WARNING (MainThread) [supervisor.homeassistant.core] Updating Home Assistant image failed

Rebooting or restarting or power cycling doesnt help. I can update addons, just not core or HAOS.
edit, it seems I can no longer update addons either, I just tried updating node-red, and it gave a similar error:

23-02-21 20:31:11 INFO (SyncWorker_4) [supervisor.docker.addon] Updating image ghcr.io/hassio-addons/node-red/aarch64:14.0.2 to ghcr.io/hassio-addons/node-red/aarch64:14.0.3
23-02-21 20:31:11 INFO (SyncWorker_4) [supervisor.docker.interface] Downloading docker image ghcr.io/hassio-addons/node-red/aarch64 with tag 14.0.3.
23-02-21 20:32:37 ERROR (SyncWorker_4) [supervisor.docker.interface] Can't install ghcr.io/hassio-addons/node-red/aarch64:14.0.3: 404 Client Error for http+docker://localhost/v1.41/images/ghcr.io/hassio-addons/node-red/aarch64:14.0.3/json: Not Found ("no such image: ghcr.io/hassio-addons/node-red/aarch64:14.0.3: No such image: ghcr.io/hassio-addons/node-red/aarch64:14.0.3")

Googling I found one other user with a similar problem, and he “solved” it by reinstalling HAOS and then restoring a backup. I thought Id try that, so I prepared a new SSD with the Pi4 image, booted from that, got the “preparing HA” screen and for 4 hours thats all I got. When I checked my router, I could see HA was still ‘active’, as periodically I would see a few or few dozen MB/s download traffic. But nothing else happened, and this shouldnt last for hours upon hours.

I dont have a spare Pi to keep running overnight or troubleshoot, so I just inserted the old SSD again.

Im at a loss what to do or what causes this. The only thing I can think off is that around the time HA stopped updating, I installed a Deco mesh router, but its properly configured, IP address is reserved, everything is forwarded to the pi, it has full internet access, I configured google DNS, I have remote access to it… Ideas?
Can I SSH in to a fresh install, or can I prepare an SSD that already has HA fully installed instead of downloading it?

It is not uncommon for GitHub to have issues and they can last for hours and sometimes a few days.
The install process, updates, integrations and addons all get their files from GitHub.

No, thats not it, this has been going on for a month.

Could be a dns resolution problem. Try going to Open your Home Assistant instance and manage your systems network configuration. and change your ipv4 settings and use something like 8.8.8.8 or 1.1.1.1 as the DNS server.

Im already using 8.8.4.4, tried others too, and disabled ipv6

Can you resolve ghcr.io?

yes
image

Dunno, it is maddening isn’t it?

Its gotta be a networking thing somehow. The Pi in question was at a neighbour’s house. I just prepared HAOS on a mini PC at my house, then Installed a backup from their Pi, then did updates, all went fine. I deliberately didnt update one addon (node red). Went to the other house with the mini PC, and sure enough, that addon will not update, same error. I havent tried yet, but Im fairly confident the Pi will update at my house (edit: it does).

Both houses have the same ISP, same ISP router, same Deco mesh (in router mode, so double NAT in both cases with the deco in the ISP DMZ). In both cases IPv6 is disabled.

Really interested in how you solve, I’m having similar problem updating at my house. Thanks for posting.

You move it around?
Then MAC addresses are changed.
No idea if routes store MAC addresses, but on a IP layer of a network packets are routed based on a MAC address on the next hop and not on IP addresses (even though the layer is called IP).

First of all, the problem started long before I “moved things around”, and there is no mac filtering being done, it completely shouldnt matter. Its not like the HA machines dont have internet access, everything works, various cloud API’s, remote access, its only updates, and apparently only updates from github (or rather ghcr.io)

As I said, GitHub often have issues and sometimes for days.
Can you open the link on another machine?

I already posted a screenshot pinging ghcr.io from within HA, and already mentioned its been going on for over a month. I appreciate you wanting to help, but this is clearly an obscure issue and not something trivial. Its okay if you dont know the answer, I dont know it either, I dont know if anyone does, I just hope someone does or we can figure out a pattern.

Ok, tried the Pi at my house, and it updates with no problems. So it has to be networking related somehow. If you are having the same issue, do you spot something in common that stands out? Double NAT, TPlink Deco or something?

I can only say that I have seen similar threads. Search the forum and you may find. If I come across one I’ll let you know.

Ive seen a few very similar reports too; one was a thread with several posters from (IIRC) Saudi Arabia using cellular internet; I live in Belgium, not a lot of internet censorship here. The other was the one I referenced, someone who solved it with a fresh install, but even that doesnt work for me. And I saw a few ones that where never resolved and seemed to suggest it was a docker related problem, but that doesnt seem to apply to me either, given that moving the machine a few 100 meter “solves” it or creates the issue. If you find anything else, Im all ears.

Grasping at straws here, but in my house (where everything works) I use 192.168.0.x subnet for my LAN (192.168.1.x is used by my ISP’s router), my neighbor, where it doesnt work, uses 192.168.1.x.; I had to change the IP/subnet for the HA machines to be reachable. I cant imagine that being a problem, but then I cant imagine what would, so thought Id share for those facing the same problem.

You might be able to ping the server, but that does not show if the actual page/file exist there.
You logs do actually show a 404 error, so test it.

From HA machine unable to update:

From my machine:

Dont ask me if that means anything, but its identical.