HAOS update failing

TiberiumBlue · August 26, 2024, 9:29pm

Hi guys,
I’m quite new to Home Assistant, so maybe i’m missing something. My home assistant is running on a raspberry pi 5.

Earlier I got notifications to update core or OS and that went fine, so I think that in the basis, my setup should be correct.
Now when trying to install OS 13.1, I get a notification “Failed to perform the action update/install. Error updating Home Assistant Operating System: Unknown error, see supervisor”. I got the same error when trying to update to 13.0 (from 12.3), but eventually skipped that install. This is already for a few days now.

When looking at the supervisor output, I get this line;

2024-08-26 23:00:52.509 ERROR (MainThread) [supervisor.os.manager] Home Assistant Operating System update failed with: Installation error: Failed updating slot boot.0: failed to run slot hook: Child process exited with code 1

I tried rebooting, unplugging, rebooting the whole network. That all didn’t help.
Above the advice is given to try trough ssh;

dig github. com

Response;

; <<>> DiG 9.18.27 <<>> github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 540
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: d6bb6887ef217901 (echoed)
;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             55      IN      A       140.82.121.3

;; Query time: 8 msec
;; SERVER: 172.30.32.3#53(172.30.32.3) (UDP)
;; WHEN: Mon Aug 26 23:14:52 CEST 2024
;; MSG SIZE  rcvd: 77

host github. com
response;

github.com has address 140.82.121.3
github.com mail is handled by 5 alt2.aspmx.l.google.com.
github.com mail is handled by 10 alt4.aspmx.l.google.com.
github.com mail is handled by 10 alt3.aspmx.l.google.com.
github.com mail is handled by 1 aspmx.l.google.com.
github.com mail is handled by 5 alt1.aspmx.l.google.com.

Traceroute github. com
respons;

traceroute to github.com (140.82.121.4), 30 hops max, 46 byte packets
 1  router.domain_not_set.invalid (192.168.1.254)  0.617 ms  1.048 ms  0.607 ms
 2  *  *  *
 3  *  *  *
 4  100.64.0.18 (100.64.0.18)  3.902 ms  3.769 ms  3.706 ms
 5  *  *  *
 6  62.45.255.114 (62.45.255.114)  4.743 ms  4.936 ms  4.649 ms
 7  er1.ams1.nl.above.net (80.249.208.122)  5.268 ms  4.827 ms  5.103 ms
 8  *  *  *
 9  *  *  *
10  ae1.mcs1.fra6.de.eth.zayo.com (64.125.29.57)  9.993 ms  9.977 ms  10.097 ms
11  82.98.193.31.IPYX-270403-001-ZYO.zip.zayo.com (82.98.193.31)  9.486 ms  9.842 ms  14.318 ms
12  *  *  *
13  *  *  *
14  *  *  *
15  *  *  *
16  *  *  *
17  *  *  *
18  *  *  *
19  *  *  *
20  *  *  *
21  *  *  *
22  *  *  *
23  *  *  *
24  *  *  *
25  *  *  *
26  *  *  *
27  *  *  *
28  *  *  *
29  *  *  *
30  *  *  *

I think that my Home Assistant is still connecting to Github (also because a core update yesterday worked flawless).

Can anyone help me to find the cause that the OS updates are not working anymore?

If more info is necessary, please ask.
Any advice would be appreciated.

With kind regards,
Danny

p.s. some more info about my setup;

Board; Raspberry Pi 5
Core; 2024.8.3
Supervisor: 2024.08.0
Operating system; 12.3
Frontend: 20240809.0
Storage; 2% used (separate SSD)
Wired connection to network, using DHCP

digitalyours · September 1, 2024, 1:21pm

I’m on a RPi4, but exactly same behaviour. It is not DNS and not disk space related. Nothing helped so far.

nickrout · September 1, 2024, 7:54pm

What about the gchr.io address?

sairon · September 2, 2024, 7:19am

It seems there’s some DNS or connectivity problem for the Supervisor container only. Could you try attaching to the Supervisor container using docker exec -ti hassio_supervisor bash (either directly on the host/VM after typing login or through the Advanced SSH terminal; standard SSH/web terminal won’t work) and check what fails there? I will start with dig A github.com and ping dns and/or ping 172.30.32.3 (as DNS should be resolved through the CoreDNS plugin).

mipioter · September 2, 2024, 3:59pm

@sairon, here are the results:

d974dd697d8c:/# dig A github.com

; <<>> DiG 9.18.27 <<>> A github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 31257
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: aabda0d0713c5bdd (echoed)
;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             49      IN      A       140.82.121.3

;; Query time: 4 msec
;; SERVER: 172.30.32.3#53(172.30.32.3) (UDP)
;; WHEN: Mon Sep 02 16:02:23 UTC 2024
;; MSG SIZE  rcvd: 77

d974dd697d8c:/# ping dns
PING dns (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.107 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.215 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.220 ms
^C
--- dns ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.107/0.180/0.220 ms

PING 172.30.32.3 (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.235 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.171 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.192 ms
^C
--- 172.30.32.3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.171/0.199/0.235 ms

I tried wgetting the update from the docker. It fails to connect:

d974dd697d8c:/# wget https://os-artifacts.home-assistant.io/13.1/haos_ova-13.1.raucb
Connecting to os-artifacts.home-assistant.io (172.67.68.90:443)
Connecting to github.com (140.82.121.3:443)

...no more output here

At the same time, pinging the IP wget tries to connect succeeds:

d974dd697d8c:/# ping 140.82.121.3
PING 140.82.121.3 (140.82.121.3): 56 data bytes
64 bytes from 140.82.121.3: seq=0 ttl=52 time=25.662 ms
64 bytes from 140.82.121.3: seq=1 ttl=52 time=25.423 ms
64 bytes from 140.82.121.3: seq=2 ttl=52 time=25.587 ms
^C
--- 140.82.121.3 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 25.423/25.557/25.662 ms

TiberiumBlue · September 2, 2024, 7:02pm

What about the gchr.io address?<

I have tried to google gchr.io address, but couldn’t find what that is. Are you referring to the ghcr (GitHub Container Regitry)?
How can I find the answer to your question in my system?

nickrout · September 2, 2024, 7:04pm

Yes, sorry I was.

Same as above.

TiberiumBlue · September 2, 2024, 7:11pm

@sairon; hereby the output from my system;
docker exec -ti hassio_supervisor bash:

89673d8bbb04:/# dig A github.com

; <<>> DiG 9.18.27 <<>> A github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51600
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 56f922e69e5fd32f (echoed)
;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             47      IN      A       140.82.121.4

;; Query time: 8 msec
;; SERVER: 172.30.32.3#53(172.30.32.3) (UDP)
;; WHEN: Mon Sep 02 19:07:41 UTC 2024
;; MSG SIZE  rcvd: 77

ping dns:

89673d8bbb04:/# ping dns
PING dns (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.167 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.088 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.096 ms
64 bytes from 172.30.32.3: seq=3 ttl=64 time=0.093 ms
64 bytes from 172.30.32.3: seq=4 ttl=64 time=0.100 ms
64 bytes from 172.30.32.3: seq=5 ttl=64 time=0.092 ms
^C
--- dns ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.088/0.106/0.167 ms

ping 172.30.32.3:

89673d8bbb04:/# ping 172.30.32.3
PING 172.30.32.3 (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.123 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.093 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.099 ms
64 bytes from 172.30.32.3: seq=3 ttl=64 time=0.091 ms
64 bytes from 172.30.32.3: seq=4 ttl=64 time=0.092 ms
^C
--- 172.30.32.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.091/0.099/0.123 ms

sairon · September 3, 2024, 2:06pm

Thanks, at this point I think we can rule out DNS - actually, it should have been obvious to me from the previous logs, where it resolves the address but fails to connect to it.

Anyway, I’m still quite puzzled. It even successfully connects to the Cloudflare HTTPS server which does a redirect to Github and then it fails. Using cURL could be a bit more informative, could you please try it as well? I.e. run curl -vL https://os-artifacts.home-assistant.io/13.1/haos_ova-13.1.raucb -O /dev/null

Also, please try it in both containers (i.e. in the hassio_supervisor and in the SSH add-on), that’s where I still don’t understand what’s different between those two scenarios.

FloatingBoater · September 3, 2024, 3:04pm

After seeing a few sporadic instances of updates not working posted on this forum, I’ve started to think about the old quote:

Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke

To put it another way, modern infrastructure has so many layers from LAN, MAN, WAN, CDN, hyperscaler, etc. that tying down a specific issue someone else is seeing from another part of the planet has become almost impossible.

You are unlikely to have the same local infrastructure as another
server, LAN, mDNS, DNS, router, etc
You are unlikely to hit the same cloud infrastructure as another
geographic DNS, load balancer, CDN cache server, etc
You are unlikely to see the same infrastructure state in 6 hours
DNS balancing, cache updates / misses, etc

This feels a little defeatist to this Engineer, but suggests more than one way to do something might be useful. I really don’t have a robust answer, only a few ideas…

Try the “golden path” first - main URI linked to a global CDN for fast and local updates.
Gently complain if something fails with a non-scary unique message that can be easily searched for (e.g. grep in code like I did above).
If you can, try another way - could be another DNS name, another hosting service.
Offer a fall-back - link to docs, perhaps offer a link for manual download and xfer a file via USB worse-case.
Ideally spit out a code that tells the developer something about state (not a next-to-useless BSOD, but more like the Linux 6.12 QR code).

To give a related example, my bank’s web interface failed on Monday, so I spend an hour disabling browser plug-ins, changing DNS blocking, switching from VDSL to 5G networks, trying a phone app, different PC, etc.
Nope, all in vein - it was the bank infrastructure, that might have had a root-cause deep in Azure.
Two hours later, everything worked.

TiberiumBlue · September 3, 2024, 7:10pm

This is the output from the SSH (Putty):
Unfortunately, I have to cut the output in 2 parts due to forum limitations.

~ # curl -vL https://os-artifacts.home-assistant.io/13.1/haos_ova-13.1.raucb -O
/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*                                                                                                                                                              Host os-artifacts.home-assistant.io:443 was resolved.
* IPv6: 2606:4700:20::ac43:445a, 2606:4700:20::681a:5ee, 2606:4700:20::681a:4ee
* IPv4: 104.26.4.238, 104.26.5.238, 172.67.68.90
*   Trying 104.26.4.238:443...
* Connected to os-artifacts.home-assistant.io (104.26.4.238) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2318 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKe                                                                                                                                                             y
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=os-arti                                                                                                                                                             facts.home-assistant.io
*  start date: Sep 28 00:00:00 2023 GMT
*  expire date: Sep 26 23:59:59 2024 GMT
*  subjectAltName: host "os-artifacts.home-assistant.io" matched cert's "os-arti                                                                                                                                                             facts.home-assistant.io"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha1WithRSAEncryption
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://os-artifacts.home-assistant.io/13.1/hao                                                                                                                                                             s_ova-13.1.raucb
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: os-artifacts.home-assistant.io]
* [HTTP/2] [1] [:path: /13.1/haos_ova-13.1.raucb]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /13.1/haos_ova-13.1.raucb HTTP/2
> Host: os-artifacts.home-assistant.io
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Tue, 03 Sep 2024 18:53:20 GMT
< content-type: text/html
< content-length: 143
< location: https://github.com/home-assistant/operating-system/releases/download                                                                                                                                                             /13.1/haos_ova-13.1.raucb
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-c                                                                                                                                                             heck=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=                                                                                                                                                             %2FYKO0TQVwCd66XnIAFoSmQAFjTBJNLAs7i5A3bOfLlYE7jn5st8ziO%2FVUgEX6OQdvtD1DR58mYlD                                                                                                                                                             pXoN%2Bq7Ep8R%2F51xYU6AHvBQ2bGw0cUDmvEbPhfGm5hRTaNo%2FB9JRCwWU23gADAE7m3YwMkJS4Q                                                                                                                                                             %3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8bd7f1293aefb8b4-AMS
< alt-svc: h3=":443"; ma=86400
<
* Ignoring the response-body
{ [5 bytes data]
100   143  100   143    0     0   2347      0 --:--:-- --:--:-- --:--:--  2383
* Connection #0 to host os-artifacts.home-assistant.io left intact
* Issue another request to this URL: 'https://github.com/home-assistant/operatin                                                                                                                                                             g-system/releases/download/13.1/haos_ova-13.1.raucb'
* Host github.com:443 was resolved.
* IPv6: (none)
* IPv4: 140.82.121.3
*   Trying 140.82.121.3:443...
* Connected to github.com (140.82.121.3) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3137 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKe                                                                                                                                                             y
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo                                                                                                                                                              ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), si                                                                                                                                                             gned using ecdsa-with-SHA384
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://github.com/home-assistant/operating-sys                                                                                                                                                             tem/releases/download/13.1/haos_ova-13.1.raucb
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: github.com]
* [HTTP/2] [1] [:path: /home-assistant/operating-system/releases/download/13.1/h                                                                                                                                                             aos_ova-13.1.raucb]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /home-assistant/operating-system/releases/download/13.1/haos_ova-13.1.rauc                                                                                                                                                             b HTTP/2
> Host: github.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< server: GitHub.com
< date: Tue, 03 Sep 2024 18:53:20 GMT
< content-type: text/html; charset=utf-8
< vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Acc                                                                                                                                                             ept, X-Requested-With
< location: https://objects.githubusercontent.com/github-production-release-asse                                                                                                                                                             t-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e4?X-Amz-Algorithm=AWS4-HMA                                                                                                                                                             C-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Fa                                                                                                                                                             ws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=300&X-Amz-Signature=74bd85                                                                                                                                                             5fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb8c0c&X-Amz-SignedHeaders=h                                                                                                                                                             ost&actor_id=0&key_id=0&repo_id=115992009&response-content-disposition=attachmen                                                                                                                                                             t%3B%20filename%3Dhaos_ova-13.1.raucb&response-content-type=application%2Foctet-                                                                                                                                                             stream
< cache-control: no-cache
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: no-referrer-when-downgrade
{ [5 bytes data]

TiberiumBlue · September 3, 2024, 7:15pm

Part 2:

< content-security-policy: default-src 'none'; base-uri 'self'; child-src github                                                                                                                                                             .com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/a                                                                                                                                                             ssets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com co                                                                                                                                                             llector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amaz                                                                                                                                                             onaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-produ                                                                                                                                                             ction-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-                                                                                                                                                             6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.a                                                                                                                                                             pi.visualstudio.com api.githubcopilot.com objects-origin.githubusercontent.com c                                                                                                                                                             opilot-proxy.githubusercontent.com/v1/engines/github-completion/completions prox                                                                                                                                                             y.enterprise.githubcopilot.com/v1/engines/github-completion/completions *.action                                                                                                                                                             s.githubusercontent.com wss://*.actions.githubusercontent.com productionresultss                                                                                                                                                             a0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ production                                                                                                                                                             resultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ pr                                                                                                                                                             oductionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows                                                                                                                                                             .net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core                                                                                                                                                             .windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.b                                                                                                                                                             lob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionres                                                                                                                                                             ultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ pro                                                                                                                                                             ductionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.window                                                                                                                                                             s.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.c                                                                                                                                                             ore.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultss                                                                                                                                                             a18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-p                                                                                                                                                             roduction-repository-image-32fea6.s3.amazonaws.com github-production-release-ass                                                                                                                                                             et-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com; font-src                                                                                                                                                              github.githubassets.com; form-action 'self' github.com gist.github.com copilot-w                                                                                                                                                             orkspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'n                                                                                                                                                             one'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com                                                                                                                                                             ; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com                                                                                                                                                              camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com                                                                                                                                                              github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.                                                                                                                                                             githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.gi                                                                                                                                                             thubusercontent.com opengraph.githubassets.com github-production-user-asset-6210                                                                                                                                                             df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com                                                                                                                                                              objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self                                                                                                                                                             '; media-src github.com user-images.githubusercontent.com/ secured-user-images.g                                                                                                                                                             ithubusercontent.com/ private-user-images.githubusercontent.com github-productio                                                                                                                                                             n-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubas                                                                                                                                                             sets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-re                                                                                                                                                             quests; worker-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/                                                                                                                                                             assets/ gist.github.com/assets-cdn/worker/
< content-length: 0
{ [5 bytes data]
< x-github-request-id: 9AA8:AF024:146536AB:14C049EA:66D75B20
<
* Ignoring the response-body
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #1 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/gith                                                                                                                                                             ub-production-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e                                                                                                                                                             4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F202                                                                                                                                                             40903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=                                                                                                                                                             300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb                                                                                                                                                             8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-con                                                                                                                                                             tent-disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&response-content                                                                                                                                                             -type=application%2Foctet-stream'
* Host objects.githubusercontent.com:443 was resolved.
* IPv6: (none)
* IPv4: 185.199.111.133, 185.199.109.133, 185.199.108.133, 185.199.110.133
*   Trying 185.199.111.133:443...
* Connected to objects.githubusercontent.com (185.199.111.133) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3099 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*  start date: Mar 15 00:00:00 2024 GMT
*  expire date: Mar 14 23:59:59 2025 GMT
*  subjectAltName: host "objects.githubusercontent.com" matched cert's "*.github                                                                                                                                                             usercontent.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [193 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://objects.githubusercontent.com/github-pr                                                                                                                                                             oduction-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e4?X-A                                                                                                                                                             mz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903                                                                                                                                                             %2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=300&X                                                                                                                                                             -Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb8c0c&                                                                                                                                                             X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-content-                                                                                                                                                             disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&response-content-type                                                                                                                                                             =application%2Foctet-stream
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: objects.githubusercontent.com]
* [HTTP/2] [1] [:path: /github-production-release-asset-2e65be/115992009/cab0904                                                                                                                                                             8-b8d6-414e-9faa-4f3a766db1e4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=                                                                                                                                                             releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202                                                                                                                                                             40903T185320Z&X-Amz-Expires=300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde                                                                                                                                                             6c89698faeb9c1dd680960f2b7bb8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&re                                                                                                                                                             po_id=115992009&response-content-disposition=attachment%3B%20filename%3Dhaos_ova                                                                                                                                                             -13.1.raucb&response-content-type=application%2Foctet-stream]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /github-production-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-                                                                                                                                                             4f3a766db1e4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetprodu                                                                                                                                                             ction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-A                                                                                                                                                             mz-Expires=300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd6                                                                                                                                                             80960f2b7bb8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&r                                                                                                                                                             esponse-content-disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&respo                                                                                                                                                             nse-content-type=application%2Foctet-stream HTTP/2
> Host: objects.githubusercontent.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
< HTTP/2 200
< content-type: application/octet-stream
< last-modified: Wed, 21 Aug 2024 16:39:19 GMT
< etag: "0x8DCC1FFCDCC2C95"
< server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 3f6c359a-001e-003a-67ea-f3fb2d000000
< x-ms-version: 2020-10-02
< x-ms-creation-time: Wed, 21 Aug 2024 16:39:19 GMT
< x-ms-lease-status: unlocked
< x-ms-lease-state: available
< x-ms-blob-type: BlockBlob
< content-disposition: attachment; filename=haos_ova-13.1.raucb
< x-ms-server-encrypted: true
< via: 1.1 varnish, 1.1 varnish
< fastly-restarts: 1
< accept-ranges: bytes
< age: 2597
< date: Tue, 03 Sep 2024 18:53:20 GMT
< x-served-by: cache-iad-kiad7000102-IAD, cache-ams2100094-AMS
< x-cache: HIT, HIT
< x-cache-hits: 1, 0
< x-timer: S1725389600.454264,VS0,VE1
< content-length: 225982529
<
{ [5 bytes data]
100  215M  100  215M    0     0  23.8M      0  0:00:09  0:00:09 --:--:-- 24.9M
* Connection #2 to host objects.githubusercontent.com left intact
* URL rejected: No host part in the URL
* Closing connection
curl: (3) URL rejected: No host part in the URL

mipioter · September 4, 2024, 5:28am

I uploaded logs to my cloud, as they’re too big to include in the post:

sairon · September 10, 2024, 2:55pm

Sorry for the delay, but honestly, I’m none the wiser after reading those logs. Getting tcpdump traces could provide some more detail, I’ll provide the instructions how to get them if you’re willing to go down that rabbit hole

For a simpler test, can you try downloading a similar file from a different URL? OS development build would be best, i.e. replace the URL with https://os-artifacts.home-assistant.io/13.2.dev20240909/haos_ova-13.2.dev20240909.ova. Again, just to be sure, check that the stable release link still doesn’t work at the same time.

mipioter · September 11, 2024, 4:48am

Please post the instructions – I might try it.

I’ll also try downloading this file later today or tomorrow and post the results.

mipioter · September 12, 2024, 5:28pm

This works, the file started to be downloaded. The URL for stable *.raucb file still does not work.

mipioter · September 13, 2024, 4:44am

OK, the issue has just got much worse.

After vm host reboot Home Assistant can no longer access the store with addons. The store is empty and all addons are listed without icons etc. Also, HA reports problems “installed addon has been removed from repository” for every single addon I have.

And that killed Matter entirely, as it does not load the entities, displaying this message instead: “Failed to get the Matter Server add-on info: Addon core_matter_server with version latest does not exist in the store”. So now all my Matter devices are unusable.

Cloud based integration work. It looks like the problem is with everything that eventually connects to GitHub.

Nick4 · September 13, 2024, 8:25am

Not that I have a solution for you, it’s now 29 days ago since your first report about this and only now you are mentioning VM…
Apart of that, you’re not sharing much about your setup.

mipioter · September 13, 2024, 3:15pm

I did share it in a GitHub issue, from which Jan actually came here I expect.

You’re right, I probably should have posted it here as well. Sorry. Actually I thought it will be the issue where it will be discussed, but Jan decided to continue here and he already had the information from the issue.

So, all the info about my configuration is in the linked issue. If I can provide anything more to help track this issue down, please ask.

sairon · September 16, 2024, 10:21am

That’s quite strange. There should be no difference in how the network is set up in the Supervisor container and in the other ones. Out of curiosity, have you tried if you can reproduce the same issue in a different VM? Then we can proceed with further checks - it will be also good to start with that, so we can capture network traffic only from that single VM.