HAOS update failing

Installation Home Assistant OS
28

@sairon; hereby the output from my system;
docker exec -ti hassio_supervisor bash:

89673d8bbb04:/# dig A github.com

; <<>> DiG 9.18.27 <<>> A github.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 51600
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 56f922e69e5fd32f (echoed)
;; QUESTION SECTION:
;github.com.                    IN      A

;; ANSWER SECTION:
github.com.             47      IN      A       140.82.121.4

;; Query time: 8 msec
;; SERVER: 172.30.32.3#53(172.30.32.3) (UDP)
;; WHEN: Mon Sep 02 19:07:41 UTC 2024
;; MSG SIZE  rcvd: 77

ping dns:

89673d8bbb04:/# ping dns
PING dns (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.167 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.088 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.096 ms
64 bytes from 172.30.32.3: seq=3 ttl=64 time=0.093 ms
64 bytes from 172.30.32.3: seq=4 ttl=64 time=0.100 ms
64 bytes from 172.30.32.3: seq=5 ttl=64 time=0.092 ms
^C
--- dns ping statistics ---
6 packets transmitted, 6 packets received, 0% packet loss
round-trip min/avg/max = 0.088/0.106/0.167 ms

ping 172.30.32.3:

89673d8bbb04:/# ping 172.30.32.3
PING 172.30.32.3 (172.30.32.3): 56 data bytes
64 bytes from 172.30.32.3: seq=0 ttl=64 time=0.123 ms
64 bytes from 172.30.32.3: seq=1 ttl=64 time=0.093 ms
64 bytes from 172.30.32.3: seq=2 ttl=64 time=0.099 ms
64 bytes from 172.30.32.3: seq=3 ttl=64 time=0.091 ms
64 bytes from 172.30.32.3: seq=4 ttl=64 time=0.092 ms
^C
--- 172.30.32.3 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max = 0.091/0.099/0.123 ms
29

Thanks, at this point I think we can rule out DNS - actually, it should have been obvious to me from the previous logs, where it resolves the address but fails to connect to it.

Anyway, I’m still quite puzzled. It even successfully connects to the Cloudflare HTTPS server which does a redirect to Github and then it fails. Using cURL could be a bit more informative, could you please try it as well? I.e. run curl -vL https://os-artifacts.home-assistant.io/13.1/haos_ova-13.1.raucb -O /dev/null

Also, please try it in both containers (i.e. in the hassio_supervisor and in the SSH add-on), that’s where I still don’t understand what’s different between those two scenarios.

30

After seeing a few sporadic instances of updates not working posted on this forum, I’ve started to think about the old quote:

Any sufficiently advanced technology is indistinguishable from magic.
Arthur C. Clarke

To put it another way, modern infrastructure has so many layers from LAN, MAN, WAN, CDN, hyperscaler, etc. that tying down a specific issue someone else is seeing from another part of the planet has become almost impossible.

  • You are unlikely to have the same local infrastructure as another
    server, LAN, mDNS, DNS, router, etc
  • You are unlikely to hit the same cloud infrastructure as another
    geographic DNS, load balancer, CDN cache server, etc
  • You are unlikely to see the same infrastructure state in 6 hours
    DNS balancing, cache updates / misses, etc

This feels a little defeatist to this Engineer, but suggests more than one way to do something might be useful. I really don’t have a robust answer, only a few ideas…

  • Try the “golden path” first - main URI linked to a global CDN for fast and local updates.
  • Gently complain if something fails with a non-scary unique message that can be easily searched for (e.g. grep in code like I did above).
  • If you can, try another way - could be another DNS name, another hosting service.
  • Offer a fall-back - link to docs, perhaps offer a link for manual download and xfer a file via USB worse-case.
  • Ideally spit out a code that tells the developer something about state (not a next-to-useless BSOD, but more like the Linux 6.12 QR code).

To give a related example, my bank’s web interface failed on Monday, so I spend an hour disabling browser plug-ins, changing DNS blocking, switching from VDSL to 5G networks, trying a phone app, different PC, etc.
Nope, all in vein - it was the bank infrastructure, that might have had a root-cause deep in Azure.
Two hours later, everything worked. :frowning: :face_with_symbols_over_mouth: :man_facepalming: :mage:

31

This is the output from the SSH (Putty):
Unfortunately, I have to cut the output in 2 parts due to forum limitations.

~ # curl -vL https://os-artifacts.home-assistant.io/13.1/haos_ova-13.1.raucb -O
/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*                                                                                                                                                              Host os-artifacts.home-assistant.io:443 was resolved.
* IPv6: 2606:4700:20::ac43:445a, 2606:4700:20::681a:5ee, 2606:4700:20::681a:4ee
* IPv4: 104.26.4.238, 104.26.5.238, 172.67.68.90
*   Trying 104.26.4.238:443...
* Connected to os-artifacts.home-assistant.io (104.26.4.238) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [2318 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / id-ecPublicKe                                                                                                                                                             y
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=os-arti                                                                                                                                                             facts.home-assistant.io
*  start date: Sep 28 00:00:00 2023 GMT
*  expire date: Sep 26 23:59:59 2024 GMT
*  subjectAltName: host "os-artifacts.home-assistant.io" matched cert's "os-arti                                                                                                                                                             facts.home-assistant.io"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha1WithRSAEncryption
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://os-artifacts.home-assistant.io/13.1/hao                                                                                                                                                             s_ova-13.1.raucb
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: os-artifacts.home-assistant.io]
* [HTTP/2] [1] [:path: /13.1/haos_ova-13.1.raucb]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /13.1/haos_ova-13.1.raucb HTTP/2
> Host: os-artifacts.home-assistant.io
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [230 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< date: Tue, 03 Sep 2024 18:53:20 GMT
< content-type: text/html
< content-length: 143
< location: https://github.com/home-assistant/operating-system/releases/download                                                                                                                                                             /13.1/haos_ova-13.1.raucb
< cache-control: private, max-age=0, no-store, no-cache, must-revalidate, post-c                                                                                                                                                             heck=0, pre-check=0
< expires: Thu, 01 Jan 1970 00:00:01 GMT
< report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=                                                                                                                                                             %2FYKO0TQVwCd66XnIAFoSmQAFjTBJNLAs7i5A3bOfLlYE7jn5st8ziO%2FVUgEX6OQdvtD1DR58mYlD                                                                                                                                                             pXoN%2Bq7Ep8R%2F51xYU6AHvBQ2bGw0cUDmvEbPhfGm5hRTaNo%2FB9JRCwWU23gADAE7m3YwMkJS4Q                                                                                                                                                             %3D%3D"}],"group":"cf-nel","max_age":604800}
< nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< server: cloudflare
< cf-ray: 8bd7f1293aefb8b4-AMS
< alt-svc: h3=":443"; ma=86400
<
* Ignoring the response-body
{ [5 bytes data]
100   143  100   143    0     0   2347      0 --:--:-- --:--:-- --:--:--  2383
* Connection #0 to host os-artifacts.home-assistant.io left intact
* Issue another request to this URL: 'https://github.com/home-assistant/operatin                                                                                                                                                             g-system/releases/download/13.1/haos_ova-13.1.raucb'
* Host github.com:443 was resolved.
* IPv6: (none)
* IPv4: 140.82.121.3
*   Trying 140.82.121.3:443...
* Connected to github.com (140.82.121.3) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3137 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [80 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / id-ecPublicKe                                                                                                                                                             y
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=github.com
*  start date: Mar  7 00:00:00 2024 GMT
*  expire date: Mar  7 23:59:59 2025 GMT
*  subjectAltName: host "github.com" matched cert's "github.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo                                                                                                                                                              ECC Domain Validation Secure Server CA
*  SSL certificate verify ok.
*   Certificate level 0: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA256
*   Certificate level 1: Public key type EC/prime256v1 (256/128 Bits/secBits), s                                                                                                                                                             igned using ecdsa-with-SHA384
*   Certificate level 2: Public key type EC/secp384r1 (384/192 Bits/secBits), si                                                                                                                                                             gned using ecdsa-with-SHA384
} [5 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://github.com/home-assistant/operating-sys                                                                                                                                                             tem/releases/download/13.1/haos_ova-13.1.raucb
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: github.com]
* [HTTP/2] [1] [:path: /home-assistant/operating-system/releases/download/13.1/h                                                                                                                                                             aos_ova-13.1.raucb]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /home-assistant/operating-system/releases/download/13.1/haos_ova-13.1.rauc                                                                                                                                                             b HTTP/2
> Host: github.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [57 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
< HTTP/2 302
< server: GitHub.com
< date: Tue, 03 Sep 2024 18:53:20 GMT
< content-type: text/html; charset=utf-8
< vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Acc                                                                                                                                                             ept, X-Requested-With
< location: https://objects.githubusercontent.com/github-production-release-asse                                                                                                                                                             t-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e4?X-Amz-Algorithm=AWS4-HMA                                                                                                                                                             C-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Fa                                                                                                                                                             ws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=300&X-Amz-Signature=74bd85                                                                                                                                                             5fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb8c0c&X-Amz-SignedHeaders=h                                                                                                                                                             ost&actor_id=0&key_id=0&repo_id=115992009&response-content-disposition=attachmen                                                                                                                                                             t%3B%20filename%3Dhaos_ova-13.1.raucb&response-content-type=application%2Foctet-                                                                                                                                                             stream
< cache-control: no-cache
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: no-referrer-when-downgrade
{ [5 bytes data]
32

Part 2:

< content-security-policy: default-src 'none'; base-uri 'self'; child-src github                                                                                                                                                             .com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/a                                                                                                                                                             ssets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com co                                                                                                                                                             llector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amaz                                                                                                                                                             onaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-produ                                                                                                                                                             ction-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-                                                                                                                                                             6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.a                                                                                                                                                             pi.visualstudio.com api.githubcopilot.com objects-origin.githubusercontent.com c                                                                                                                                                             opilot-proxy.githubusercontent.com/v1/engines/github-completion/completions prox                                                                                                                                                             y.enterprise.githubcopilot.com/v1/engines/github-completion/completions *.action                                                                                                                                                             s.githubusercontent.com wss://*.actions.githubusercontent.com productionresultss                                                                                                                                                             a0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ production                                                                                                                                                             resultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ pr                                                                                                                                                             oductionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows                                                                                                                                                             .net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core                                                                                                                                                             .windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.b                                                                                                                                                             lob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionres                                                                                                                                                             ultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ pro                                                                                                                                                             ductionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.window                                                                                                                                                             s.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.c                                                                                                                                                             ore.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultss                                                                                                                                                             a18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-p                                                                                                                                                             roduction-repository-image-32fea6.s3.amazonaws.com github-production-release-ass                                                                                                                                                             et-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com; font-src                                                                                                                                                              github.githubassets.com; form-action 'self' github.com gist.github.com copilot-w                                                                                                                                                             orkspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'n                                                                                                                                                             one'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com                                                                                                                                                             ; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com                                                                                                                                                              camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com                                                                                                                                                              github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.                                                                                                                                                             githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.gi                                                                                                                                                             thubusercontent.com opengraph.githubassets.com github-production-user-asset-6210                                                                                                                                                             df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com                                                                                                                                                              objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self                                                                                                                                                             '; media-src github.com user-images.githubusercontent.com/ secured-user-images.g                                                                                                                                                             ithubusercontent.com/ private-user-images.githubusercontent.com github-productio                                                                                                                                                             n-user-asset-6210df.s3.amazonaws.com gist.github.com; script-src github.githubas                                                                                                                                                             sets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-re                                                                                                                                                             quests; worker-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/                                                                                                                                                             assets/ gist.github.com/assets-cdn/worker/
< content-length: 0
{ [5 bytes data]
< x-github-request-id: 9AA8:AF024:146536AB:14C049EA:66D75B20
<
* Ignoring the response-body
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #1 to host github.com left intact
* Issue another request to this URL: 'https://objects.githubusercontent.com/gith                                                                                                                                                             ub-production-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e                                                                                                                                                             4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F202                                                                                                                                                             40903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=                                                                                                                                                             300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb                                                                                                                                                             8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-con                                                                                                                                                             tent-disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&response-content                                                                                                                                                             -type=application%2Foctet-stream'
* Host objects.githubusercontent.com:443 was resolved.
* IPv6: (none)
* IPv4: 185.199.111.133, 185.199.109.133, 185.199.108.133, 185.199.110.133
*   Trying 185.199.111.133:443...
* Connected to objects.githubusercontent.com (185.199.111.133) port 443
* ALPN: curl offers h2,http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [3099 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [36 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [36 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.io
*  start date: Mar 15 00:00:00 2024 GMT
*  expire date: Mar 14 23:59:59 2025 GMT
*  subjectAltName: host "objects.githubusercontent.com" matched cert's "*.github                                                                                                                                                             usercontent.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert Global G2 TLS RSA SHA256 2020 CA1
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed usi                                                                                                                                                             ng sha256WithRSAEncryption
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [193 bytes data]
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://objects.githubusercontent.com/github-pr                                                                                                                                                             oduction-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-4f3a766db1e4?X-A                                                                                                                                                             mz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetproduction%2F20240903                                                                                                                                                             %2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-Amz-Expires=300&X                                                                                                                                                             -Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd680960f2b7bb8c0c&                                                                                                                                                             X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-content-                                                                                                                                                             disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&response-content-type                                                                                                                                                             =application%2Foctet-stream
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: objects.githubusercontent.com]
* [HTTP/2] [1] [:path: /github-production-release-asset-2e65be/115992009/cab0904                                                                                                                                                             8-b8d6-414e-9faa-4f3a766db1e4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=                                                                                                                                                             releaseassetproduction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=202                                                                                                                                                             40903T185320Z&X-Amz-Expires=300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde                                                                                                                                                             6c89698faeb9c1dd680960f2b7bb8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&re                                                                                                                                                             po_id=115992009&response-content-disposition=attachment%3B%20filename%3Dhaos_ova                                                                                                                                                             -13.1.raucb&response-content-type=application%2Foctet-stream]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
} [5 bytes data]
> GET /github-production-release-asset-2e65be/115992009/cab09048-b8d6-414e-9faa-                                                                                                                                                             4f3a766db1e4?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=releaseassetprodu                                                                                                                                                             ction%2F20240903%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20240903T185320Z&X-A                                                                                                                                                             mz-Expires=300&X-Amz-Signature=74bd855fc1fa2550f2fa1939e87f5fde6c89698faeb9c1dd6                                                                                                                                                             80960f2b7bb8c0c&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&r                                                                                                                                                             esponse-content-disposition=attachment%3B%20filename%3Dhaos_ova-13.1.raucb&respo                                                                                                                                                             nse-content-type=application%2Foctet-stream HTTP/2
> Host: objects.githubusercontent.com
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
{ [5 bytes data]
< HTTP/2 200
< content-type: application/octet-stream
< last-modified: Wed, 21 Aug 2024 16:39:19 GMT
< etag: "0x8DCC1FFCDCC2C95"
< server: Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0
< x-ms-request-id: 3f6c359a-001e-003a-67ea-f3fb2d000000
< x-ms-version: 2020-10-02
< x-ms-creation-time: Wed, 21 Aug 2024 16:39:19 GMT
< x-ms-lease-status: unlocked
< x-ms-lease-state: available
< x-ms-blob-type: BlockBlob
< content-disposition: attachment; filename=haos_ova-13.1.raucb
< x-ms-server-encrypted: true
< via: 1.1 varnish, 1.1 varnish
< fastly-restarts: 1
< accept-ranges: bytes
< age: 2597
< date: Tue, 03 Sep 2024 18:53:20 GMT
< x-served-by: cache-iad-kiad7000102-IAD, cache-ams2100094-AMS
< x-cache: HIT, HIT
< x-cache-hits: 1, 0
< x-timer: S1725389600.454264,VS0,VE1
< content-length: 225982529
<
{ [5 bytes data]
100  215M  100  215M    0     0  23.8M      0  0:00:09  0:00:09 --:--:-- 24.9M
* Connection #2 to host objects.githubusercontent.com left intact
* URL rejected: No host part in the URL
* Closing connection
curl: (3) URL rejected: No host part in the URL
33

I uploaded logs to my cloud, as they’re too big to include in the post:

34

Sorry for the delay, but honestly, I’m none the wiser after reading those logs. Getting tcpdump traces could provide some more detail, I’ll provide the instructions how to get them if you’re willing to go down that rabbit hole :sweat_smile:

For a simpler test, can you try downloading a similar file from a different URL? OS development build would be best, i.e. replace the URL with https://os-artifacts.home-assistant.io/13.2.dev20240909/haos_ova-13.2.dev20240909.ova. Again, just to be sure, check that the stable release link still doesn’t work at the same time.

35

Please post the instructions – I might try it.

I’ll also try downloading this file later today or tomorrow and post the results.

36

This works, the file started to be downloaded. The URL for stable *.raucb file still does not work.

37

OK, the issue has just got much worse.

After vm host reboot Home Assistant can no longer access the store with addons. The store is empty and all addons are listed without icons etc. Also, HA reports problems “installed addon has been removed from repository” for every single addon I have.

And that killed Matter entirely, as it does not load the entities, displaying this message instead: “Failed to get the Matter Server add-on info: Addon core_matter_server with version latest does not exist in the store”. So now all my Matter devices are unusable.

Cloud based integration work. It looks like the problem is with everything that eventually connects to GitHub.

38

Not that I have a solution for you, it’s now 29 days ago since your first report about this and only now you are mentioning VM…
Apart of that, you’re not sharing much about your setup.

39

I did share it in a GitHub issue, from which Jan actually came here I expect.

You’re right, I probably should have posted it here as well. Sorry. Actually I thought it will be the issue where it will be discussed, but Jan decided to continue here and he already had the information from the issue.

So, all the info about my configuration is in the linked issue. If I can provide anything more to help track this issue down, please ask.

40

That’s quite strange. There should be no difference in how the network is set up in the Supervisor container and in the other ones. Out of curiosity, have you tried if you can reproduce the same issue in a different VM? Then we can proceed with further checks - it will be also good to start with that, so we can capture network traffic only from that single VM.

41

@sairon, actually I do. I checked the Nextcloud AIO I have on another VM and I cannot use the menu script. It relies on GitHub as well. Also, the Apps section cannot be accessed as well and it reports there is no Internet connection. At the same time file sync, apps and web app work both inside and outside the network.

Some time ago I moved to the other place temporarily and I moved the computer to other network. I reconfigured DNS records etc. I’m starting to think it can be related, although weirdly the HA addon store didn’t fail right after moving, but only after about a month when I restarted the machine.

The most significant difference is that I don’t have IPv6 now and I did have it before.

Network config on both HA and Nextcloud is automatic. I do have SSL configured, browser reports certificate as valid.

I have a feeling that two factors involved here might be GitHub and IPv4, allthough I don’t have a clue if it makes sense.

Oh. And the Home Assistant Core has updated successfully right now to version 2024.9.2.

42

@sairon
Sorry for my late response. Been a bit busy with work.

I tried both the stable version download and the other one you asked for. Both outputs end with “URL rejected: No host part in the URL”.
I compared the output with my previous output and there are some differences like timestamps etc.
Are there any specific texts I should be looking for in the output?

43

OK.

Eventually also due to sub-par connection speed I decided to replace the router with a new one (the previous was very old, I think it had Wi-Fi 3 only or so).

All the problems are gone, store works, Matter works, Home Assistant OS updated successfully to 13.1.

I have totally no idea what exactly the problem was. I also have no idea if that router was somehow faulty apart from being ancient or is there really some problem with Home Assistant itself that affects only some configurations.

Thanks for your effort to help! :slight_smile:

44

This turned out to be a combination of two things, one of which I consider a deficiency in HA, versus my connection to the outside world.

  1. HA sets up containers with their network interface MTU at 1500, even though it (correctly) gets a local network MTU (which can be lower) for it’s host-OS external interface. Result: the TCP SYN carries an MSS for full-size packets.

  2. Github is ignoring ICMP frag-needed packets sent its way. Although it’s possibly my internet providers fault, that is less likely if others are getting this problem.

So a big packet from github (eg. the TLS Server Hello) never gets back to HA, and the https connection never gets fully made. So all your add-ons repos disappear, and HA claims that the add-ons you are running have been deleted from their repos.

My workaround is to get my local firewall to clamp outbound TCP MSS values. Unfortunately this is unlikely to be feasible for many people.

2 Likes
45

I have looked into my router settings, but there I cannot find an option to limit package size or something like that.

Also tried to update HA by downloading the file, putting it onto an usb drive and inserting it into the raspberry, that seems to do nothing, also not after a restart. Probably i’m doing something wrong there because if I understand it correctly, this way of updating should work.

As far as I know, my setup is very common (just a raspberry pi 5 with a standard HA setup). What could trigger my HA to not be able to update, while it seems that a there are not a lot of other people having the same issue?

Hope there is someone out there that can help me. If any extra info is needed, please ask.

With kind regards,
Danny

46

Last update for my issue; I found on another forum that this issue can be caused by an incorrectly installed SSD. That was also the root cause in my system; after installing HA on an SD-card, I attached an SSD drive and moved the files to the SSD. It seems to be that the system files are also moved, but also remain on the SD-card (if I understood the explanation on the other forum correctly). Then there are 2 instances of the software and during updating, it doesn’t know which one to update.

The solution in my case; make a full backup, connect the SSD drive to my PC and burn the HA image on there. Remove the SD-card from the Raspberry Pi and connect the SSD. Start HA and put the backup back.
I only had to fix the MQTT user credentials. After that, everything was ok.

Hope that someone in the future has some use for this info.

Regards, Danny

47

Yeah, the SSD issue is another problem not related to the issue above when not being able to download the update from github.

I can’t upgrade from 13.2 - > 14.0 as the update throws a “failed updating slot boot.0: failed to run slot hook: child process exited with code 1” error.

Yes, I’m running HA from nvme SSD attachet to a nvme hat on my pi5. I however never had any sdcard attached with HA on it, I flashed the SSD directly from pios with piimager…

So if anyone has any idea, please let me know!

Maybe a clean install is the only way forward.

EDIT: It turned out that the error above was an issue from trying out HAOS 14.0 rc-1, which failed. It messed up with the boot slots. A clean install (wipe of SSD) solved that problem. However, now, when I can upgrade to 14.0, it fails during boot.