Updating OS 9.0 (NOT Core/Supervisor) is failing with “Response payload is not completed”

A couple days ago I got notifications about new versions of Core, Supervisor, and OS being available. I updated Core and Supervisor without issue, but for OS I get the following (whether I try to update in the web/app UI or via ha os update:

22-09-20 16:16:29 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb: Response payload is not completed

As reported by About, I am running:

Home Assistant 2022.9.5
Supervisor 2022.09.1
Operating System 8.5
Frontend 20220907.2 - latest

Searching around for Response payload is not completed, it seems to be a pretty idiosyncratic issue—each query has a different answer (or, often, no answer at all). I did try the various things I saw mentioned:

  1. Restarted Supervisor
  2. Restarted the app
  3. Restarted the VM
  4. Restarted the hypervisor

all to no effect. (One reporter solved their issue by “rebooting the router”, but I’m not clear what that means in this context. The hypervisor’s bridge VLAN? The hypervisor OS? My cable modem?)

On both the HA OS VM and another Linux VM, if I try to do curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb, I get an empty file. If I try a different valid release filename (say, curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/haos_generic-aarch64-9.0.raucb), I also get an empty file.

If I try a nonexistent file (say, curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/thisDoesNotExist I get a file consisting of simply Not Found. So I do have connectivity to GitHub.

(If I try these on separate devices, rather than VM’s, I also get an empty file or Not Found. The only way I’ve actually downloaded a release is using a full-fledged browser.)

So:

  1. How might I go about continuing to troubleshoot this?
  2. I assume running the latest 9.5 Core and 9.1 Supervisor with an older 8.5 OS is not a recommended configuration—but what issues, if any, am I likely to run into while in this state?

So this part at least makes sense. If you repeat this command but change -O for -i you will see why:

# curl -i https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
HTTP/2 302
server: GitHub.com
date: Tue, 20 Sep 2022 22:13:15 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/115992009/65b0503a-56b3-462a-a182-e136b73b0fcc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220920T221315Z&X-Amz-Expires=300&X-Amz-Signature=4313b2c9719f08a16be8c0abc8b9702fc0fc98103b0c1faa18fa4bed96559235&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-content-disposition=attachment%3B%20filename%3Dhaos_ova-9.0.raucb&response-content-type=application%2Foctet-stream

Add -L to follow redirects like a browser does and get to the real file.

That being said, I’m stumped why supervisor would be hitting this. It makes the call like this:

And if you look in aiohttp docs, follow_redirects defaults to true.

If you run this command:

ha supervisor options --logging debug

And then try again, is there any more useful information in the supervisor logs?

2 Likes
22-09-21 16:18:52 DEBUG (MainThread) [supervisor.api.middleware.security] Passthrough /core/api/states/binary_sensor.backups_stale
22-09-21 16:18:52 DEBUG (MainThread) [supervisor.api.proxy] /core/api/states/binary_sensor.backups_stale access from cebe7a76_hassio_google_drive_backup
22-09-21 16:18:52 INFO (MainThread) [supervisor.api.middleware.security] /os/update access from core_ssh
22-09-21 16:18:52 INFO (MainThread) [supervisor.os.manager] Fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
22-09-21 16:18:53 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb: Response payload is not completed
22-09-21 16:18:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': False}}, 'id': 84}
22-09-21 16:18:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":84,"type":"result","success":true,"result":null}', extra='')
22-09-21 16:18:56 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh

Oh, and indeed:

[core-ssh ~]$ curl -OL https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  183M  100  183M    0     0  52.4M      0  0:00:03  0:00:03 --:--:-- 58.4M
[core-ssh ~]$ ls -lh haos_ova-9.0.raucb 
-rw-r--r--    1 root     root      183.7M Sep 21 16:21 haos_ova-9.0.raucb

Same problem, every time there is an OS update I’ve got the message.
Trying many times to update, sometimes works… I don’t understand why… :face_with_raised_eyebrow:

This is my log:

22-09-22 15:27:53 INFO (MainThread) [supervisor.os.manager] Fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_rpi4-64-9.0.raucb
22-09-22 15:27:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': True}}, 'id': 31}
22-09-22 15:27:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":31,"type":"result","success":true,"result":null}', extra='')
22-09-22 15:28:17 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_rpi4-64-9.0.raucb: Response payload is not completed
22-09-22 15:28:17 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': False}}, 'id': 32}