Updating OS 9.0 (NOT Core/Supervisor) is failing with “Response payload is not completed”

TreyH · September 20, 2022, 8:53pm

A couple days ago I got notifications about new versions of Core, Supervisor, and OS being available. I updated Core and Supervisor without issue, but for OS I get the following (whether I try to update in the web/app UI or via ha os update:

22-09-20 16:16:29 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb: Response payload is not completed

As reported by About, I am running:

Home Assistant 2022.9.5
Supervisor 2022.09.1
Operating System 8.5
Frontend 20220907.2 - latest

Searching around for Response payload is not completed, it seems to be a pretty idiosyncratic issue—each query has a different answer (or, often, no answer at all). I did try the various things I saw mentioned:

Restarted Supervisor
Restarted the app
Restarted the VM
Restarted the hypervisor

all to no effect. (One reporter solved their issue by “rebooting the router”, but I’m not clear what that means in this context. The hypervisor’s bridge VLAN? The hypervisor OS? My cable modem?)

On both the HA OS VM and another Linux VM, if I try to do curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb, I get an empty file. If I try a different valid release filename (say, curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/haos_generic-aarch64-9.0.raucb), I also get an empty file.

If I try a nonexistent file (say, curl -O https://github.com/home-assistant/operating-system/releases/download/9.0/thisDoesNotExist I get a file consisting of simply Not Found. So I do have connectivity to GitHub.

(If I try these on separate devices, rather than VM’s, I also get an empty file or Not Found. The only way I’ve actually downloaded a release is using a full-fledged browser.)

So:

How might I go about continuing to troubleshoot this?
I assume running the latest 9.5 Core and 9.1 Supervisor with an older 8.5 OS is not a recommended configuration—but what issues, if any, am I likely to run into while in this state?

CentralCommand · September 20, 2022, 10:18pm

So this part at least makes sense. If you repeat this command but change -O for -i you will see why:

# curl -i https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
HTTP/2 302
server: GitHub.com
date: Tue, 20 Sep 2022 22:13:15 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Encoding, Accept, X-Requested-With
location: https://objects.githubusercontent.com/github-production-release-asset-2e65be/115992009/65b0503a-56b3-462a-a182-e136b73b0fcc?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWNJYAX4CSVEH53A%2F20220920%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20220920T221315Z&X-Amz-Expires=300&X-Amz-Signature=4313b2c9719f08a16be8c0abc8b9702fc0fc98103b0c1faa18fa4bed96559235&X-Amz-SignedHeaders=host&actor_id=0&key_id=0&repo_id=115992009&response-content-disposition=attachment%3B%20filename%3Dhaos_ova-9.0.raucb&response-content-type=application%2Foctet-stream

Add -L to follow redirects like a browser does and get to the real file.

That being said, I’m stumped why supervisor would be hitting this. It makes the call like this:

github.com

home-assistant/supervisor/blob/c406814794476e0fb23f7332e58756ca4a3bbbcd/supervisor/os/manager.py#L99


      
              url = raw_url.format(
                  version=str(version), board=update_board, os_name=update_os_name
              )
              return url
          
          async def _download_raucb(self, url: str, raucb: Path) -> None:
              """Download rauc bundle (OTA) from URL."""
              _LOGGER.info("Fetch OTA update from %s", url)
              try:
                  timeout = aiohttp.ClientTimeout(total=60 * 60, connect=180)
                  async with self.sys_websession.get(url, timeout=timeout) as request:
                      if request.status != 200:
                          raise HassOSUpdateError(
                              f"Error raised from OTA Webserver: {request.status}",
                              _LOGGER.error,
                          )
          
                      # Download RAUCB file
                      with raucb.open("wb") as ota_file:
                          while True:
                              chunk = await request.content.read(1_048_576)

And if you look in aiohttp docs, follow_redirects defaults to true.

If you run this command:

ha supervisor options --logging debug

And then try again, is there any more useful information in the supervisor logs?

TreyH · September 21, 2022, 8:20pm

22-09-21 16:18:52 DEBUG (MainThread) [supervisor.api.middleware.security] Passthrough /core/api/states/binary_sensor.backups_stale
22-09-21 16:18:52 DEBUG (MainThread) [supervisor.api.proxy] /core/api/states/binary_sensor.backups_stale access from cebe7a76_hassio_google_drive_backup
22-09-21 16:18:52 INFO (MainThread) [supervisor.api.middleware.security] /os/update access from core_ssh
22-09-21 16:18:52 INFO (MainThread) [supervisor.os.manager] Fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
22-09-21 16:18:53 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb: Response payload is not completed
22-09-21 16:18:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': False}}, 'id': 84}
22-09-21 16:18:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":84,"type":"result","success":true,"result":null}', extra='')
22-09-21 16:18:56 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/logs access from core_ssh

TreyH · September 21, 2022, 8:23pm

Oh, and indeed:

[core-ssh ~]$ curl -OL https://github.com/home-assistant/operating-system/releases/download/9.0/haos_ova-9.0.raucb
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100  183M  100  183M    0     0  52.4M      0  0:00:03  0:00:03 --:--:-- 58.4M
[core-ssh ~]$ ls -lh haos_ova-9.0.raucb 
-rw-r--r--    1 root     root      183.7M Sep 21 16:21 haos_ova-9.0.raucb

Stokaz · September 22, 2022, 1:26pm

Same problem, every time there is an OS update I’ve got the message.
Trying many times to update, sometimes works… I don’t understand why…

Stokaz · September 22, 2022, 1:36pm

This is my log:

22-09-22 15:27:53 INFO (MainThread) [supervisor.os.manager] Fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_rpi4-64-9.0.raucb
22-09-22 15:27:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': True}}, 'id': 31}
22-09-22 15:27:53 DEBUG (MainThread) [supervisor.homeassistant.websocket] Received: WSMessage(type=<WSMsgType.TEXT: 1>, data='{"id":31,"type":"result","success":true,"result":null}', extra='')
22-09-22 15:28:17 ERROR (MainThread) [supervisor.os.manager] Can't fetch OTA update from https://github.com/home-assistant/operating-system/releases/download/9.0/haos_rpi4-64-9.0.raucb: Response payload is not completed
22-09-22 15:28:17 DEBUG (MainThread) [supervisor.homeassistant.websocket] Sending: {'type': <WSType.SUPERVISOR_EVENT: 'supervisor/event'>, 'data': {'event': <WSEvent.SUPERVISOR_UPDATE: 'supervisor_update'>, 'update_key': 'network', 'data': {'supervisor_internet': False}}, 'id': 32}

Stokaz · September 28, 2022, 6:52am

I have solved the problem… the problem was the powerline , connected directly to the router solved the issue!