Supervisor timeout on call (API) and global system freeze

Stefano201 · February 20, 2024, 12:51pm

Hi, I’ve been trying to understand what happened to my HA instance for days without a solution, despite having read hundreds of posts around the web.

For some issues, I decided to create a new HA instance but I have the same problem

I go straight to the point, because after a series of tests, I realized that everything comes from the Supervisor and these are some errors that appear (at least once or twice in an hour) with a freeze of the system for a few minutes and sometimes (not always) can also lead to an automatic reboot.

The issues starts when I launch Node-red that use mosquitto broker to discover the entities of my alarm panel

Supervisor:

24-02-18 20:00:06 INFO (MainThread) [supervisor.host.info] Updating local host information
24-02-18 20:00:07 INFO (MainThread) [supervisor.host.services] Updating service information
24-02-18 20:00:07 INFO (MainThread) [supervisor.host.network] Updating local network information
24-02-18 20:00:08 INFO (MainThread) [supervisor.host.sound] Updating PulseAudio information
24-02-18 20:00:09 INFO (MainThread) [supervisor.host.manager] Host information reload completed

24-02-18 20:02:25 WARNING (MainThread) [supervisor.misc.tasks] Watchdog missing application response from core_mosquitto
24-02-18 20:02:26 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

24-02-18 20:02:27 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 20:02:27 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257

24-02-18 20:02:27 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 20:02:28 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 20:02:44 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running

Core:

2024-02-18 20:02:38.404 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547237622336] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.01.1 aiohttp/3.9.3 Python/3.12): Unexpected error inside websocket API
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 385, in async_handle
msg = await wsock.receive()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 486, in receive
await self.pong(msg.data)
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 328, in pong
await self._writer.pong(message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 709, in pong
await self._send_frame(message, WSMsgType.PONG)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

2024-02-18 20:02:39.402 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 715, in ping
await self._send_frame(message, WSMsgType.PING)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

2024-02-18 20:02:39.440 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 715, in ping
await self._send_frame(message, WSMsgType.PING)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

Supervisor:

24-02-18 18:50:32 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 18:50:32 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 18:51:02 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
24-02-18 19:06:04 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:06:04 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 19:06:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:06:05 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed

24-02-18 19:06:37 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:06:42 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:06:42 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:07:28 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:07:29 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

24-02-18 19:07:32 WARNING (MainThread) [supervisor.homeassistant.websocket] Connection is closed
24-02-18 19:07:36 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 19:07:37 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 19:07:37 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 19:07:37 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 19:07:38 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 19:07:38 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 19:21:08 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token

24-02-18 19:28:21 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

24-02-18 19:28:29 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:28:29 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:28:30 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 19:28:30 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 19:28:50 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-02-18 19:46:39 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:46:48 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:46:48 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-02-18 19:46:48 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot proxy websocket message of unsupported type: 257
24-02-18 19:46:48 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed

24-02-18 19:47:44 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:47:45 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.
24-02-18 19:48:20 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

24-02-18 19:48:29 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-02-18 19:48:30 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize

Core:

2024-02-18 19:29:41.691 ERROR (MainThread) [custom_components.tapo_control] HTTPSConnectionPool(host='192.168.1.113', port=443): Max retries exceeded with url: /stok=453c37d512095564d7903c214d0e3661/ds (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1000)')))

2024-02-18 19:48:25.700 ERROR (Thread-12 (_thread_main)) [paho.mqtt.client] failed to receive on socket: [Errno 32] Broken pipe

2024-02-18 19:48:25.810 WARNING (Thread-12 (_thread_main)) [homeassistant.components.mqtt.client] Disconnected from MQTT server 192.168.1.49:1883 (7)

2024-02-18 19:48:26.996 ERROR (MainThread) [custom_components.tapo_control] ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

2024-02-18 19:48:27.128 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547244313024] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.01.1 aiohttp/3.9.3 Python/3.12): Unexpected error inside websocket API
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 385, in async_handle
msg = await wsock.receive()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 486, in receive
await self.pong(msg.data)
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 328, in pong
await self._writer.pong(message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 709, in pong
await self._send_frame(message, WSMsgType.PONG)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

2024-02-18 19:48:27.454 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [547244316864] Supervisor from 172.30.32.2 (HomeAssistantSupervisor/2024.01.1 aiohttp/3.9.3 Python/3.12): Unexpected error inside websocket API
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 385, in async_handle
msg = await wsock.receive()
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 486, in receive
await self.pong(msg.data)
File "/usr/local/lib/python3.12/site-packages/aiohttp/web_ws.py", line 328, in pong
await self._writer.pong(message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 709, in pong
await self._send_frame(message, WSMsgType.PONG)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2024-02-18 19:48:27.722 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 715, in ping
await self._send_frame(message, WSMsgType.PING)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2024-02-18 19:48:27.831 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 715, in ping
await self._send_frame(message, WSMsgType.PING)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 682, in _send_frame
self._write(header + message)
File "/usr/local/lib/python3.12/site-packages/aiohttp/http_websocket.py", line 702, in _write
raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport

I tried to downgrade the Core (from 2024.2.2 to 2023.12.0), because I didn’t have these issues in November and December, but I didn’t solve it.

My current versions are:

Core 2023.12.0
<Supervisor 2024.01.1
Operating System 11.5

daneboom · February 26, 2024, 7:41pm

Similar issue

OvalZyre · March 8, 2024, 4:01pm

Similar issue after latest update. Restarted and system just hangs and logs show Timeout on call errors. Restored to previous backup and things are fine. Not sure what the underlying issue is.

daneboom · March 8, 2024, 8:30pm

so frustrating. SOMETHING has changed. I keep updating to the latest version hoping it will resolve my restart issue but it doesn’t.

daydy · June 3, 2024, 7:55pm

Did you find the root cause? I’m having similar issues with the 2024.5 installation on a raspberry pi 4.

[supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

boulder240 · June 4, 2024, 4:51am

I am having the same issue!

T_L · June 6, 2024, 10:33am

The same error here
The system also hangs, lots of reading of disk and higher CPU usage.

Cbarth3 · June 7, 2024, 8:40pm

I just recently started seeing this same issue of random freezing. It started after I upgraded zwavejs to 3.7.1 but downgraded to 3.6.0 and still see the issue. I had been running 2024.5.1 without any issues and tried updating to 2024.6.0 to see if it would help. So far nothing has helped.

Update: well not certain yet what caused this for me but i restored a backup from 5/30 (automatic backups every other day) and no lock ups. Glad to see my backups worked and to be able to have it working again.

Stefano201 · June 30, 2024, 3:34pm

I fixed it by replacing my Rpi4 with an Intel Mini Pc and a backup of my HaOS instance. No more freeze or error in supervisor logs.

Tomsh · August 13, 2024, 8:02am

Hi,

Same error with HAOS on Intel NUC.

It started few months ago but since last release (2024.08 / 2024.08.1) it’s just crashing few times again.

Anyone managed to fixed it? Any guidance regarding how to seek for relevant logs?

Thanks!

webmediart-github · August 13, 2024, 3:15pm

Same error on 2024.7.4, no significant changes to the system. It just randomly stopped working yesterday and is not fixed by power cycling. GUI down, SSH works (how I got the errors in the supervisor log). Quite frustrating, not knowing what caused it or where to start for a fix.
For me the “ha core check” failed with a timeout, which I find odd and no similar cases on the web.

1aranzant · August 27, 2024, 5:11pm

Got a similar issue, (HAOS on lenovo tinypc), nothing works anymore… Here are my issues:

Cannot open any add-ons from HA → Unable to create an Ingress session
Cannot update HAOS → Failed to perform the action update/install. Error updating Home Assistant Operating System: ‘OSManager.update’ blocked from execution, no supervisor internet connection
Cannot restart HA from UI. (even SSH ha core restart & ha host reboot don’t do anything)

all this while Home Assistant observer says everything is connected/supported/healthy

mhoogenbosch · October 3, 2024, 2:25pm

With the upgrade to 2024.10.0 i’ve got the same issue too. Don’t think it is related, more a coincidence. Because when I restore my backup from older versions, the issue persists.

I’ve updated my system to docker 27.3.1 too, maybe that’s something? I cannot seem to resolve the issue.

Did any of you guys resolve the issue?

jmad · October 3, 2024, 6:15pm

Same issue here today with 2024.10.0.

Was up fine since the upgrade last night, but started seeing errrors in the supervisor log with timeouts today. At a loss to explain.

mhoogenbosch · October 3, 2024, 6:44pm

Using Alexa integration? There is a banner saying it causes issues with the latest version.

lobmarib · October 3, 2024, 9:49pm

Same here… stops working and very high cpu usage

lobmarib · October 3, 2024, 10:04pm

I’ve reverted to 2024.9.3 and same problem… Anyone can help?

platini761 · October 3, 2024, 10:16pm

same problem here with the new update of ngix, alexa and 2024.10

dgcoliveira · October 4, 2024, 12:07am

Just had the same issue and disabling Alexa Media Player seems to have fixed it.

Tismo · October 4, 2024, 2:15am

I remember updating alexa media player few hours ago. thank you. Did you disable the integration?

UPDATE: there is a github issue on the matter.