Home Assistant automatic restart for API call error?

drmilos · December 6, 2023, 12:56pm

I was doing everything. Disabled all add-ons one by one. Disabled all integrations. Then uninstalled all of them and I got “cleaned up” not fresh install but cleaned up HA. Issue was existing. I left it as it is and after I replaced router after several days I noticed that HA HW is running for 3 days already without reboot. I left it as it is for few more days, then initiated reboot and enabled all integrations and add-ons as it was obvious that they not creating any issues.

Maybe it makes no sense, but it doesn’t cost you anything to test it. Just make other reservation and leave it for couple of days and see

spanzetta · December 7, 2023, 3:16pm

I have also now a surprise…

My poor Raspberry PI3A+ (with 512Mb Ram and SD Card) is not rebooting from 19 hours… which is a real record for me from over one Year…
But I didn’t change anything in IP/router settings…
So maybe… it was the latest update that fixed something?
Will see in the next days…

PS really I changed one thing… changed record commit_interval from 5 sec (default) to 10 sec… but I don’t think that this was making this surprise

spanzetta · December 9, 2023, 8:11am

It was just an illusion…

After one day without a single restart…at one point it did totally stop responding (the web ui… observer instead was responding with Connected, Supported, Healthy)…

And after a manual reboot (had to remove power) the behavior is now the usual: it restarts many times per day…

I noticed always these warning/error in the log (the only relevant imho)… could it be one of these the cause of the reboot?
Why it happens so often?

23-12-09 08:21:17 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-09 08:22:46 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/config:

23-12-09 08:49:33 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-09 08:50:35 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-09 08:50:35 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!

23-12-09 08:57:46 WARNING (SyncWorker_0) [supervisor.addons.validate] Add-on config ‘startup’ with ‘before’ is deprecated. Please report this to the maintainer of MQTT Importer
23-12-09 08:57:46 WARNING (SyncWorker_0) [supervisor.addons.validate] Add-on config ‘startup’ with ‘before’ is deprecated. Please report this to the maintainer of Openvpn Client

23-12-09 10:01:13 ERROR (MainThread) [supervisor.homeassistant.api] on call https://172.30.32.1:8123/api/core/state:
23-12-09 10:01:51 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-09 10:01:52 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
23-12-09 10:01:57 INFO (SyncWorker_0) [supervisor.docker.manager] Restarting homeassistant
23-12-09 10:03:31 WARNING (MainThread) [supervisor.misc.tasks] Watchdog/Application found a problem with observer plugin!

spanzetta · December 10, 2023, 4:12pm

Now it is again sistematic…

Every few hours (it can be 2 up to max 4 or 5) I see these errors in the log and my systems reboots

How many people here has the same behavior??

23-12-10 16:07:02 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
23-12-10 16:21:10 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-10 16:22:35 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:
23-12-10 16:22:35 ERROR (MainThread) [supervisor.misc.tasks] Watchdog found a problem with Home Assistant API!
23-12-10 16:22:41 INFO (SyncWorker_2) [supervisor.docker.manager] Restarting homeassistant

spanzetta · December 22, 2023, 8:28am

after many tests it came out (at least in my case, my setup) that the problem was caused by MQTT …
Removed it and the system is again much much more stable/it doesn’t restart spontaneously so often

MatteGary · December 22, 2023, 11:23am

Hi Stefano, is your issue completely solved?
I have the same issue in my installation, with container being constantly mounted again by the watchdog, and the system is really slow and unresponsive.
Did you find any root cause?
Thanks

spanzetta · December 22, 2023, 12:56pm

Yes… now my system is absolutelly much more stable (no reboot in 24 hours…)
I just disabled (removed) MQTT broker and integration

RSPT · December 25, 2023, 3:09pm

I think this is just a symptom of the lack of resources a rpi3 has, for what hass currently needs.
I don’t have mqtt broker running and still have occasional hangs and reboots
I bet nobody has this issue when running rpi4 with 4gb or more

spanzetta · December 26, 2023, 6:42am

Sure… but now that error about Api call is not happening… so there must be something wrong thar was causing that error (and then the reboots)

DarthJacks · January 4, 2024, 9:19am

I am having the same issue for 2 months now.
I thought it was a minor fix that would be adressed in a following update of supervisor but no supervisor update after 2023.12.0
On my side i also get multiple other errors/warnings
search online points me to the scenario that my network is misbehaving… i dont know if that should cause HA to produce so many errors in supervisor logs
pasting some logs here:

return await self._method(obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/store/git.py", line 178, in pull
    raise StoreGitError() from err
supervisor.exceptions.StoreGitError
24-01-04 08:59:31 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-1147821' coro=<Repository.update() done, defined at /usr/src/supervisor/supervisor/store/repository.py:104> exception=StoreGitError()>
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/store/git.py", line 136, in pull
    await self.sys_run_in_executor(
  File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/git/remote.py", line 1014, in fetch
    res = self._get_fetch_info_from_stderr(proc, progress, kill_after_timeout=kill_after_timeout)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/git/remote.py", line 853, in _get_fetch_info_from_stderr
    proc.wait(stderr=stderr_text)
  File "/usr/local/lib/python3.11/site-packages/git/cmd.py", line 600, in wait
    raise GitCommandError(remove_password_if_present(self.args), status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git fetch -v --update-shallow --depth=1 -- origin
  stderr: 'fatal: unable to access 'https://github.com/allenporter/stream-addons/': Could not resolve host: github.com'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/store/repository.py", line 108, in update
    await self.git.pull()
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 296, in wrapper
    raise err
  File "/usr/src/supervisor/supervisor/jobs/decorator.py", line 289, in wrapper
    return await self._method(obj, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/supervisor/supervisor/store/git.py", line 178, in pull
    raise StoreGitError() from err
supervisor.exceptions.StoreGitError
24-01-04 09:07:32 INFO (MainThread) [supervisor.homeassistant.core] Home Assistant Core state changed to RUNNING
24-01-04 09:07:32 INFO (MainThread) [supervisor.homeassistant.core] Detect a running Home Assistant instance
....
....
24-01-04 10:20:59 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
24-01-04 10:31:26 INFO (MainThread) [supervisor.api.middleware.security] /supervisor/info access from cebe7a76_hassio_google_drive_backup
24-01-04 10:31:30 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
24-01-04 10:32:02 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state:

DarthJacks · January 5, 2024, 10:44am

new logs from today when my system rebooted again:

24-01-05 12:24:19 INFO (MainThread) [supervisor.api.middleware.security] /backups access from cebe7a76_hassio_google_drive_backup
24-01-05 12:28:51 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-01-05 12:30:28 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
24-01-05 12:31:00 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-01-05 12:31:00 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-01-05 12:31:01 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-01-05 12:34:00 ERROR (MainThread) [supervisor.homeassistant.api] Error on call https://172.30.32.1:8123/api/core/state: 
24-01-05 12:34:00 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-9397' coro=<WSClient.start_listener() done, defined at /usr/src/supervisor/supervisor/homeassistant/websocket.py:97> exception=ConnectionResetError('Cannot write to closing transport')>
Traceback (most recent call last):
  File "/usr/src/supervisor/supervisor/homeassistant/websocket.py", line 104, in start_listener
    await self._receive_json()
  File "/usr/src/supervisor/supervisor/homeassistant/websocket.py", line 113, in _receive_json
    msg = await self._client.receive()
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_ws.py", line 280, in receive
    await self.pong(msg.data)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/client_ws.py", line 160, in pong
    await self._writer.pong(message)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 709, in pong
    await self._send_frame(message, WSMsgType.PONG)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 675, in _send_frame
    self._write(header + mask + message)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 702, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
24-01-05 12:34:00 ERROR (MainThread) [asyncio] Task exception was never retrieved
future: <Task finished name='Task-12556' coro=<WebSocketWriter.ping() done, defined at /usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py:711> exception=ConnectionResetError('Cannot write to closing transport')>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 715, in ping
    await self._send_frame(message, WSMsgType.PING)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 675, in _send_frame
    self._write(header + mask + message)
  File "/usr/local/lib/python3.11/site-packages/aiohttp/http_websocket.py", line 702, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
24-01-05 12:34:01 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API error: Cannot write to closing transport
24-01-05 12:34:01 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API connection is closed
24-01-05 12:34:14 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request initialize
24-01-05 12:34:15 INFO (MainThread) [supervisor.api.proxy] WebSocket access from a0d7b954_nodered
24-01-05 12:34:15 INFO (MainThread) [supervisor.api.proxy] Home Assistant WebSocket API request running
24-01-05 12:39:35 INFO (MainThread) [supervisor.resolution.check] Starting system checks with state running
24-01-05 12:39:35 INFO (MainThread) [supervisor.resolution.checks.base] Run check for multiple_data_disks/system
24-01-05 12:39:35 INFO (MainThread) [supervisor.resolution.checks.base] Run check for ipv4_connection_problem/system
24-01-05 12:39:35 INFO (MainThread) [supervisor.resolution.checks.base] Run check for trust/supervisor
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_ipv6_error/dns_server
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for docker_config/system
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for security/core
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for dns_server_failed/dns_server
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for pwned/addon
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.checks.base] Run check for free_space/system
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.check] System checks complete
24-01-05 12:39:36 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state running
24-01-05 12:39:41 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete
24-01-05 12:39:41 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state running
24-01-05 12:39:41 INFO (MainThread) [supervisor.resolution.fixups.store_execute_reset] Reset corrupt Store: 3d360630
24-01-05 12:39:44 INFO (MainThread) [supervisor.store.git] Cloning add-on https://github.com/allenporter/stream-addons repository
24-01-05 12:39:49 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete

paoloantinori · January 18, 2024, 1:54pm

I’ve managed to mitigate the issue this way:

github.com/home-assistant/core

Home Assistant Supervisor: Could not fetch stats

opened 07:25PM - 03 Jul 22 UTC

closed 04:06AM - 29 Oct 23 UTC

besiktas97

integration: hassio stale

### The problem Logger: homeassistant.components.hassio Source: components/has…sio/__init__.py:993 Integration: Home Assistant Supervisor (documentation, issues) First occurred: July 2, 2022 at 01:39:03 (507 occurrences) Last logged: 21:14:34 Could not fetch stats for 03b5fe11_ssh: Could not fetch stats for core_samba: Could not fetch stats for 15d21743_samba_backup: Could not fetch stats for 15d21743_samba_backup: 404 Client Error for http+docker://localhost/v1.41/containers/addon_15d21743_samba_backup/json: Not Found ("No such container: addon_15d21743_samba_backup") Could not fetch stats for core_configurator: ### What version of Home Assistant Core has the issue? core-2022.6.7 ### What was the last working version of Home Assistant Core? _No response_ ### What type of installation are you running? Home Assistant OS ### Integration causing the issue Samba backup ### Link to integration documentation on our website https://community.home-assistant.io/t/samba-backup-create-and-store-backups-on-a-samba-share/199471 ### Diagnostics information _No response_ ### Example YAML snippet _No response_ ### Anything in the logs that might be useful for us? ```txt Logger: homeassistant.components.hassio.handler Source: components/hassio/handler.py:263 Integration: Home Assistant Supervisor (documentation, issues) First occurred: July 2, 2022 at 02:09:39 (13 occurrences) Last logged: 21:14:34 Timeout on /addons/03b5fe11_ssh/stats request Timeout on /addons/core_samba/stats request Timeout on /addons/15d21743_samba_backup/stats request Timeout on /supervisor/stats request Timeout on /addons/core_configurator/stats request ``` ### Additional information _No response_

disable_watchdog: "curl -sSL -H \"Authorization: Bearer $SUPERVISOR_TOKEN\" -H \"Content-Type: application/json\" -d '{\"watchdog\": false }' http://supervisor/core/options"

triggered by an automation that kicks in at ha boot

- alias: HA startup
  trigger:
    platform: homeassistant
    event: start
  action:
  - delay: 00:01:00
  - service: shell_command.disable_watchdog
    data: {}

spanzetta · February 7, 2024, 12:01pm

After have had again the multiple reboot evey day again for many days… starting from yesterday, after the latest updates, my PI3A+ is NOT rebooting from over ONE entire day (which is a miracle)…

I have to assume that the latest version really solved the problem…

Now the question is: should I stop any future update for not risking to get again the problem ?

Current version is

Core 2024.1.6
Supervisor 2024.01.1
Operating System 11.5
Frontend 20240104.0

spanzetta · February 19, 2024, 8:35am

Updated to these versions and it is still very very stable

Core 2024.2.2
Supervisor 2024.01.1
Operating System 11.5
Frontend 20240207.1
Very happy…
I hope it will continue like this…

spanzetta · February 19, 2024, 11:05am

sometimes I only see these … but the system is not restarting so frequently as in the past

Login attempt or request with invalid authentication from supervisor (172.30.32.2). Requested URL: ‘/api/core/state’. (HomeAssistantSupervisor/2024.01.1 aiohttp/3.9.3 Python/3.12)

and

24-02-19 09:42:38 INFO (MainThread) [supervisor.homeassistant.api] Updated Home Assistant API token
24-02-19 09:50:15 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call https://172.30.32.1:8123/api/core/state.

googlehupf · February 25, 2024, 3:28pm

I have a similiar problem as you guys facing. It got worse when I played around with dashboards today. I was creating and modifying a markdown template dashboard item, where I looped over something like this

{{ 
  expand(states.sensor) 
  |selectattr('state', 'eq', 'unavailable') 
  |map(attribute='entity_id')
  | join('\n') 
}}

when I saw the UI freeze and the supervisor log saying

24-02-25 16:23:04 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call http://172.30.32.1:8123/api/core/state.

So I guess for the state something is wrong which makes it freeze. I try to fix it now by deleting the part in the dashboard.

bgoncal · February 28, 2024, 2:21pm

I recently had the same issue and the root cause CALDav integration connected to an icloud email. Removing that fixed everything

spanzetta · March 1, 2024, 11:32am

In my case… it is happening again a frequent reboot (:-() and it seems it’s caused by core_mosquitto…
I already had in the past this feeling (that core mosquitt was the cause of reboot) but then with latest updates it did not create any problem…
Now it started again…

24-03-01 10:54:37 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call https://172.30.32.1:8123/api/core/state.
24-03-01 10:54:42 ERROR (MainThread) [supervisor.homeassistant.api] Timeout on call https://172.30.32.1:8123/api/core/state.
24-03-01 10:54:43 WARNING (MainThread) [supervisor.misc.tasks] Watchdog missed an Home Assistant Core API response.
24-03-01 10:55:33 WARNING (MainThread) [supervisor.misc.tasks] Watchdog found a problem with core_mosquitto application!
24-03-01 10:55:45 INFO (SyncWorker_7) [supervisor.docker.manager] Stopping addon_core_mosquitto application
24-03-01 10:56:24 INFO (SyncWorker_7) [supervisor.docker.manager] Cleaning addon_core_mosquitto application
24-03-01 10:56:47 ERROR (MainThread) [supervisor.misc.tasks] Watchdog missed 2 Home Assistant Core API responses in a row. Restarting Home Assistant Core API!

Salsalove · March 9, 2024, 2:49pm

@spanzetta Hai risolto? Io non ne riesco a venire a capo da settimane ormai.
I miei problemi iniziano quando avvio Node-red (che utilizza core-mosquitto). Ma se uso solo core-mosquitto (per un altro add-on che ne fa uso) non ho di questi errori.

spanzetta · March 9, 2024, 3:21pm

Purtroppo la situazione è estremamente variabile… e non credo avra’ mai una soluzione definitiva.

Ad ogni aggiornamento la situazione può migliorare o peggiorare … e quando sembra stabile (es dopo update Core a 2024.3) poi magari ad un certo punto senza motivo inizia a non rispondere piu e inizia a fare reboot uno dietro l’altro… poi magari torna ad essere stabile…(mi e’ successo proprio ieri, e non era la prima volta).

Dai log l’unica cosa che ho notato è che i due add-on che piu di altri sembrano provocare i blocchi (e quindi i reboot dovuti al watchdog) sono Core Mosquitto (che pero mi serve per cui e’ ON) e File Editor (che tengo spento e accendo solo se serve)…

Ma può anche essere fuorviante… che i problemi sono causati da altro… difficile scoprirlo…

Sino ad un riavvio ogni uno/due giorni lo considero normale (per un RPI con 512Mb di Ram) … quando si riavvia 10 volte in due ore anche no…

Io non ho node-red…