Suffering from "out of memory" crashes every few weeks

johnhegs · December 16, 2021, 10:38am

Hi all. I am running HAOS as a VM on a Proxmox server. Everything runs really nicely except every few weeks HA crashes with this screen shot on the console.

The VM has 8GB ram so I don’t think this is a genuine “not enough memory to run HA”, but it might be a memory leak somewhere. Any ideas as to where to look next or what I can do to monitor things to get a lead on the cause ?

There isn’t much in the way of clues in the logs. Here is what the logs say around the time it failed -

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step delay 0:00:00

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Test condition template: True

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Choose at step 6: default: Running automation actions

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Choose at step 6: default: Executing step call service

2021-12-15 13:00:25 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:15:57 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'click' when rendering '{{ value_json.click }}'

2021-12-15 13:15:57 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:26:39 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:40:31 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 14:06:10 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'click' when rendering '{{ value_json.click }}'

2021-12-15 14:06:10 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_samba/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/7ad98f9c_zigbee2mqtt/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_influxdb/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_mosquitto/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_configurator/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_esphome/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/cebe7a76_hassio_google_drive_backup/stats request

2021-12-15 14:07:38 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data:

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:03 INFO (MainThread) [hass_nabucasa.iot] Connection closed

2021-12-15 14:12:03 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [140412314325344] Connection closed by client

2021-12-15 14:12:03 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [140412352188320] Connection closed by client

2021-12-15 14:12:09 INFO (MainThread) [hass_nabucasa.iot] Connected

2021-12-15 14:12:27 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for binary_sensor.remote_ui (<class 'homeassistant.components.cloud.binary_sensor.CloudRemoteBinary'>) took 14.146 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+cloud%22

2021-12-15 14:13:11 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sun.sun (<class 'homeassistant.components.sun.Sun'>) took 0.550 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue

2021-12-15 14:13:32 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_power (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.542 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:13:48 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_today (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.918 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:04 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_totalstarttime (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 1.191 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:13 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_voltage (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 1.143 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:39 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.tasmota_6420_energy_total (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.569 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:46 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.tasmota_6420_energy_yesterday (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.482 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:58 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140412688882800] Unexpected error inside websocket API

Traceback (most recent call last):

  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 210, in async_handle

    msg = await wsock.receive()

  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_ws.py", line 438, in receive

    await self.pong(msg.data)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_ws.py", line 293, in pong

    await self._writer.pong(message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 667, in pong

    await self._send_frame(message, WSMsgType.PONG)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 650, in _send_frame

    self._write(header + message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 660, in _write

    raise ConnectionResetError("Cannot write to closing transport")

ConnectionResetError: Cannot write to closing transport

2021-12-15 14:14:59 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 673, in ping

    await self._send_frame(message, WSMsgType.PING)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 650, in _send_frame

    self._write(header + message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 660, in _write

    raise ConnectionResetError("Cannot write to closing transport")

ConnectionResetError: Cannot write to closing transport

2021-12-15 14:14:59 ERROR (MainThread) [snitun.multiplexer.core] Ping fails, no response from peer

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Restarting

2021-12-15 14:14:59 INFO (MainThread) [hass_nabucasa.iot] Connection closed

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Running automation actions

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step call service

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Test condition template: True

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step wait for trigger

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Initialized trigger

2021-12-15 14:15:01 INFO (MainThread) [snitun.utils.aiohttp_client] AioHTTP snitun client connected to: eu-west-2-0.ui.nabu.casa:443

2021-12-15 14:15:09 INFO (MainThread) [hass_nabucasa.iot] Connected

anon63427907 · December 16, 2021, 11:04am

I would suggest starting from open topics, eg: Home Assistant Core shutting down unexpectedly

johnhegs · December 22, 2021, 12:51pm

Thanks, that thread was useful in that it prompted me to add the host sensor and add a graph for memory to one of my dashboards. I have been able to reproduce the issue by just rebooting and watching the memory used by core increase until the process is killed again.

Interestingly, I have managed to make it stable by running a “ha core restart” command on the console after it has failed. That leads to a normal memory usage. So once this is good for a few days I will run some more experiments to see if I can establish what is going wrong immediately after a reboot that is different from a simple restart of core.

tlewandster · July 5, 2022, 3:17pm

Hi, have you come to any conclusions? I have the same problem

nebbiadigiorno · August 16, 2022, 12:32pm

same problem here…

anon63427907 · August 16, 2022, 3:36pm

Come on, restarting home assistant regularly is far from stable but i understand you, i have gone through the same thing

I will summarize my case here;

i had a setup on raspberry pi 4b with raspberian OS and installed home assistant supervised.
when i start some add-ons, ha core was being shut down because of memory leak.
i just moved into debian based installation and no more memory leaks. (Still using supervised installation)
i tracked top activities for couple of days and the system was hanging on swapd process, which is supposed to write memory into file when memory is getting full. When this is happening, total memory usage was less than 25%>
i set an alarm using home assistant automations yo notify me when cpu usage is more than a threshold for a duration and jumping into investigation when this had happened.