Suffering from "out of memory" crashes every few weeks

Hi all. I am running HAOS as a VM on a Proxmox server. Everything runs really nicely except every few weeks HA crashes with this screen shot on the console.


The VM has 8GB ram so I don’t think this is a genuine “not enough memory to run HA”, but it might be a memory leak somewhere. Any ideas as to where to look next or what I can do to monitor things to get a lead on the cause ?

There isn’t much in the way of clues in the logs. Here is what the logs say around the time it failed -

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step delay 0:00:00

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Test condition template: True

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Choose at step 6: default: Running automation actions

2021-12-15 12:49:45 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Choose at step 6: default: Executing step call service

2021-12-15 13:00:25 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:15:57 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'click' when rendering '{{ value_json.click }}'

2021-12-15 13:15:57 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:26:39 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 13:40:31 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 14:06:10 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'click' when rendering '{{ value_json.click }}'

2021-12-15 14:06:10 WARNING (MainThread) [homeassistant.helpers.template] Template variable warning: 'dict object' has no attribute 'action' when rendering '{{ value_json.action }}'

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_samba/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/7ad98f9c_zigbee2mqtt/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_influxdb/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_mosquitto/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/core_configurator/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/a0d7b954_esphome/stats request

2021-12-15 14:07:38 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /addons/cebe7a76_hassio_google_drive_backup/stats request

2021-12-15 14:07:38 WARNING (MainThread) [homeassistant.components.hassio] Can't read Supervisor data:

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio.handler] Timeout on /ingress/validate_session request

2021-12-15 14:12:02 ERROR (MainThread) [homeassistant.components.hassio] Failed to to call /ingress/validate_session -

2021-12-15 14:12:03 INFO (MainThread) [hass_nabucasa.iot] Connection closed

2021-12-15 14:12:03 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [140412314325344] Connection closed by client

2021-12-15 14:12:03 INFO (MainThread) [homeassistant.components.websocket_api.http.connection] [140412352188320] Connection closed by client

2021-12-15 14:12:09 INFO (MainThread) [hass_nabucasa.iot] Connected

2021-12-15 14:12:27 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for binary_sensor.remote_ui (<class 'homeassistant.components.cloud.binary_sensor.CloudRemoteBinary'>) took 14.146 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+cloud%22

2021-12-15 14:13:11 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sun.sun (<class 'homeassistant.components.sun.Sun'>) took 0.550 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue

2021-12-15 14:13:32 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_power (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.542 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:13:48 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_today (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.918 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:04 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_totalstarttime (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 1.191 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:13 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.kitchen_rear_door_energy_voltage (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 1.143 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:39 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.tasmota_6420_energy_total (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.569 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:46 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.tasmota_6420_energy_yesterday (<class 'homeassistant.components.mqtt.sensor.MqttSensor'>) took 0.482 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+mqtt%22

2021-12-15 14:14:58 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140412688882800] Unexpected error inside websocket API

Traceback (most recent call last):

  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 210, in async_handle

    msg = await wsock.receive()

  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_ws.py", line 438, in receive

    await self.pong(msg.data)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/web_ws.py", line 293, in pong

    await self._writer.pong(message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 667, in pong

    await self._send_frame(message, WSMsgType.PONG)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 650, in _send_frame

    self._write(header + message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 660, in _write

    raise ConnectionResetError("Cannot write to closing transport")

ConnectionResetError: Cannot write to closing transport

2021-12-15 14:14:59 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved

Traceback (most recent call last):

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 673, in ping

    await self._send_frame(message, WSMsgType.PING)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 650, in _send_frame

    self._write(header + message)

  File "/usr/local/lib/python3.9/site-packages/aiohttp/http_websocket.py", line 660, in _write

    raise ConnectionResetError("Cannot write to closing transport")

ConnectionResetError: Cannot write to closing transport

2021-12-15 14:14:59 ERROR (MainThread) [snitun.multiplexer.core] Ping fails, no response from peer

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Restarting

2021-12-15 14:14:59 INFO (MainThread) [hass_nabucasa.iot] Connection closed

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Running automation actions

2021-12-15 14:14:59 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step call service

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Test condition template: True

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Executing step wait for trigger

2021-12-15 14:15:00 INFO (MainThread) [homeassistant.components.automation.new_automation] Cat Room Undercounter LEDs: Initialized trigger

2021-12-15 14:15:01 INFO (MainThread) [snitun.utils.aiohttp_client] AioHTTP snitun client connected to: eu-west-2-0.ui.nabu.casa:443

2021-12-15 14:15:09 INFO (MainThread) [hass_nabucasa.iot] Connected

I would suggest starting from open topics, eg: Home Assistant Core shutting down unexpectedly

Thanks, that thread was useful in that it prompted me to add the host sensor and add a graph for memory to one of my dashboards. I have been able to reproduce the issue by just rebooting and watching the memory used by core increase until the process is killed again.

Interestingly, I have managed to make it stable by running a “ha core restart” command on the console after it has failed. That leads to a normal memory usage. So once this is good for a few days I will run some more experiments to see if I can establish what is going wrong immediately after a reboot that is different from a simple restart of core.

1 Like