Repeated crashes due to MemoryError

Configuration
1

Hi all,

my homeassistant installation on a Raspberry Pi 4 4GB / SD Card is crashing every couple of days. Seems the OS continues to run though, as it is not rebooting. Deconz (running via supervisor) also stops working, so my entire light setup stops working.

I found two other threads but no solution (Memory exhausted by python3 process - process/container crash - #5 by basnijholt) (HA repeatedly crashing every few hours, suspected memory leak - #2 by gmoney)

I added logging of RAM, CPU, Swap usage and also a persistent log for errors. Before crash RAM is at ~25% and nothing exceptional in CPU or Swap either. I get an MemoryError which is logged as Error and I see some problem with Deconz in the regular log:

Error log:

06:43 2021-12-14 ERROR [homeassistant] Error doing job: Task exception was never retrieved Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 197, in async_init
    flow, result = await task
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 224, in _async_init
    result = await self._async_handle_step(flow, flow.init_step, data, init_done)
  File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 320, in _async_handle_step
    result: FlowResult = await getattr(flow, method)(user_input)
  File "/config/custom_components/bosch_shc/config_flow.py", line 198, in async_step_zeroconf
    self.info = await self._get_info(host)
  File "/config/custom_components/bosch_shc/config_flow.py", line 232, in _get_info
    return await self.hass.async_add_executor_job(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/bosch_shc/config_flow.py", line 73, in get_info_from_host
    information = session.mdns_info()
  File "/usr/local/lib/python3.9/site-packages/boschshcpy/session.py", line 277, in mdns_info
    return SHCInformation(
  File "/usr/local/lib/python3.9/site-packages/boschshcpy/information.py", line 80, in __init__
    self.get_unique_id(zeroconf)
  File "/usr/local/lib/python3.9/site-packages/boschshcpy/information.py", line 134, in get_unique_id
    self._listener = SHCListener(zeroconf, self.filter)
  File "/usr/local/lib/python3.9/site-packages/boschshcpy/information.py", line 32, in __init__
    ServiceBrowser(zeroconf, "_http._tcp.local.", handlers=[self.service_update])
  File "/usr/local/lib/python3.9/site-packages/zeroconf/_services/browser.py", line 516, in __init__
    self.start()
  File "/usr/local/lib/python3.9/threading.py", line 892, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
06:48 2021-12-14 ERROR [root] Uncaught thread exception Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/src/homeassistant/homeassistant/components/stream/__init__.py", line 284, in _run_worker
    stream_worker(
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 489, in stream_worker
    segment_buffer.mux_packet(packet)
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 175, in mux_packet
    self.flush(packet, last_part=True)
  File "/usr/src/homeassistant/homeassistant/components/stream/worker.py", line 236, in flush
    self._av_output.close()
  File "av/container/output.pyx", line 197, in av.container.output.OutputContainer.close
  File "av/container/output.pyx", line 23, in av.container.output.close_output
  File "av/container/core.pyx", line 258, in av.container.core.Container.err_check
  File "av/error.pyx", line 323, in av.error.err_check
  File "av/container/pyio.pyx", line 38, in av.container.pyio.pyio_write_gil
MemoryError
07:18 2021-12-14 ERROR [homeassistant] Error doing job: Fatal read error on socket transport Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError
07:18 2021-12-14 ERROR [homeassistant] Error doing job: Fatal read error on socket transport Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError

homeassistant.log:

2021-12-14 07:17:59 INFO (MainThread) [homeassistant.components.automation.flutlicht_eingang_an] Flutlicht Eingang an: Running automation actions
2021-12-14 07:17:59 INFO (MainThread) [homeassistant.components.automation.flutlicht_eingang_an] Flutlicht Eingang an: Executing step device automation
2021-12-14 07:18:06 ERROR (MainThread) [homeassistant] Error doing job: Fatal read error on socket transport
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError
2021-12-14 07:18:06 INFO (MainThread) [pydeconz.websocket] Reconnecting to deCONZ (core-deconz)
2021-12-14 07:18:06 INFO (MainThread) [homeassistant.components.automation.write_persistent_log] write persistent log error: Running automation actions
2021-12-14 07:18:06 INFO (MainThread) [homeassistant.components.automation.write_persistent_log] write persistent log error: Executing step call service
2021-12-14 07:18:06 ERROR (MainThread) [homeassistant] Error doing job: Fatal read error on socket transport
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError
2021-12-14 07:18:06 INFO (MainThread) [pydeconz.websocket] Reconnecting to deCONZ (core-deconz) failed, scheduling retry at an interval of 15 seconds
2021-12-14 07:18:06 INFO (MainThread) [homeassistant.components.automation.write_persistent_log] write persistent log error: Running automation actions
2021-12-14 07:18:06 INFO (MainThread) [homeassistant.components.automation.write_persistent_log] write persistent log error: Executing step call service

So I am a bit lost as to what could be the problem. Maybe deconz is failing or consuming too much memory? But wouldn’t this show in system RAM usage?

I hope someone can help.
Thanks in advance.

Daniel

2

The errors does not indicate that the memory runs low.
It looks more like a memory allocation error, which is when a program is reserving a part of the memory for storing some value.

3

Which can happen when the memory is full, right? I was suspecting that some process reaches a memory limit where it can no longer allocate new memory.

4

Yes, that is true, it it can also happen when the os can not find a suitable size of memory in one flow or if the memory allocated for a value is tried filled with a value that is actually larger than the allocated block.

5

Hi.

I’m having the same issue. Every couple of days HA stops responding and only a restart will fix this. I’m running latest version of HA Container on RPI 4 using docker-compose.

If I check the logs all the components will give Memory Error.

2022-01-30 21:16:28 ERROR (MainThread) [homeassistant] Error doing job: Fatal read error on socket transport
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError
2022-01-30 21:16:28 ERROR (MainThread) [custom_components.hacs] <Integration custom-components/sensor.sonarr_upcoming_media> 
2022-01-30 21:16:28 ERROR (MainThread) [custom_components.hacs] HacsTask<prosess_queue> failed: 
2022-01-30 21:16:28 ERROR (MainThread) [homeassistant] Error doing job: Fatal read error on socket transport
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError
2022-01-30 21:16:28 ERROR (MainThread) [homeassistant] Error doing job: Fatal read error on socket transport
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/asyncio/selector_events.py", line 856, in _read_ready__data_received
    data = self._sock.recv(self.max_size)
MemoryError

image

Any help whould be much appreciated.
Thanks !