Virtual Machine: Out of memory: Killed process 1017513 (python3)

jacoscar · May 20, 2021, 7:21am

I have Home Assistant running on a VirtualBox virtual machine on a 2015 i7 laptop and 6GB of RAM assigned

Yesterday night at 22.31 I received a notification that home assistant had restarted
It had been running smoothly for about 2 weeks now

I found this error on the virtual machine window, but I am not sure if it’s related as I don’'t usually look at it every day

The restart (or at least the setup after restart) happened at 22:31, I am not sure how long it went down for, but I guess at least 20 minutes as I have an automation that prints some quantities to a file every 10 minutes and it’s missing a few lines between 22:00 and 22:40 (highlighted in bold)

2021-05-19T20:40:00.023815+00:00 ;19-05-2021 21:40,6824.89,203.62,5944.39,-5.01,312.43
2021-05-19T20:50:00.064739+00:00 ;19-05-2021 21:50,7063.33,203.6,6245.19,-6.0,314.27
2021-05-19T21:00:00.110499+00:00 ;19-05-2021 22:00,7073.26,204.0,6185.69,-6.87,315.95
2021-05-19T21:40:00.020380+00:00 ;19-05-2021 22:40,6955.52,203.33,4717.95,-11.05,325.26
2021-05-19T21:50:00.020934+00:00 ;19-05-2021 22:50,7140.63,207.05,4724.98,-12.0,327.78
2021-05-19T22:00:00.011927+00:00 ;19-05-2021 23:00,5773.36,209.75,4848.92,-12.63,329.59
2021-05-19T22:10:00.008613+00:00 ;19-05-2021 23:10,1113.98,234.38,4848.92,-13.24,331.41
2021-05-19T22:20:00.016659+00:00 ;19-05-2021 23:20,1189.78,234.12,4848.92,-14.34,335.12

I also have these errors

File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 74, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 135, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 131, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 43, in get
    return await WebSocketHandler(request.app["hass"], request).async_handle()
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 141, in async_handle
    await wsock.prepare(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_ws.py", line 135, in prepare
    payload_writer = await super().prepare(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_response.py", line 378, in prepare
    return await self._start(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_response.py", line 386, in _start
    await self._write_headers()
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_response.py", line 458, in _write_headers
    await writer.write_headers(status_line, self._headers)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_writer.py", line 119, in write_headers
    self._write(buf)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_writer.py", line 67, in _write
    raise ConnectionResetError("Cannot write to closing transport")
ConnectionResetError: Cannot write to closing transport
2021-05-19 21:48:47 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 21:48:49 WARNING (MainThread) [homeassistant.components.automation.input_amp_limit] input amp limit: Already running
2021-05-19 21:48:56 WARNING (MainThread) [homeassistant.components.automation.input_amp_limit] input amp limit: Already running
2021-05-19 21:49:56 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140095721951584] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2021-05-19 21:50:01 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140095721951584] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2021-05-19 22:00:21 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:00:35 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:00:35 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.cpu_io_wait is taking over 10 seconds
2021-05-19 22:00:40 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:00:58 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:01:03 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:02:32 WARNING (MainThread) [homeassistant.helpers.entity] Updating state for sensor.energia_casa (<class 'homeassistant.components.integration.sensor.IntegrationSensor'>) took 12.915 seconds. Please create a bug report at https://github.com/home-assistant/core/issues?q=is%3Aopen+is%3Aissue+label%3A%22integration%3A+integration%22
2021-05-19 22:28:28 WARNING (Recorder) [homeassistant.components.recorder.util] Ended unfinished session (id=38 from 2021-05-17 20:37:57)
2021-05-19 22:28:36 ERROR (MainThread) [homeassistant.config] Invalid config for [automation]: [service] is an invalid option for [automation]. Check: automation->action->0->service. (See /config/configuration.yaml, line 9). 
2021-05-19 22:29:40 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform filter is taking over 10 seconds.
2021-05-19 22:31:11 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:31:13 WARNING (MainThread) [homeassistant.components.binary_sensor] Updating workday binary_sensor took longer than the scheduled update interval 0:00:30
2021-05-19 22:31:13 WARNING (MainThread) [homeassistant.components.binary_sensor] Updating ping binary_sensor took longer than the scheduled update interval 0:00:30
2021-05-19 22:31:13 WARNING (MainThread) [homeassistant.components.sensor] Updating tado sensor took longer than the scheduled update interval 0:00:30
2021-05-19 22:31:16 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-19 22:32:40 WARNING (MainThread) [homeassistant.components.script.control] Control: Already running
2021-05-19 22:40:13 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-19 22:40:13 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-19 23:48:55 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-19 23:48:55 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 01:05:46 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 01:05:46 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 02:25:28 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 02:25:28 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 04:31:40 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 04:31:41 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 06:33:42 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-20 06:41:04 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from vodafone.broadband (192.168.0.1). (Mozilla/5.0 (iPhone; CPU iPhone OS 14_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2021.5 (io.robbie.HomeAssistant; build:2021.150; iOS 14.5.1) Mobile/HomeAssistant, like Safari)
2021-05-20 06:48:16 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140619405075072] Client unable to keep up with pending messages. Stayed over 512 for 5 seconds
2021-05-20 07:13:07 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from vodafone.broadband (192.168.0.1). (Mozilla/5.0 (iPhone; CPU iPhone OS 14_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2021.5 (io.robbie.HomeAssistant; build:2021.150; iOS 14.5.1) Mobile/HomeAssistant, like Safari)
2021-05-20 07:13:07 WARNING (MainThread) [homeassistant.components.http.ban] Login attempt or request with invalid authentication from vodafone.broadband (192.168.0.1). (Mozilla/5.0 (iPhone; CPU iPhone OS 14_5_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Home Assistant/2021.5 (io.robbie.HomeAssistant; build:2021.150; iOS 14.5.1) Mobile/HomeAssistant, like Safari)
2021-05-20 07:19:44 ERROR (MainThread) [metno] https://aa015h6buqvih86i1.api.met.no/weatherapi/locationforecast/2.0/complete returned 503
2021-05-20 07:30:29 WARNING (MainThread) [homeassistant.components.sensor] Updating command_line sensor took longer than the scheduled update interval 0:00:03
2021-05-20 07:45:55 WARNING (Thread-3) [homeassistant.components.mqtt] Disconnected from MQTT server core-mosquitto:1883 (1)
2021-05-20 07:45:55 ERROR (MainThread) [homeassistant.components.template.template_entity] TemplateError('ZeroDivisionError: float division by zero') while processing template 'Template("{{(   states('sensor.prism_power_grid') | float / (states('sensor.prism_tensione_attuale') | float) ) | round(3) }}")' for attribute '_state' in entity 'sensor.prism_corrente_grid'
2021-05-20 07:48:29 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_protocol.py", line 422, in _handle_request
    resp = await self._request_handler(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_app.py", line 499, in _handle
    resp = await handler(request)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_middlewares.py", line 119, in impl
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 56, in security_filter_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 18, in request_context_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 74, in ban_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 135, in auth_middleware
    return await handler(request)
  File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 131, in handle
    result = await result
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 43, in get
    return await WebSocketHandler(request.app["hass"], request).async_handle()
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 233, in async_handle
    await self._writer_task
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/http.py", line 78, in _writer
    await self.wsock.send_str(message)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/web_ws.py", line 300, in send_str
    await self._writer.send(data, binary=False, compress=compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 687, in send
    await self._send_frame(message, WSMsgType.TEXT, compress)
  File "/usr/local/lib/python3.8/site-packages/aiohttp/http_websocket.py", line 656, in _send_frame
    await self.protocol._drain_helper()
  File "/usr/local/lib/python3.8/site-packages/aiohttp/base_protocol.py", line 87, in _drain_helper
    await waiter
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 935, in _write_ready
    n = self._sock.send(self._buffer)
BrokenPipeError: [Errno 32] Broken pipe
2021-05-20 07:57:32 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary
2021-05-20 07:57:32 WARNING (MainThread) [homeassistant.components.mqtt.mixins] JSON result was not a dictionary

shanem · November 19, 2021, 7:25pm

I have the same problem, did you ever figure out what was causing it?

bagg2 · November 26, 2021, 10:43am

Hi, I have the same problem.
HA running on vmware workstation (4GB ram) and “out of memory: killed process 880184 (python3)”

shanem · November 28, 2021, 12:08am

From what ive figured out so far, seems like the computer doesnt have enough RAM for what the VM is trying to use. Decreasing the amount of ram my VM has seems to have fixed the problem (for now). Its been working fine for a couple of days now.

nebbiadigiorno · August 16, 2022, 12:31pm

decreasing? you run out of ram and you have decreasing it? i have the same problem and i am in search for a solution.

shanem · August 16, 2022, 7:38pm

The problem came back after a while. I tried everything, and ended up just giving up on running Home Assistant in a VM. None of the solutions I found worked. Running HA in a VM made Z-Wave, ZigBee, and Bluetooth a lot more complicated anyway.

nebbiadigiorno · August 17, 2022, 11:20am

Where are you running home assistant now?
For me is a daily problem now.

jacoscar · August 17, 2022, 12:00pm

I am running it on a Raspberry PI 4; I have had random restarts since then but they seem to have stopped once I installed a cooling kit

shanem · August 17, 2022, 11:02pm

I’m using a cheap mini PC that I got for $180, similar to an Intel NUC. The performance seems to be pretty good, and these days it’s not much more than a Raspberry Pi 4, but it’s more powerful.

Spiritus · August 31, 2022, 12:10pm

The same is here with 2022.08. It crashes every 20 mins with 8Gb of RAM.

Frank_R · December 10, 2022, 10:46am

any solution to troubleshoot this.
for me this never was a problem until the 2022.12 release

I increased from 4GB to 6GM but I don;t think that will solve it.
There is obvious some memory leak here

svanhattum · January 16, 2023, 8:56pm

Is this under investigation by the HA developers ?
I also have this issue for quite a long time and still couldn’t find anything that might lead to a memory leak. Looks indeed like it is in HA itself since the new 2022.12 version.
I now have an automation created to call for a service to reboot every day at night.

thomasdb · January 17, 2023, 5:09pm

happens here too,since around then in time before i started noticing and have to reboot regularly i did swap the hyperv vm between two computers i thought it maybe hardware related and try to place it back, but now iam not sure

chrisCooper1983 · January 25, 2023, 7:37am

Hey guys,
I have actually the same problem. My ha instance is running for ~10 minutes, then i gets an error, see screenshot below:
My proxmox vm has 8vGB of RAM, 2xvCPU cores (host: AMD Ryzen 7 3700X) and 32 GB of vHDD
first i had 2022.12 version, today i upgraded to latest 2023.x version and error still appears.

What can i do to fix this error and get my HA back to life??

angainor · January 29, 2023, 7:55am

I have the same problem with HA 2022.12.6, Home Assistant OS 9.4 running inside a VBox VM. It seems that this could be some VirtualBox issue, as the HA itself is not using more memory. The day of startup:

HA
  VIRT    RES      %MEM
  1233.0m 428.6m   14.8   5:05.28 S python3 -m homeassistant --config /config                               

Host os
  VIRT    RES      %MEM
   3295452 600628  15,3   5:21.82 VBoxHeadless

and next day

HA
  VIRT    RES      %MEM
   1302.7m 434.9m   15.0  27:40.30 S python3 -m homeassistant --config /config                               

Host os
  VIRT    RES      %MEM
   3662040   2,0g   52,6 414:48.88 VBoxHeadless

At the same time free on the HA system shows plenty of free memory after I have cleared the caches:

# sync; echo 3 > /proc/sys/vm/drop_caches 
# free
               total        used        free      shared  buff/cache   available
Mem:         2969956      776272     1657556        4264      536128     2182948
Swap:         742488           0      742488

So a lot of free RAM, but VBox is for some reason not giving it back to the host. I will keep looking how this develops after I cleared the caches.

jnauber · June 30, 2023, 8:06am

Hi Stephan,
How do you created the automation.

All the best
Johnnes

lorcott · July 15, 2023, 3:22pm

same issue here

Geekamacker · October 23, 2023, 1:34am

What’s the max vRAM anyone has tried it with?

luckiesdavid · November 17, 2023, 8:01am

same issue here after restoring an Backup,
wich before runs in an VM (previous memory utilization in System->HW: 2GB RAM), but restored to an Raspberry Pi 4 4GB:
Backupsize:(3,8GB)

Out of memory: Killed process2641 (python3) total-vm: 4175392kB, anon-rss:1632764kB, file-rss:0kB, UID:0, pytables:5076kB oon_score_adj:-300

if i Restore it in a Proxmox VM with 10 GB of RAM i received:

now i can’t restore my Homeassistant on any hw…

does anyone have suggestions what i could try?

Mikehunt · January 4, 2024, 8:36am

Hi

I’m still facing this issue, HA restarting every 10 mins. Did anyone solve this?