Memory leak Home Assistant (2022)?

fresh installs of 9.3 are going straight to 4gb in first hour and crashing before I can even restore a backup

I have tried a good dozen fresh re-installs of 9.3 today thinking I might have to roll back to the 8’s and hope the cores are not updated
edit
crap I just missed out on doing a 8 install and now they updated it for 10 dev and made 9 stable :S
gave 10-dev a spin and it died before it even created a vm

tho I think most of the issue is google-drive-backup
as it seems to be installed first then tries doing a backup while the system is being restored …

proxmox with

bash -c "$(wget -qLO - https://github.com/tteck/Proxmox/raw/main/vm/haos-vm-v4.sh)"

log file from fresh install 46 mins before OOM

I think I have the same problem. My memory usage is very high. I have 8 gb of ram plus 4 gb of space.

When I was playing around with my bluetooth dongle trying to make it work with dbus my memory usage just exploded. My server nearly used all memory but I was able to kill “node” process eating nearly all memory. I don’t know what was node process is/was that was eating all the memory. I don’t have ONVIF integration.
This could be related to dbus but this is just my guess.
And this is how it looks when it start to eat memory. I can’t find out what this node proccess actually is?

I had hassos running on a pi4 with 8gb Ram that kept hanging until a switched it off and on again, migrated over to a VM on my QNAP and spotted on vnx an out of memory error.

Watching the hardware memory usage graph the memory would climb every minute until it maxed out and core restarted or I had to reboot.

Disabled just about everything when I found this post and I can confirm it was ONVIF. I disabled my 2 ONVIF cameras and the problem stopped immediately.

Home Assistant 2022.11.5
Supervisor 2022.11.2
Operating System 9.3
Frontend 20221108.0 - latest

1 Like

So guys, how long do we continue to discus the new posibly leaks under the topic name Memory leak from 2022.7 , which has been solved weeks ago?

I still have the issue - and none of the solutions in this thread have proven a root cause. Look for a better way to troubleshoot, but my HA instance is running out of memory about every 3 days…

I ended up moving all my databases to be external ,
seems to now take 3 days to reach 3Gb instead of minutes , still no idea what is using the Memory

The only solution I can offer is do a backup and restore using a old version

I will just not run any updates until the dev’s stop pointing fingers at someone else
I had noticed memory usage double 1 month before this massive memory leak

Just renamed the topic, any useful suggestions else?

This topic I hope helps people to solve their problems , aren”t ?

1 Like

I”ve been used the same behavior (restore old system), until found the possible reason of mine memory leaks somewhere on the web.

this is what my intance is doing since upgrade to 12.0/.1

i have homeassistant automation to reboot at 75% so that the system does not lock up

i5 intel nuc, 8gb ram

OS: HAOS 2023.01
Hardware: Raspberry Pi 3, external 128 GB SSD, 1GB ram
Software: no ONVIF integration involved, Z2m, VSCode, GDrive backup, HACS

Hello every body,

I am very happy to read this thread, cause I was read to drastically upgrade my hw, since I associated the memory leak to its age :smiley:

Unfortunately, I have the same issues of you. I don’t use the ONVIF integration and I don’t have any kind of camera installed.
I already tried to stop integrations to understand if some of them were making the ram usage grow so fast, but none of them seemed to cause the issue.
I don’t thing is related to DB usage, as I only track few entities (3 to 5 usually) for a timespan of 3 days and I regularly purge it.

I think I will go for the restart automation or for the hack suggested by @Neil_Brownlee, even if it is only a problem walkaround and does not actually solve its root cause.

I would like to understand the root cause, to definitely solve it.

2 Likes

Also can confirm that disabling ONVIF cameras solved the problem. Is there any other way to use ONVIF cameras without crashing everything?

I have the same problem. There is no ONVIF integration.

I seem to have at least identified the big memory usage jumps. Every time a Home Assistant Core update is installed (including making a backup) and the core is restarted, my memory usage more than doubles. The only way to solve it (for me) is to restart the host, after which the memory usage drops back to about 1,2GB. Since I’m running Home Assistant OS I do this from the CLI.

Who else is experiencing this?

Note that I’m also experiencing the very slow increase in memory usage as well (about 200MB in the last 3 days).
I’m running Proxmox (7.3) on a NUC (i3 with 32GB memory and an SSD) with a VM running HAOS and another VM running Deconz (on Ubuntu). And I’ve got the hack suggested by @Neil_Brownlee running as well (which kills rogue HA instances).


(I’ve got more screenshots, but I’m not allowed to add those as a new user)

1 Like

I only wouldn’t care about ram usage less then 50%, it’s just mine opinion.

I’m having a similar issue. Every time I run a backup on Home Assistant, memory usage climbs to ~90%. Did you ever figure out the issue? I’ve had this problem for several weeks and only recently connected it with a backup running.

Hi,

having same issue. Had Memory-Profiler running and saw that the “camera” integration (shipped with HassIO) was taking over 1GB of RAM in short time. So i’ve disabled the old camera integration and moved to the new integration.

But the leak is not gone completely. Now the memory takes longer to reach 100%.

Also i have upgraded from 2GB RAM to 8GB of RAM.

For this slow increasing of space-usage the profiler is not perfect to monitor this issue.

Current HPY:

.loader: _load_stat
.format: SetFormat
.timemade: 1678811197.900172
.count: 63871
.size: 8108936
.kindname:
.kindheader: Referrers by Kind (class / dict of class)
.numrows: 608
.r: 20268 1611740 dict (no owner)
.r: 7266 1138142 list
.r: 632 404480 sqlalchemy.orm.state.InstanceState
.r: 625 400000 homeassistant.components.recorder.db_schema.States
.r: 1710 337520 homeassistant.core.State
.r: 3118 323393 homeassistant.util.read_only_dict.ReadOnlyDict
.r: 2432 309792 function
.r: 1546 307834 homeassistant.core.State, homeassistant.util.read_only_dict.ReadOnlyDict
.r: 2803 306632 dict of sqlalchemy.orm.state.InstanceState
.r: 1092 280883 functools._lru_cache_wrapper
.r: 75 272208 dict of aiounifi.models.client.Client
.r: 3271 186431 tuple
.r: 2069 102732 dict of homeassistant.components.recorder.db_schema.States
.r: 1490 93261 set
.r: 474 90067 dict of homeassistant.helpers.template.RenderInfo
.r: 1079 85762 _asyncio.Task
.r: 37 82968 dict of aiounifi.models.device.Port, dict of aiounifi.models.port.Port, list
.r: 361 77632 homeassistant.core.Event
.r: 244 74470 homeassistant.components.mqtt.models.ReceiveMessage
.r: 584 74356 types.CellType
.r: 483 69472 homeassistant.core.HassJob
.r: 14 65744 dict of aiounifi.models.device.Device
.r: 16 60592 dict of aiounifi.models.client.Client, list
.r: 633 40592 weakref.ReferenceType
.r: 174 39440 pyunifiprotect.data.user.Permission
.r: 360 39265 hatasmota.discovery.TasmotaDiscoveryMsg
.r: 664 35088 dict (no owner), functools._lru_cache_wrapper
.r: 161 34776 dict of homeassistant.helpers.event.TrackStates
.r: 10 33274 aiohttp.http_websocket.WSMessage
.r: 131 30392 homeassistant.helpers.template.RenderInfo
.r: 64 29984 types.CoroutineType
.r: 625 29152 asyncio.events.TimerHandle
.r: 507 27872 homeassistant.core.Event, homeassistant.core.State
.r: 182 27798 types.FrameType
.r: 560 26880 dict of homeassistant.components.recorder.db_schema.States, types.MethodType
.r: 220 25205 dict of homeassistant.helpers.trace.TraceElement
.r: 50 23096 dict of asyncio.queues.Queue
.r: 244 20864 dict (no owner), dict of homeassistant.components.recorder.db_schema.States
.r: 15 20657 dict of aiohttp.http_websocket.WebSocketReader
.r: 182 20384 homeassistant.helpers.template.TemplateState, tuple
.r: 184 20197 dict (no owner), tuple
.r: 12 18752 pyunifiprotect.data.devices.Camera
.r: 250 18750 dict of homeassistant.components.recorder.db_schema.States, homeassistant.core.Context, homeassistant.util.read_only_dict.ReadOnlyDict
.r: 313 18497 collections.deque
.r: 229 18336 homeassistant.core.State, homeassistant.helpers.template.TemplateState, homeassistant.util.read_only_dict.ReadOnlyDict
.r: 26 16224 dict of asyncio.locks.Event
.r: 216 16200 homeassistant.core.Context, homeassistant.util.read_only_dict.ReadOnlyDict
.r: 560 15680 dict of homeassistant.components.recorder.db_schema.States, tuple
.r: 204 14721 homeassistant.core.Context
.r: 62 14416 dict of pyunifiprotect.data.user.Permission
.r: 93 13392 homeassistant.components.mqtt.subscription.EntitySubscription
.r: 56 12992 dict of hatasmota.status_sensor.TasmotaStatusSensor
.r: 144 12489 homeassistant.components.mqtt.discovery.MQTTDiscoveryPayload
.r: 31 12435 dict of homeassistant.helpers.aiohttp_client.HassClientResponse
.r: 18 12048 pyunifiprotect.data.devices.CameraChannel
.r: 223 11941 dict of homeassistant.components.recorder.db_schema.States, homeassistant.core.State, homeassistant.util.read_only_dict.ReadOnlyDict
.r: 11 11088 multidict._multidict.CIMultiDictProxy
.r: 3 10752 dict of sqlalchemy.orm.identity.WeakInstanceDict
.r: 113 10424 multidict._multidict.CIMultiDict

I notice the slow uptick in memory use on my system. I’m also seeing a very sharp uptick in memory usage (takes 2-3 minutes to go from 2 GB to just under 4 GB) when I run a backup. Does a similar thing happen on your system as well?

For me, leakage was noticed by days, not as you mentioned by hours.
I catch mine after migration to the new version of HA.
But curious, when I rollback the version of HA memory leak disappears.

Does anyone have an improvement with the new OS release?

The release claims to have:

  • [Improved data disk feature]
  • [Advanced memory management]

For anyone who’s curious, I think figured out my issue. For some reason (that may be particular to my setup). Proxmox is displaying the sum of my used and buffered/cached memory, not just my used memory.

What appears to be happening is this. Every time I run a back up, my used + buffered/cached memory essentially maxes out. After the backup, the buffer/cache isn’t purged, so Proxmox reports my memory being almost maxed out.

I verified this by doing the following:

  1. Using the directions on this page, I enabled root ssh access to Home Assistant OS.
  2. After running a back up, I ran free -h, a Linux command to output a summary of my memory usage. It reports the following:
# free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.2Gi        98Mi       4.0Mi       2.5Gi       2.6Gi
Swap:          1.3Gi       0.0Ki       1.3Gi
  1. I run the following command sync; echo 3 > /proc/sys/vm/drop_caches. This clears the pagecache, dentries, and inodes. See this page.
  2. Then, I run free -h again. It reports the following:
# free -h
               total        used        free      shared  buff/cache   available
Mem:           3.8Gi       1.2Gi       2.2Gi       4.0Mi       424Mi       2.6Gi
Swap:          1.3Gi       0.0Ki       1.3Gi

Here’s what Proxmox reports at the same time. (The dip is the moment I clear my buffer/caches.)

Why Proxmox displays things this way (or why my setup works this way) makes no sense to me. But at least I now understand why Proxmox reports such high memory usage but Home Assistant
seems fine.

Hope this helps someone else.