Hmm. Thanks for the info. I’m pretty sure my problem is the same as yours. I’m not sure why they deleted my comments on your Github issue.
Today I switched back from MySQL to SQLite, but I’m still seeing the issue.
I might try restoring an older backup next.
1 Like
I’m still having problems with memory use steadily increasing. I have to restart Home Assistant every day or two to get the memory and the load back down.
If you setup some code to log every few seconds with objgraph, most leaks can be discovered:
_LOGGER.debug("Most common types: %s", objgraph.most_common_types(limit=25))
_LOGGER.debug("Growth: %s", objgraph.growth(limit=25))
I’ve just seen this. Thanks. I’ll have to give that a go.
Has anyone else here resolved their memory leak issues?
Starting in 0.118 (releasing today)
You can use the profiler.start_log_objects service to log the growth of new objects in memory.
Then you can use the profiler.dump_log_objects service to dump the ones to the log that are unexpectedly growing.
Finally once you have found the leak, call the profiler.stop_log_objects service to disable logging.
1 Like
I get “Failed to call service profiler/start_log_objects. Service not found.” when I try to call that service from Deleloper Tools. Is there a different way I should be doing it?
Which version are you using?
Can you elaborate on how to use the dump log objects. I get required key not provided? I looked at the docs and it says it’s required to put a type but I’m just trying to figure out what would be useful to put to track a memory leak issue I’m having.
Once you have found the type of object that is leaking you need to provide the type to the service call.
For example
type: State
I didn’t manage to make sense of the profiler, but it seems to me that the problem is related to the front end. If I click through click around the UI a lot, particularly into the sensor graphs, then Home Assistant eventually runs out of memory. If I don’t view the detail of graphs it can surivive without having to be restarted daily. I guess the data from those sensors is building up in memory or something.
Hi,
I am really trying to make sense of this ;-). I do get the profiler to put things in the log, I do understand (grammatically) what you say about identifying the type of object and then having to dump that type of object, finding out the type of object however is a bit misty to me.
If, for instance, I have the following popping up:
"[homeassistant.components.profiler] Memory Growth: [('function', 91217, 2), ('DNSService', 28, 1)]"
in the logfile, is the type then “function”, “DNSService” or something else?
I am trying to get my finger behind something which is leaking memory like crazy resulting in the swap file filling up quite rapidly and HA getting very sluggish. (running HASSOS 5.13 on Odroid-N2+ (“blue”)). As an interim solution I have set up an automation which just reboots the host once swap gets to 40% or so (at which point sluggishness starts to appear).
Any insights, directions etc. much appreciated.
Cheers,
F.
1 Like
function is a generic python function
DNSService is likely a zeroconf object
OK thanks. I will keep digging and see what I find.
Also check your /developer-tools/event url to see if there are a build up of listeners on the right side of the screen under Available Events
Most systems will have less than 100 listeners per event type. If you have one that is larger, keep watching it over time to see if it grows.
OK. Will dig into that one as well. For now it seems as if running a snapshot (I run a snapshot every night) triggers the growth of the swapfile. Interesting enough updating one of the add-ons always brings down the amount of memory swapped out. The sluggishness seems to be gone and I think I was able to pinpoint a “cache” setting in one of the graph cards I use causing it.
update; peeked at the listeners as you suggested and the highest number of listeners is 33 for the “home assistant stop”; the rest is all single digit.
Are you using today’s release ?
Updating as we speak.
Hi,
It’s me again… Attached is a longer term overview I made of the memory usage and swap usage.
The green line is swap usage in %, the yellow-ish line is memory usage in %. I run an automatic snapshot each night around 1AM and every time a snapshot is created, swap usage goes up and stays up. Swap is released significantly anytime I update an add-on and the respective container gets recreated. Memory use is quite stable around 42% (around 2Gb still free). Swap keeps climbing up with nice steps until it sits at around 45% after which the increase is minimal. This (sort-of) makes sense as swappiness is set at 60 (standard value), which makes a lot of sense on systems with small memory (1Gb and such), but a lower swappiness would not hurt when there is quite some ram to spare. Then again; I believe that the internal storage on an Odroid N2+ is a lot more resilient than a memory card in a raspi ;-). On my main workstation (32Gb ram) swappiness is set at 10 or so and hardly ever do I see any swap being used, not even with memory hogs open. Unless sluggishness starts to rear its head again I can live with the current situation.
Is there one specific object type that keeps growing in memory?