I do not understand how to read it to find, where the memory is allocated.
I started directly after a system reboot (device, not only core). There it was around 45-50% (as always in the past). I left the system alone with profiler on. At the end I clicked through all pages and areas in UI of HA to see changes. Most time it stayed around 55%. I ended with 80-90% (have a gut feeling when it jumped rapitely, but have to double-check) before I stopped the profile log.
Can you or one see from the logs, where all the additional memory is taken?
It’s kind of a pain in the ass to read, but each memory growth is a list of the following: name of growth, total current size, increase in growth. The data is sorted by what had the largest growth. So, we are simply looking at the first or second elements after the phrase Memory Growth that are large. The first one is when it starts, you can ignore that one. It tells you everything that’s loaded at start thought.
At 12:21:41, there’s a huge jump in a lot of data. Same thing at 14:43.11. I’m not seeing anything that stands out. Maybe Nick has some incite.
Nothing stands out in the data. Usually when there is an obvious leak each cycle will show more of the same object types over and over. 'set', 120957, 2 is a bit high but thats not specific cause for concern.
I’d do an profiler.start next.
Sadly this one looks like one of the harder ones to track down. It may require figuring out what the specific event is that causes the memory use to increase so it can be replicated.
Here we go. VSC. The addon takes now more than more memory, never releases it back, add even more on re-opening it. Only add-on restart brings the memory back. The whole system gets otherwise more and more unresponsive, most probably because of memory swapping, etc. Including automatic system-hard-reboots, …
Here a start of the VSC UI. 40%. And Core (blue) and Supervisor (green) got swapped to disk I think. Orange is total up to over 95% what is ofc not healthy for the stability.
I seems to be only vscod-addon ans surprisingly it seems a known problem here, here or here.
Question, because I don’t see any reaction there at all: Is Frenck aware of it or perhaps on vacation or someone else took over?
I know, OT, if this thread is only about integrations causing problems. But wanted to let you and others know, how are seeins similiar and can stop digging with profiler. Thanks for your help anyways, Petro and bdraco.
That seems like its a problem with VSCode plugins or VSCode itself. I usually have to close it and restart it a few times a day when using it locally depending on how much I’ve used it. I’d guess its the same for running it on a server. Its probably something that has to be fixed upstream and likely nothing that can be done with the addon itself.
Could be, but I’m not sure. Because VSC addon was already uptodate the last weeks. The problem only started (at least here) with updating Core and HAOS. So perhaps a new memory handling of HAOS/Supervisor as well.
I noticed an error in the logs occurring before the memory began to increase, specifically an issue with the assist pipeline. I recalled that I had previously set up Piper but had disabled it, leaving Atom Echo active. After I disabled it, the RAM consumption ceased to grow. I will continue to investigate to determine if this is the root cause or merely a coincidence.
it definitely is the vscode addon, specifically the custom ha extension in vscode traversing your files. I keep the vscode addon stopped unless I’m using it. It’s been like this for a long time too, not just recently.
Would it be possible to get some help understanding my profiler logs, as I’m trying to hunt down a big CPU increase [avg 30% on a RPi4, from ~10% beforehand] which has been ongoing for some weeks now (see this post for background)?
I’ve opened the Callgrind file in KCacheGrind but am not sure what to look for/at, and if I try to run gprof2dot -f callgrind -e 0.05 -n 0.25 callgrind.out.1733847020831614 -o callgrind.dot as per the Profiler page, I run into AssertionError (assert abs(call_ratio*total - partial) <= 0.001*call_ratio*total)…
The callgrind log can be found here … the cprof file is a binary though so I’m not sure which would be the preferred way to share it?
Thank you for the reply. Unfortunately, py-spy fails to run and outputs the error
$ ./py-spy top --pid 67
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: ParseIntError { kind: PosOverflow }', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/proc-maps-0.2.1/src/linux_maps.rs:81:65
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: receiving on a closed channel
I followed the guide found here, the second last post contains another person running into the same problem a while ago, but no solution unfortunately.
Hello, I am investigating a memory leak on my own system right now. Could you tell me where you got the above list from? (the one where it states that you have sensor (6637 topics). Thank you
I’ve been noticing RAM usage increase on my system as well. I have ran the diagnostics recommended in the Tracking down a memory leak of python objects section of this post and ran the profiler over a period of time (longer than 1 hour). Than stopped the profiler. Here is the result from the logs: dpaste: GREVZR9LF
However, I am unable to determine what might lead to the memory usage incrase based on this data. This output does not tell me anything. I don’t know where to look anymore.
Good luck finding your memory leak! I don’t have any pointers unfortunately, more than to mention that my (cpu increase, not memory though) was related to a LilyGo32 LoRa USB device which for some reason added all these MQTT entries. I deleted them all (through MQTT Explorer) and the cpu usage decreases significally. I have since stopped using this device.
