Memory leak with my configuration, any tip to find the culprit module?

Configuration
1

Dear all, I have a Home assistant core setup on a Proxmox LXC container.
Everything works just fine and I installed some integrations from HACS and setup some automations.

Unfortunately there is a very slow memory leak causing HA to eat something like 500MB every month.
When I start HA the container HA has about 350MB occupied, after a month 900MB, after another month 1400MB an so on until I restart the container, after that it starts again with 350MB.

I have updated HA to the latest version (2023.6) and I can still see the same bahaviour.

I think it’s related to a custom integration but I don’t know which one.

I tried profiling HA with py-spy but it only gives me the CPU usage, not the RAM one. No modules seems to use lot of CPU.

I then tried with memory_profiler but it shows me the total RAM used by HA, not which module is using it.

Any tip to troubleshoot the issue?

Thank you!

2

What do you have setup in your configuration.yaml?
What is the size of your log?
The logger & recorder sections can be set to exclude some items and set retention length.
Recoder
Logger

Depending on what integrations you have, the log can get very lengthy; before I made changes my logs were in the gigs.

If you have an extended delay to write to disk, this could eat of memory (possible)

3

Nine times out of ten it is a third party integration. Start by disabling them all and adding them back one by one until you identify a culprit.

4

Hi, I’m sure it’s a third party integration but the leak is so slow I cannot find it by disabling them…
Recorder e logger have default settings.

I installed profiler and started the “PROFILER.START_LOG_OBJECTS” service.

This is the output. Can you decode it?

Memory Growth: [('cell', 40017, 10), ('HassJob', 815, 3), ('DNSAddress', 16, 1), ('DNSText', 9, 1), ('DNSService', 9, 1), ('DNSPointer', 18, 1), ('_FilterableJob', 163, 1), ('LogEntry', 15, 1)]
Memory Growth: [('TimerHandle', 91, 10)]
Memory Growth: [('ReadOnlyDict', 1822, 6), ('TimerHandle', 96, 5), ('MemoryBIO', 4, 4), ('ReferenceType', 23447, 3), ('memoryview', 3, 2), ('managedbuffer', 3, 2), ('ResponseHandler', 4, 2), ('HttpResponseParser', 3, 2), ('StreamReader', 5, 2), ('TimerContext', 4, 2), ('SSLProtocol', 2, 2), ('SSLObject', 2, 2), ('_SSLSocket', 2, 2), ('_SSLProtocolTransport', 2, 2), ('ConnectionKey', 2, 1)]
Memory Growth: []
Memory Growth: [('States', 1, 1)]
5

While it is an issue, given that i restart my instance for many other reasons multiple times throughout a month, i would not be bothered :slight_smile:

I would focus on custom integrations with cameras (streaming)

6

why not use a proxmox VM instead of LXC?

7

It would be a wasteful use of resources to run Home Assistant Core in a virtual machine.

8

@tteck True! What a coincidence you are replying. I’m trying to get InfluxDB in LXC using your script, but I can’t it working. When using the command in shell nothing happens. Can you tell what i’m doing wrong or missing? I run other LXC containers without any issue in Proxmox.

9

It is highly probable that you are encountering DNS problems.

1 Like
10

OMG! That was it. PVE was running on old DNS setting. THANKS!

1 Like
11

Care to expand on that a little?

12

In Proxmox.

Datacenter > Nodes > PVE. Check DNS adress on the right. Make sure it’s the DNS address of your ISP.

1 Like
13

I second this. For me, it was the Wyoming Protocol integration that produced a memory leak. No matter how much RAM I allocated (up to 16GB), the memory always reached 95% utilization after a short time. By disabling the Wyoming Protocol integration, I was able to resolve the issue on my end. Now, my total memory consumption remains stable around ± 1GB.