Runaway memory 11G+ in core - heeelp

I’m seeing run away memory in Home Assistant Core it seems and I can’t for the life of me work out why.

9/10 when HA starts the ‘homeassistant’ core docker uses <1G up until the point where it says “Home Assistant has Started” - as soon as it hits that the memory usage sky rockets, by around 1G every 10 seconds or so until it caps out. I’ve increased the VM ram to 12G and it’s still doing this.

Once it caps out, the python3 process gets killed and HA restarts.

Running latest HAOS on VM, with latest HA - been noticing random reboots for a month or so but since updating to the latest version it seems to be pretty much in a permanent reboot loop now due to hitting memory limits. Very occasionally it seems to last a few minutes after boot but always ends up leaking memory.

I’ve tried to eliminate as much as I can:

moved contents of custom_components out so they don’t load
cleaned up configuration.yaml
disabled all apps except file manager and mariadb (which HA uses as it’s DB)

ha core logs seem pretty normal for startup, other than errors about trying to load things where I’ve disabled the components by moving it etc.

How can I track down this memory leak?

Hi, you could start in safe mode to see whether that makes a difference.
Are you using custom integrations?

Rogue dashboards? There has been a lot of changes in this area recently. Do you have ones that have polling in a loop?

I have tried with various things disabled, including all automations and integration and safe mode.

I ended up setting the memory for the VM to 32G which seems to show it’s not a memory leak, just very heavy startup pressure from something. It seems it climbs to about 21G at peak and then settles down to 1G usage for the core docker.

So I’m guessing it’s some core is processing ‘post’-start but with enough resources, it manages to get back to something saner.

Still no idea why it seems to need so much memory for a startup peak though :confused: