Memory leak after upgrade to core 2024.10.x

[agala68]

My rpi4 with 4 GB RAM, supervised ha, begunto frezze and restarting the ha container for memory issue after upgrade to 2024.9.3 or 2024.10.0 (not sure of that), here some info about the core dump:

Oct 03 19:07:29 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:07:29.319 WARNING (MainThread) [homeassistant.components.mqtt.discovery] Receiv>
Oct 03 19:10:11 homeassistant.agalanet.local containerd[823]: time=“2024-10-03T19:10:11.685160404+02:00” level=error msg=“add cg to OOM monitor” error="cgroup>
Oct 03 19:10:43 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:10:43.207 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new>
Oct 03 19:10:43 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:10:43.211 INFO (MainThread) [homeassistant.components.mqtt.entity] MQTT device >
Oct 03 19:10:43 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:10:43.231 WARNING (MainThread) [homeassistant.components.mqtt.discovery] Receiv>
Oct 03 19:19:02 homeassistant.agalanet.local addon_core_mosquitto[959]: 2024-10-03 19:19:02: Saving in-memory database to /data//mosquitto.db.
Oct 03 19:22:18 homeassistant.agalanet.local kernel: out_of_memory+0xe4/0x358
Oct 03 19:22:18 homeassistant.agalanet.local kernel: do_mem_abort+0x4c/0xa8
Oct 03 19:22:18 homeassistant.agalanet.local kernel: Mem-Info:
Oct 03 19:22:18 homeassistant.agalanet.local kernel: active_anon:598184 inactive_anon:257701 isolated_anon:0
active_file:4109 inactive_file:40 isolated_file:0
unevictable:2759 dirty:5 writeback:0
slab_reclaimable:55441 slab_unreclaimable:17857
mapped:120 shmem:2899 pagetables:10827
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:6932 free_pcp:124 free_cma:292
Oct 03 19:22:18 homeassistant.agalanet.local kernel: Node 0 active_anon:2392736kB inactive_anon:1030804kB active_file:16436kB inactive_file:160kB unevictable:>
Oct 03 19:22:18 homeassistant.agalanet.local kernel: lowmem_reserve: 0 2928 2928 2928
Oct 03 19:22:18 homeassistant.agalanet.local kernel: lowmem_reserve: 0 0 0 0
Oct 03 19:22:18 homeassistant.agalanet.local kernel: 0 pages HighMem/MovableOnly
Oct 03 19:22:18 homeassistant.agalanet.local kernel: Tasks state (memory values in pages):
Oct 03 19:22:18 homeassistant.agalanet.local kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=>
Oct 03 19:22:18 homeassistant.agalanet.local kernel: Out of memory: Killed process 17490 (python3) total-vm:13122504kB, anon-rss:2630984kB, file-rss:0kB, shme>
Oct 03 19:22:23 homeassistant.agalanet.local containerd[823]: time=“2024-10-03T19:22:23.273937328+02:00” level=error msg=“add cg to OOM monitor” error="cgroup>
Oct 03 19:22:57 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:22:57.247 INFO (MainThread) [homeassistant.components.mqtt.discovery] Found new>
Oct 03 19:22:57 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:22:57.250 INFO (MainThread) [homeassistant.components.mqtt.entity] MQTT device >
Oct 03 19:22:57 homeassistant.agalanet.local homeassistant[959]: 2024-10-03 19:22:57.269 WARNING (MainThread) [homeassistant.components.mqtt.discovery] Receiv>
Oct

wf-panel-pi invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP)

l kernel: hassio: port 13(vetha28fe9a) entered forwarding state
Oct 04 23:05:51 homeassistant.agalanet.local kernel: wf-panel-pi invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_ad>
Oct 04 23:05:51 homeassistant.agalanet.local kernel: CPU: 2 PID: 1966 Comm: wf-panel-pi Tainted: G C 6.6.28+rpt-rpi-v8 #1 Debian 1:6.6.28-1+r>
Oct 04 23:05:51 homeassistant.agalanet.local kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.4 (DT)
Oct 04 23:05:51 homeassistant.agalanet.local kernel: Call trace:
Oct 04 23:05:51 homeassistant.agalanet.local kernel: dump_backtrace+0xa0/0x100
Oct 04 23:05:51 homeassistant.agalanet.local kernel: show_stack+0x20/0x38
Oct 04 23:05:51 homeassistant.agalanet.local kernel: dump_stack_lvl+0x48/0x60
Oct 04 23:05:51 homeassistant.agalanet.local kernel: dump_stack+0x18/0x28
Oct 04 23:05:51 homeassistant.agalanet.local kernel: dump_header+0x48/0x230
Oct 04 23:05:51 homeassistant.agalanet.local kernel: oom_kill_process+0x140/0x2e8
Oct 04 23:05:51 homeassistant.agalanet.local kernel: out_of_memory+0xe4/0x358
Oct 04 23:05:51 homeassistant.agalanet.local kernel: __alloc_pages+0xaec/0xe68
Oct 04 23:05:51 homeassistant.agalanet.local kernel: __folio_alloc+0x1c/0x30
Oct 04 23:05:51 homeassistant.agalanet.local kernel: __filemap_get_folio+0xf0/0x310
Oct 04 23:05:51 homeassistant.agalanet.local kernel: filemap_fault+0x4a4/0x9e0
Oct 04 23:05:51 homeassistant.agalanet.local kernel: __do_fault+0x44/0x138
Oct 04 23:05:51 homeassistant.agalanet.local kernel: __handle_mm_fault+0x4e0/0xb18
Oct 04 23:05:51 homeassistant.agalanet.local kernel: handle_mm_fault+0x188/0x330
Oct 04 23:05:51 homeassistant.agalanet.local kernel: do_page_fault+0x274/0x4b8
Oct 04 23:05:51 homeassistant.agalanet.local kernel: do_translation_fault+0xa4/0xc0
Oct 04 23:05:51 homeassistant.agalanet.local kernel: do_mem_abort+0x4c/0xa8
Oct 04 23:05:51 homeassistant.agalanet.local kernel: el0_da+0x30/0x88
Oct 04 23:05:51 homeassistant.agalanet.local kernel: el0t_64_sync_handler+0xb8/0x130
Oct 04 23:05:51 homeassistant.agalanet.local kernel: el0t_64_sync+0x190/0x198
Oct 04 23:05:51 homeassistant.agalanet.local kernel: Mem-Info:
Oct 04 23:05:51 homeassistant.agalanet.local kernel: active_anon:300218 inactive_anon:593662 isolated_anon:0
active_file:50 inactive_file:2462 isolated_file:0
unevictable:2253 dirty:0 writeback:0
slab_reclaimable:18798 slab_unreclaimable:16341
mapped:151 shmem:2390 pagetables:10878
sec_pagetables:0 bounce:0
kernel_misc_reclaimable:0
free:6997 free_pcp:1463 free_cma:963

[Edwin_D]

By any chance also have and upgraded Alexa Media Player integration?

PSA: Alexa Media Player v4.13.3 breaks all versions of Home Assistant

as FYI…looks like the plan to block AMP in HA has been reverted according to this PR now that AMP v4.13.3 has been pulled @poudenes you asked about the block too so just FYI

[MaxK (Mark)]

2024.5+: Tracking down instability issues caused by integrations

If you find your system unstable because of a misbehaving integration, there are a few methods for identifying and recovering from the problem. It is recommended to use Home Assistant’s built-in debug mode in conjunction with asyncio debug mode.

Using Home Assistant's built-in debug mode in 2024.5.x or later (preferred)

If you are using version 2024.5.x or later, Home Assistant has a built-in debug mode: <a href="https://www.home-assistant.io/docs/configuration/troubleshooting/#han…

[agala68]

Yes, it may be, but I think some days before the core upgrade.
I have Alexa Media Player 4.13.2

[agala68]

Thank you, I will try

[Edwin_D]

If you have that version, that is definitely causing problems.

[agala68]

I have rolled back to 2024.9.3 and “it seems” stable again

[agala68]

So, you suggest me to upgrade it to 4.13.3 and upgrade back core to 2024.10.0?
HACS is not suggesting the AMP upgarde, why?

[Edwin_D]

The AMP version was pulled because of the problems. 2024.10.0 and 2024.10.1 work fine for me, but I do not use AMP.

[agala68]

now, after rolling back core to 2024.9.3 and activating sar, use of RAM is stabilized and no more oom since 21h, and for AMP:

Releases 196

v4.13.2Latest <-----
2 weeks ago

according to oom log, it also seem that leak could be related to RPi Reporter MQTT2HA Daemon, that btw I am using it since a lot of time