Hoping someone can help me here without me having to reinstall everything.
So I’m running HA in a docker container on an RPi4 4GB with RaspberryPiOS Bullseye.
Everything has been running superbly until I installed AMP 4.13.3 straight after it came out. From then on things started to go badly. The UI would load but be unresponsive and then HA would crash. I somehow suspected AMP and removed the folder custom_components/alexa_media and rebooted.
This worked, in a way, as HA started and seemed ok, but from here I started to get nightly crashes. The OS itself would suddenly just die in the early hours and in the morning I’d notice that no automations were running, so would have to reboot the Pi.
Yesterday I bit the bullet and decided to install AMP 4.13.4, hoping that the fixes it contained would overwrite anything that still remained from AMP 4.13.3 in my HA install after my amateurish attempt to remove it. This seemed ok, but during the day I noticed free memory slowly reducing and HA memory increase to ~1.5gb.
This morning I woke up to no automations again, but this time the Pi itself hadn’t crashed. Bullseye decided to kill the HA process. Here’s a snippet of the journalctl log, apologies for the formatting it’s a straight copy/paste, not sure it will help anyone diagnose my issue: -
Oct 09 08:14:41 raspberrypi4 kernel: CPU: 0 PID: 1 Comm: systemd Tainted: G C 6.1.21-v8+ #1642
Oct 09 08:14:41 raspberrypi4 kernel: Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT)
Oct 09 08:14:41 raspberrypi4 kernel: Call trace:
Oct 09 08:14:41 raspberrypi4 kernel: dump_backtrace+0x120/0x130
Oct 09 08:14:41 raspberrypi4 kernel: show_stack+0x20/0x30
Oct 09 08:14:41 raspberrypi4 kernel: dump_stack_lvl+0x8c/0xb8
Oct 09 08:14:41 raspberrypi4 kernel: dump_stack+0x18/0x34
Oct 09 08:14:41 raspberrypi4 kernel: dump_header+0x4c/0x21c
Oct 09 08:14:41 raspberrypi4 kernel: oom_kill_process+0x2a8/0x2b0
Oct 09 08:14:41 raspberrypi4 kernel: out_of_memory+0xf0/0x350
Oct 09 08:14:41 raspberrypi4 kernel: __alloc_pages_slowpath.constprop.158+0x7d4/0xbc0
Oct 09 08:14:41 raspberrypi4 kernel: __alloc_pages+0x2a8/0x318
Oct 09 08:14:41 raspberrypi4 kernel: __folio_alloc+0x1c/0x28
Oct 09 08:14:41 raspberrypi4 kernel: __filemap_get_folio+0x214/0x5f0
Oct 09 08:14:41 raspberrypi4 kernel: filemap_fault+0x644/0xc40
Oct 09 08:14:41 raspberrypi4 kernel: __do_fault+0x44/0x170
Oct 09 08:14:41 raspberrypi4 kernel: __handle_mm_fault+0x508/0x9c0
Oct 09 08:14:41 raspberrypi4 kernel: handle_mm_fault+0x1c4/0x2e0
Oct 09 08:14:41 raspberrypi4 kernel: do_page_fault+0x1f4/0x480
Oct 09 08:14:41 raspberrypi4 kernel: do_translation_fault+0x54/0x70
Oct 09 08:14:41 raspberrypi4 kernel: do_mem_abort+0x48/0x98
Oct 09 08:14:41 raspberrypi4 kernel: el0_da+0x48/0xa0
Oct 09 08:14:41 raspberrypi4 kernel: el0t_64_sync_handler+0x68/0xc0
Oct 09 08:14:41 raspberrypi4 kernel: el0t_64_sync+0x18c/0x190
Oct 09 08:14:41 raspberrypi4 kernel: Mem-Info:
Oct 09 08:14:41 raspberrypi4 kernel: active_anon:476537 inactive_anon:382439 isolated_anon:0
Oct 09 08:14:41 raspberrypi4 kernel: Node 0 active_anon:1906148kB inactive_anon:1529756kB active_file:556kB inac
tive_file:1260kB unevictable:32kB isolated(anon):0kB isolated(file):0kB mapped:3840kB dirty:0kB writeback:0kB shmem:60k
B writeback_tmp:0kB kernel_stack:8992kB pagetables:24976kB sec_pagetables:0kB all_unreclaimable? no
Oct 09 08:14:41 raspberrypi4 kernel: DMA free:14772kB boost:0kB min:3104kB low:3880kB high:4656kB reserved_higha
tomic:0KB active_anon:438136kB inactive_anon:223084kB active_file:0kB inactive_file:508kB unevictable:0kB writepending:
0kB present:786432kB managed:702096kB mlocked:0kB bounce:0kB free_pcp:1064kB local_pcp:24kB free_cma:1568kB
Oct 09 08:14:41 raspberrypi4 kernel: lowmem_reserve[]: 0 2933 2933 2933
Oct 09 08:14:41 raspberrypi4 kernel: DMA32 free:13184kB boost:0kB min:13276kB low:16592kB high:19908kB reserved_
highatomic:0KB active_anon:1468012kB inactive_anon:1306672kB active_file:1336kB inactive_file:212kB unevictable:32kB wr
itepending:0kB present:3080192kB managed:3003436kB mlocked:32kB bounce:0kB free_pcp:2208kB local_pcp:492kB free_cma:0kB
Oct 09 08:14:41 raspberrypi4 kernel: lowmem_reserve[]: 0 0 0 0
Oct 09 08:14:41 raspberrypi4 kernel: DMA: 428*4kB (UMEC) 166*8kB (UMEC) 109*16kB (UMEC) 70*32kB (UMEC) 25*64kB (
UMEC) 12*128kB (UME) 7*256kB (UME) 2*512kB (UE) 2*1024kB (ME) 0*2048kB 0*4096kB = 15024kB
Oct 09 08:14:41 raspberrypi4 kernel: DMA32: 1199*4kB (UME) 478*8kB (UME) 182*16kB (UME) 68*32kB (UME) 3*64kB (U)
0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 13900kB
Oct 09 08:14:41 raspberrypi4 kernel: 8308 total pagecache pages
Oct 09 08:14:41 raspberrypi4 kernel: 7733 pages in swap cache
Oct 09 08:14:41 raspberrypi4 kernel: Free swap = 88kB
Oct 09 08:14:41 raspberrypi4 kernel: Total swap = 1048572kB
Oct 09 08:14:41 raspberrypi4 kernel: 966656 pages RAM
Oct 09 08:14:41 raspberrypi4 kernel: 0 pages HighMem/MovableOnly
Oct 09 08:14:41 raspberrypi4 kernel: 40273 pages reserved
Oct 09 08:14:41 raspberrypi4 kernel: 131072 pages cma reserved
Oct 09 08:14:41 raspberrypi4 kernel: Tasks state (memory values in pages):
Oct 09 08:14:41 raspberrypi4 kernel: [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
Oct 09 08:14:41 raspberrypi4 kernel: [ 175] 0 175 5567 18 57344 664 -1000 systemd-udevd
Oct 09 08:14:41 raspberrypi4 kernel: [ 434] 108 434 1795 74 40960 73 0 avahi-daemon
Oct 09 08:14:41 raspberrypi4 kernel: [ 435] 0 435 1673 24 49152 40 0 cron
Oct 09 08:14:41 raspberrypi4 kernel: [ 436] 104 436 2141 145 53248 160 -900 dbus-daemon
Oct 09 08:14:41 raspberrypi4 kernel: [ 438] 1000 438 131840 2442 323584 10854 0 deCONZ
Oct 09 08:14:41 raspberrypi4 kernel: [ 448] 0 448 58888 1 86016 584 0 polkitd
Oct 09 08:14:41 raspberrypi4 kernel: [ 464] 0 464 55325 113 69632 255 0 rsyslogd
Oct 09 08:14:41 raspberrypi4 kernel: [ 467] 0 467 3611 43 61440 165 0 systemd-logind
Oct 09 08:14:41 raspberrypi4 kernel: [ 469] 65534 469 1370 8 40960 48 0 thd
Oct 09 08:14:41 raspberrypi4 kernel: [ 478] 0 478 3391 15 53248 149 0 wpa_supplicant
Oct 09 08:14:41 raspberrypi4 kernel: [ 487] 108 487 1722 7 40960 71 0 avahi-daemon
Oct 09 08:14:41 raspberrypi4 kernel: [ 493] 0 493 211693 87 147456 71 0 nscd
Oct 09 08:14:41 raspberrypi4 kernel: [ 520] 0 520 795 36 32768 125 0 dhcpcd
Oct 09 08:14:41 raspberrypi4 kernel: [ 582] 0 582 23072 30 53248 10 0 rngd
Oct 09 08:14:41 raspberrypi4 kernel: [ 629] 118 629 32975 59 102400 620 0 freshclam
Oct 09 08:14:41 raspberrypi4 kernel: [ 633] 122 633 1504 15 40960 45 0 epmd
Oct 09 08:14:41 raspberrypi4 kernel: [ 637] 0 637 79236 804 229376 14528 0 meyectl
Oct 09 08:14:41 raspberrypi4 kernel: [ 638] 0 638 18394 78 126976 528 0 nmbd
Oct 09 08:14:41 raspberrypi4 kernel: [ 641] 0 641 3510 25 53248 211 0 wpa_supplicant
Oct 09 08:14:41 raspberrypi4 kernel: [ 647] 1001 647 312142 0 118784 827 0 syncthing
Oct 09 08:14:41 raspberrypi4 kernel: [ 650] 0 650 1836 1 49152 60 0 vncserver-x11-s
Oct 09 08:14:41 raspberrypi4 kernel: [ 652] 0 652 538921 2322 299008 1367 -999 containerd
Oct 09 08:14:41 raspberrypi4 kernel: [ 658] 121 658 3789 170 57344 173 0 mosquitto
Oct 09 08:14:41 raspberrypi4 kernel: [ 661] 0 661 1386 0 40960 27 0 agetty
Oct 09 08:14:41 raspberrypi4 kernel: [ 665] 0 665 61072 0 106496 438 0 ModemManager
Oct 09 08:14:41 raspberrypi4 kernel: [ 677] 0 677 7118 229 86016 367 0 vncserver-x11-c
Oct 09 08:14:41 raspberrypi4 kernel: [ 683] 0 683 3428 0 61440 234 -1000 sshd
Oct 09 08:14:41 raspberrypi4 kernel: [ 738] 115 738 2682 31 53248 136 0 chronyd
Oct 09 08:14:41 raspberrypi4 kernel: [ 761] 0 761 2208 1 40960 160 0 vncserver
Oct 09 08:14:41 raspberrypi4 kernel: [ 767] 0 767 27339 294 139264 2509 0 Xvnc-core
Oct 09 08:14:41 raspberrypi4 kernel: [ 796] 115 796 2649 0 53248 138 0 chronyd
Oct 09 08:14:41 raspberrypi4 kernel: [ 1210] 113 1210 60921 0 102400 948 0 colord
Oct 09 08:14:41 raspberrypi4 kernel: [ 1267] 0 1267 530 0 40960 34 0 hciattach
Oct 09 08:14:41 raspberrypi4 kernel: [ 1272] 0 1272 5977 0 73728 307 0 bluetoothd
Oct 09 08:14:41 raspberrypi4 kernel: [ 1873] 1001 1873 329583 9415 278528 548 0 syncthing
Oct 09 08:14:41 raspberrypi4 kernel: [ 1950] 0 1950 2026 1 49152 134 0 dbus-daemon
Oct 09 08:14:41 raspberrypi4 kernel: [ 1953] 0 1953 515 0 32768 26 0 xstartup
Oct 09 08:14:41 raspberrypi4 kernel: [ 2028] 0 2028 62663 1 110592 432 0 lxsession
Oct 09 08:14:41 raspberrypi4 kernel: [ 2055] 0 2055 675423 4110 434176 2140 -500 dockerd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2058] 0 2058 22471 56 159744 703 0 smbd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2113] 0 2113 1361 9 36864 105 0 ssh-agent
Oct 09 08:14:41 raspberrypi4 kernel: [ 2122] 0 2122 21887 23 151552 710 0 smbd-notifyd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2123] 0 2123 21885 25 143360 708 0 cleanupd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2150] 0 2150 22473 170 155648 612 0 lpqd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2212] 0 2212 10608 269 122880 6572 0 miniserv.pl
Oct 09 08:14:41 raspberrypi4 kernel: [ 2279] 0 2279 59434 1 90112 241 0 gvfsd
Oct 09 08:14:41 raspberrypi4 kernel: [ 2287] 0 2287 95143 0 90112 232 0 gvfsd-fuse
Oct 09 08:14:41 raspberrypi4 kernel: [ 2308] 0 2308 18089 0 114688 838 0 openbox
Oct 09 08:14:41 raspberrypi4 kernel: [ 2316] 0 2316 46815 0 118784 1109 0 lxpolkit
Oct 09 08:14:41 raspberrypi4 kernel: [ 2330] 0 2330 395226 0 536576 6391 0 pcmanfm
Oct 09 08:14:41 raspberrypi4 kernel: [ 2367] 0 2367 1361 0 40960 115 0 ssh-agent
Oct 09 08:14:41 raspberrypi4 kernel: [ 2392] 0 2392 76965 0 94208 161 0 agent
Oct 09 08:14:41 raspberrypi4 kernel: [ 2914] 0 2914 201089 21536 692224 3985 0 motion
Oct 09 08:14:41 raspberrypi4 kernel: [ 3274] 0 3274 23899 120 204800 15080 0 meyectl
Oct 09 08:14:41 raspberrypi4 kernel: [ 3651] 0 3651 175870 26 167936 655 0 gvfs-udisks2-vo
Oct 09 08:14:41 raspberrypi4 kernel: [ 3656] 0 3656 98433 143 122880 588 0 udisksd
Oct 09 08:14:41 raspberrypi4 kernel: [ 3699] 0 3699 39257 1 61440 180 0 menu-cached
Oct 09 08:14:41 raspberrypi4 kernel: [ 3764] 110 3764 38383 6 53248 54 0 rtkit-daemon
Oct 09 08:14:41 raspberrypi4 kernel: [ 3795] 0 3795 58404 1 77824 178 0 gvfs-mtp-volume
Oct 09 08:14:41 raspberrypi4 kernel: [ 3803] 0 3803 58445 1 81920 176 0 gvfs-goa-volume
Oct 09 08:14:41 raspberrypi4 kernel: [ 3814] 0 3814 58903 1 81920 222 0 gvfs-gphoto2-vo
Oct 09 08:14:41 raspberrypi4 kernel: [ 3820] 0 3820 77716 0 94208 235 0 gvfs-afc-volume
Oct 09 08:14:41 raspberrypi4 kernel: [ 3837] 0 3837 77973 20 94208 286 0 gvfsd-trash
Oct 09 08:14:41 raspberrypi4 kernel: [ 3865] 0 3865 40040 0 73728 153 0 gvfsd-metadata
Oct 09 08:14:41 raspberrypi4 kernel: [ 4005] 0 4005 175346 1 270336 2341 0 lxpanel
Oct 09 08:14:41 raspberrypi4 kernel: [ 4291] 0 4291 73010 30 143360 648 0 packagekitd
Oct 09 08:14:41 raspberrypi4 kernel: [ 26958] 0 26958 436162 0 135168 245 -500 docker-proxy
Oct 09 08:14:41 raspberrypi4 kernel: [ 26992] 0 26992 309443 697 98304 231 -998 containerd-shim
Oct 09 08:14:41 raspberrypi4 kernel: [ 26994] 0 26994 309443 748 102400 203 -998 containerd-shim
Oct 09 08:14:41 raspberrypi4 kernel: [ 27011] 0 27011 309379 722 98304 216 -998 containerd-shim
Oct 09 08:14:41 raspberrypi4 kernel: [ 27046] 0 27046 309443 777 102400 213 -998 containerd-shim
Oct 09 08:14:41 raspberrypi4 kernel: [ 27069] 0 27069 309443 702 98304 221 -998 containerd-shim
Oct 09 08:14:41 raspberrypi4 kernel: [ 27085] 0 27085 922 0 36864 64 0 bash
Oct 09 08:14:41 raspberrypi4 kernel: [ 27106] 0 27106 947 1 32768 63 0 bash
Oct 09 08:14:41 raspberrypi4 kernel: [ 27124] 0 27124 106 1 28672 11 0 s6-svscan
Oct 09 08:14:41 raspberrypi4 kernel: [ 27132] 0 27132 319422 2630 204800 221 0 portainer
Oct 09 08:14:41 raspberrypi4 kernel: [ 27142] 0 27142 922 0 36864 63 0 bash
Oct 09 08:14:41 raspberrypi4 kernel: [ 27167] 0 27167 107010 0 212992 5373 0 python3
Oct 09 08:14:41 raspberrypi4 kernel: [ 27170] 0 27170 25238 1 94208 3558 0 python3
Oct 09 08:14:41 raspberrypi4 kernel: [ 27192] 0 27192 409708 198 528384 27200 0 python3
Oct 09 08:14:41 raspberrypi4 kernel: [ 27202] 0 27202 52 1 28672 4 0 s6-supervise
Oct 09 08:14:41 raspberrypi4 kernel: [ 27203] 0 27203 48 0 24576 3 0 s6-linux-init-s
Oct 09 08:14:41 raspberrypi4 kernel: [ 27228] 0 27228 52 1 28672 4 0 s6-supervise
Oct 09 08:14:41 raspberrypi4 kernel: [ 27232] 0 27232 52 1 28672 4 0 s6-supervise
Oct 09 08:14:41 raspberrypi4 kernel: [ 27250] 0 27250 49 1 28672 5 0 s6-ipcserverd
Oct 09 08:14:41 raspberrypi4 kernel: [ 27300] 0 27300 52 1 28672 4 0 s6-supervise
Oct 09 08:14:41 raspberrypi4 kernel: [ 27302] 0 27302 2051855 797012 14110720 129652 0 python3
Oct 09 08:14:41 raspberrypi4 kernel: [ 27347] 0 27347 80903 0 253952 19809 0 piper
Oct 09 08:14:41 raspberrypi4 kernel: [ 235804] 0 235804 7661 163 77824 448 0 cupsd
Oct 09 08:14:41 raspberrypi4 kernel: [ 235814] 0 235814 44382 1 110592 1136 0 cups-browsed
Oct 09 08:14:41 raspberrypi4 kernel: [ 235943] 7 235943 4182 0 61440 209 0 dbus
Oct 09 08:14:41 raspberrypi4 kernel: [ 311660] 0 311660 2775 181 49152 0 0 vncserverui
Oct 09 08:14:41 raspberrypi4 kernel: [ 311661] 0 311661 10905 2844 122880 4281 0 /usr/share/webm
Oct 09 08:14:41 raspberrypi4 kernel: [ 311688] 0 311688 2851 133 57344 8 0 cron
Oct 09 08:14:41 raspberrypi4 kernel: [ 311696] 0 311696 3495 129 53248 0 -250 systemd-journal
Oct 09 08:14:41 raspberrypi4 kernel: [ 311698] 0 311698 2851 133 57344 8 0 cron
Oct 09 08:14:41 raspberrypi4 kernel: [ 311727] 0 311727 515 23 36864 0 0 sh
Oct 09 08:14:41 raspberrypi4 kernel: [ 311728] 0 311728 515 23 32768 0 0 sh
Oct 09 08:14:41 raspberrypi4 kernel: [ 311729] 0 311729 486 25 28672 0 0 run-parts
Oct 09 08:14:41 raspberrypi4 kernel: [ 311730] 0 311730 486 25 36864 0 0 run-parts
Oct 09 08:14:41 raspberrypi4 kernel: [ 311734] 0 311734 515 22 32768 0 0 fake-hwclock
Oct 09 08:14:41 raspberrypi4 kernel: [ 311735] 0 311735 22473 181 147456 599 0 lpqd
Oct 09 08:14:41 raspberrypi4 kernel: [ 311737] 0 311737 515 23 32768 0 0 apt-show-versio
Oct 09 08:14:41 raspberrypi4 kernel: [ 311738] 0 311738 515 24 32768 0 0 fake-hwclock
Oct 09 08:14:41 raspberrypi4 kernel: [ 311740] 0 311740 1660 25 36864 0 0 apt-show-versio
Oct 09 08:14:41 raspberrypi4 kernel: [ 311742] 0 311742 500 16 32768 0 0 date
Oct 09 08:14:41 raspberrypi4 kernel: [ 311746] 0 311746 22471 78 135168 681 0 smbd
Oct 09 08:14:41 raspberrypi4 kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=init.scope,mems_
allowed=0,global_oom,task_memcg=/,task=python3,pid=27302,uid=0
Oct 09 08:14:41 raspberrypi4-camera kernel: Out of memory: Killed process 27302 (python3) total-vm:8207420kB, anon-rss:
3188048kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:13780kB oom_score_adj:0
Oct 09 08:14:41 raspberrypi4 systemd[1]: systemd-journald.service: Failed with result 'timeout'.
Oct 09 08:14:41 raspberrypi4 systemd[1]: Failed to start Journal Service.
Oct 09 08:14:41 raspberrypi4 systemd[1]: systemd-journald.service: Consumed 15.872s CPU time.
Oct 09 08:14:41 raspberrypi4 systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at
3.
Oct 09 08:14:41 raspberrypi4 systemd[1]: Starting Daily apt upgrade and clean activities...
Oct 09 08:14:41 raspberrypi4 systemd[1]: Stopped Journal Service.
Oct 09 08:14:41 raspberrypi4 systemd[1]: systemd-journald.service: Consumed 15.872s CPU time.
Oct 09 08:14:41 raspberrypi4 systemd[1]: Starting Journal Service...
Oct 09 08:14:42 raspberrypi4 systemd[1]: Started Journal Service.
Oct 09 08:14:32 raspberrypi4 systemd-udevd[175]: Worker [311841] terminated by signal 9 (KILL)
Oct 09 08:14:32 raspberrypi4 systemd-udevd[175]: ttyACM0: Worker [311841] failed
Oct 09 08:01:48 raspberrypi4 rtkit-daemon[3764]: Demoting known real-time threads.
Oct 09 08:14:25 raspberrypi4 dhcpcd[520]: route socket overflowed - learning interface state
Oct 09 08:14:25 raspberrypi4 rtkit-daemon[3764]: Demoted 0 threads.
After the restart HA started normally, but memory usage in the HA python3 process is again creeping up gradually.
Does anyone have any suggestions? Many thanks
