Reduce RAM use on 1GB orangepi (slow CPU, high RAM)

[aik]

Running HA 2023.3.4 on OrangePiOne+ (aarch64, 1GB, 1gbit ethernet), Ubuntu2210, docker. Moscquito + z2m + NUT run elsewhere. HA locks up for 5+ minutes several times a day. Profiler says HA spends 77% of time in poll() (if I am reading it right, the picture below), memory profiler just makes everything worse and never seems to finish.

home-assistant_v2.db was 520MB, I removed it and -shm and -wal, restarted HA, this used to help but not anymore. home-assistant_v2.db-wal is 3.5MB after half an hour of HA running.

iotop says 6…11MB/s “current disk write” and it is none of the apps so I assume it is swapping. HA, swap and /tmp are on USB2.0 attached SSD, not SD card. With disabled swap, OOM would just kill HA every hour or two. Only about 750MB of swap is used, and this number is not growing, “available” in “free” says at least 40MB is always available. /me puzzled. /tmp in RAM is disabled.

About 40 zigbee devices + 5 tasmotas + 7 onvif devices + 1 Lyfx (light).

I was wondering if there is a way to limit RAM use by tweaking the recorder/logger/docker/etc?
Or this is it, a newer bigger Pi is needed?

CONTAINER ID   NAME            CPU %     MEM USAGE / LIMIT   MEM %     NET I/O   BLOCK I/O        PIDS
7e6986c89239   homeassistant   40.88%    307MiB / 985.1MiB   31.16%    0B / 0B   426MB / 99.1MB   33

[Tinkerer (aka DubhAd on GitHub)]

@aik

home-assistant_v2.db was 520MB, I removed it and -shm and -wal, restarted HA, this used to help but not anymore. home-assistant_v2.db-wal is 3.5MB after half an hour of HA running.

That has nothing to do with RAM usage, removing it isn’t going to help there.

That file is for your recorder database, it’s the write ahead log and deleting it isn’t a good thing to do.

You don’t provide the output of top or free to show actual RAM use, and what you do provide doesn’t show a problem.

[aik]

Fair enough.

top - 20:41:37 up 34 days,  7:22,  0 users,  load average: 1.24, 1.70, 2.39
Tasks: 161 total,   2 running, 159 sleeping,   0 stopped,   0 zombie
%Cpu(s): 25.3 us,  5.1 sy,  0.0 ni, 68.4 id,  1.3 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :    985.1 total,     36.1 free,    903.2 used,     45.8 buff/cache
MiB Swap:   3960.0 total,   3661.3 free,    298.7 used.     26.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                            
 147621 root      20   0 1054368 388876   1624 R  88.9  38.6   4:44.61 python3                                                                                            
      1 root      20   0  167468   2640   2340 S   0.0   0.3   5:04.91 systemd                                                                                            
      2 root      20   0       0      0      0 S   0.0   0.0   1:06.67 kthreadd                                                                                           
aik@hass:~$ 
aik@hass:~$ free
               total        used        free      shared  buff/cache   available
Mem:         1008704      918600       49476           4       40628       35956
Swap:        4055036      330232     3724804
aik@hass:~$ df
Filesystem     1K-blocks     Used Available Use% Mounted on
tmpfs             100872    10828     90044  11% /run
/dev/mmcblk0p1  30254472  8304512  21609264  28% /
tmpfs             504352        0    504352   0% /dev/shm
tmpfs               5120        4      5116   1% /run/lock
armbian-ramlog     51200     1976     49224   4% /var/log
/dev/sda3      102626232 10133100  87233868  11% /mnt/sda
/dev/sda4       15559884       36  14747648   1% /tmp
tmpfs             100868        0    100868   0% /run/user/1000

I just restarted the instance, hence lower-ish swap “used”.

[aik]

@Tinkerer

That file is for your recorder database

But some of the database is cached in RAM so it is not something totally unrelated to the problem, is not it?

[aik]

Disabling ONVIF seems helping quite a lot btw:

aik@hass:~/.homeassistant$ free
               total        used        free      shared  buff/cache   available
Mem:         1008704      777840      150284         416       80580      157532
Swap:        4055036      324004     3731032

but memory profiler cannot produce a report, forces swapping and nothing happens.

[Tinkerer (aka DubhAd on GitHub)]

@aik

But some of the database is cached in RAM so it is not something totally unrelated to the problem, is not it?

Sure, but that file isn’t RAM cache…

@aik

MiB Mem :    985.1 total,     36.1 free,    903.2 used,     45.8 buff/cache
MiB Swap:   3960.0 total,   3661.3 free,    298.7 used.     26.0 avail Mem 

The earlier output showed that HA is only responsible for 31% of that, and the top output says 38%. It’ll be interesting to see where the rest went… sort top by %MEM.

[aik]

There was tmux as well, consuming 150M. I stopped it, run memory profiler in HA and this time it hung so bad that I could not ssh to it anymore, had to reboot.

[Tinkerer (aka DubhAd on GitHub)]

That 150 MB doesn’t account for the other 50% of the RAM in use.

[aik]

True. After reboot, it is like this:

aik@hass:~$ free
               total        used        free      shared  buff/cache   available
Mem:         1008704      326668       46648        3744      635388      588180
Swap:        4055036       19152     4035884

without onvif.
What is the expected RAM use for “40 zigbee devices + 5 tasmotas + 7 onvif devices”?

[Tinkerer (aka DubhAd on GitHub)]

I doubt there is an “expected RAM use” documented like that. Even with that though, less than half your RAM is being used by HA (according to what you shared).

If you’re using stream and have Preload camera enabled (which may even be the default) then that’ll increase RAM and CPU usage. For 7 cameras potentially by quite a lot.

The kernel logs should tell you more about what’s going on, if it’s invoking the OOM killer for instance, but if swap is going up then that’s a bad sign.

[aik]

Without swap, OOM was invoked a lot. But 1yo HA did not have this problem. I was not streaming video buuuuut those static images were opening very quickly so this may be related.

[Tinkerer (aka DubhAd on GitHub)]

A year is a lot of change in HA itself, plus all the changes you’ve made to your configuration

I’d start by disabling the preload for streams and see if that helps.

[aik]

@Tinkerer

If you’re using stream and have Preload camera enabled (which may even be the default) then that’ll increase RAM and CPU usage. For 7 cameras potentially by quite a lot.

btw where should this option appear? I cannot find it in the UI, is it a camera’s option, or “stream” integration option(and then it is configuration.yml)?

upd: 1 onvif camera with the defaults consumes at least 70MB of RAM. So in my case 70*6 = 420MB which puts the setup on the edge (it is 6 onvifs + 1 android ipcamera, not 7 onvifs).

[Tinkerer (aka DubhAd on GitHub)]

It’s on the camera entity settings. Find the entity in Devtools → States, select the
next to it, and then the
in the top right. Then you’ll see:

[aik]

Well, not in my HA, it only lets me disable the entity or change the area but no “preload” anywhere :-/