Can anyone help me debug this issue? My swap use of my Pi 3b+ seems to keep climbing, until it gets to 100% when things start to slow down significantly. Here’s why my system lovelace looks like.
As you can see in the bottom right history graph, there’s no significant memory spikes that would make swap increase. One suspicious element in the history however is that the swap use spikes a bit around 7am each morning, which is pretty much when I wake up.
I have a bunch of aqara motion sensors in groups for each room, and when a group turns on it changes an input_select of that room to ‘Active’. Then if the same group turns off for a variable period of time (specified by an input_slider for each room), it sets the input_select to ‘Inactive’. I use the state of these input_selects to perform my automations such as lights and climate control. Do you think this setup would cause a lot of caching to be happening and hence my Swap space climbs when I start moving through the house in the morning?
I know that’s why I came here. Perhaps it’s something to do with the logger? My home-assistant_v2.db is huge currently at 1.9 GB. This morning I tried purging the recorder using the service recorder.purge and keep_days: 2 and repack: yes and my Swap has now jumped to 75%.
Not shown on your graphs is the buffers and cache memory. It is possible that for performance some large file is being memory mapped and large portions of it accessed. This will allocate memory for buffers and cache. If you have unused stuff in memory the OS will swap it to disk to make more room for currently active stuff, even if that is just cache/buffers.
The only way to know for sure is to log into the box and run “top” or “htop” and then hit SHIFT+M to sort by memory, placing the most intense users at the top.
IIRC the Pi 3b+ has 1Gb of RAM. So swapping when running large applications with 2Gb database is probably normal. We all love “the little PI that could”, but the reality is, they really can’t in a lot of cases.
I just checked mine:
22544 root 20 0 352.5m 189.8m 24.6m S 1.6 1.2 165:44.75 /usr/local/bin/python3 /usr/src/ homeassistant/homeassistant/main.py --config /config
The 352.5m (in VIRT column) is saying that HA has requested 352Mb of RAM. Of which 189.8Mb (RES) is actually allocated because it has been used. (Linux uses a sparse memory model). Finding out what files it has mapped is a little more tricky and when I checked in /proc I did not see anything but python libraries. Was expecting to find the SQLite DB, but didn’t, maybe I’m looking in the wrong place.
Also of interest is that it’s running from /usr/src which is terribly bad practice and is running as root, this might be a side-effect of how the docker image was setup. It might not be inherent to HA’s install. I’ll have to check.
Unfortunately I don’t see anything relevant to me in top, and htop isn’t present. This is probably as I’m running hass.io so everything is locked down. Here what I get:
I’m not aware of any ‘other’ ssh. I’ve only managed to access via SSH with the Hass.io addon “SSH Server” which defaults to the usual port 22. What is this ‘other’ you speak of (now I feel like I’m in an episode of lost!).
ooh nice find! Ok, I’ll have to wait till I get home as I clearly need physical access to get this working. Will report back asap. Thanks for your help so far
So I think I may have found the issue, but it may also just be anecdotal. Your comments about the memory size of the Pi made me think about how it must be managing the homeassistant_v2.db size being 1.9gb. the fact that HA took forever to load the history looked like a symptom, so just went ahead and deleted the DB then put much stricter controls on the recorder component (5 days, disabled the motion sensors).
24 hours later, and swap has been constant around 12%. Will give it a bit more time but I think the DB was definitely a part of it.
Ok I spoke too soon. This morning swap use had spiked at 7am to 50%.
But I think I found something. I run the Auto backup to Google Drive add-on. It runs at 4am, taking a snapshot of my configuration then uploads it to Google drive. The upload seems to finish around 7am. Coincidence? I’m thinking that as the snapshot size is around 300 mb, maybe the add-on loads it into memory while uploading it to Google drive, forcing the Pi to utilise swap?
Once swapped memory will remain swapped until something tries to access the pages or the pages are freed. If it was the backup/upload using cache memory, the symptoms would show the cache memory drop significantly after the process exits. That memory would then be free for stuff to be unswapped if it is needed, but would not immediately cause that to happen.
Using lots of swap is not, in itself an issue. There are often many Mb of pages of RAM that aren’t accessed very often or even again and moving them out of physical RAM can improve performance.
However your original post mentioned that it will continue to rise over some days until things start to get very slow. This is a symptom of memory leakage. It’s not an uncommon bug, a process allocates memory but does not free it when done, often throwing the reference to it away and leaving it allocated forever until that process is exited. This is not only an issue for languages like C/C++ with manual memory management, but exists in managed applications like Python or Java too.
If you restart services (without rebooting) does the swap usage drop? I expect you would need access to the actual host to do so.
Still best way to find out is to run “top” on the actual underlying host to see what is using most of the memory.
I hear you mate, but I seem to be having an issue getting it working with the public key method as described in your link. No doubt I’m doing something wrong. I used putty to create a public key, copied it into an authorized_keys file with ansi encoding and Unix line breaks, put it into the root of a fat USB stick, imported it from Hassio system, but no joy.
Unfortunately the top results above doesn’t really tell me much. Memory is well within the capabilities of the Pi, so can’t really figure out where the memory leak is. Tomorrow morning I’ll run it when the Swap seems to increase the most dramatically. Unless you have any other suggestions @paulcam ?
Sorry, I was about to reply with that suggestion, but didn’t hit send and then forgot… here is my reply
Interesting. Doesn’t look all that bad, asides the swap. Load factor is a little high, but the PI is a dual? core, so 2.0 would be fully loaded. Might be interesting to capture it when the swap is spiking.
It looks like you have enough memory to do a forced swap purge. If you want to try that.
As root:
swapoff -a
swapon -a
It might take a while and if you don’t have enough RAM it will error out of terminate some processes.
