Memory leak in 2021.7

andy73 · July 19, 2021, 1:19pm

Since updating to 2021.7 my PI 3 / HASS IO is quite unstable and it used to be very stable for the past months.

The only thing I can see is memory slowly coming up to 87% when the system became unresponsive and restarts itself after few or more minutes.
Minimal number of addons too, all up to date.

Anyone experiencing the same thing? Any hints on what can I do to try to pinpoint the leak or instability?

Thanks!

tom_l · July 19, 2021, 2:32pm

Start by disabling your custom integrations (not addons).

If that fixes it, put them back one by one (testing in-between) until you discover the culprit.

andy73 · July 19, 2021, 2:51pm

Good point. I have 3 integrations but those did not changed for some months… I’ll review them.

Marsman · July 20, 2021, 7:42am

I have the same experience, I also have the MariaDB, File Editor and Terminal and SSH addons active. And a number of custom integrations. But not able to pinpoint at the moment because I am not at home.

tmcb82 · July 21, 2021, 8:48pm

I’m having the same problem but I’m using a Pi 4 4GB in Docker on Ubuntu 21.04. When I look a htop I can tell it’s coming from Home Assistant specifically. Rebooting lower it but over a day it just creeps back up. Im only using one custom integration and I’ve tried disabling it but it stop the memory leak.

andy73 · July 26, 2021, 8:22am

Hi all,

I have run several tries and the one that could get me over the 2 days stability issues was running the recorder running with sql in memory (which of course has the disadvantage of all stats being gone at restart).

The following stabilized my environment

uninstalled Maria DB and let system run with default db
added scan_interval: 180 to platform: systemmonitor to reduce the number of database inserts related to the health of the system (default is 15 seconds so bumped up to 180 s)
change purge_keep_days: 7 (I could not even run with 2 in MariaDB)

System is quick and stable and the sql db is at 25Mb after 2 days.

Maybe is the MariaDB integration that de-stabilized my system or the recorder link with MariaDB.

I will post my status at the end of this week after the purge kept the database to 7 days of events.

Hope that helps.
memory and CPU for the past 24 h with 2 days uptime

andy73 · July 26, 2021, 1:23pm

system just rebooted itself.
I can see in the log
The system could not validate that the sqlite3 database at //config/home-assistant_v2.db was shutdown cleanly

Memory is down to 49% from the maximum recorded on 88.8 %…

Not sure what to do anymore…

tom_l · July 26, 2021, 1:25pm

Do you use the Ring integration?

andy73 · July 26, 2021, 1:29pm

yes I do have Ring integration!

This is the timestamp when it crashed.

21-07-26 14:19:08

21-07-26 14:17:38 INFO (MainThread) [supervisor.resolution.fixup] Starting system autofix at state CoreState.RUNNING
21-07-26 14:17:38 INFO (MainThread) [supervisor.resolution.fixup] System autofix complete
21-07-26 14:18:32 INFO (MainThread) [supervisor.updater] Fetching update data from https://version.home-assistant.io/stable.json
21-07-26 14:19:08 WARNING (MainThread) [supervisor.misc.tasks] Watchdog miss API response from Home Assistant
21-07-26 14:19:21 INFO (MainThread) [supervisor.resolution.evaluate] Starting system evaluation with state CoreState.RUNNING
21-07-26 14:19:21 INFO (MainThread) [supervisor.resolution.evaluate] System evaluation complete

123 · July 26, 2021, 1:31pm

Memory leak in ~~2017.7~~ 2021.7

andy73 · July 26, 2021, 1:32pm

disabled ring integration and rebooted system and will wait and report…

tom_l · July 26, 2021, 1:39pm

Thanks for pointing that out. Fixed.

andy73 · July 26, 2021, 1:45pm

do you mean was the ring integration fixed?

tom_l · July 26, 2021, 1:46pm

Ha. No, Taras just pointed out that your title and first post text had the wrong version number (2017.7 instead of 2021.7). I fixed it for you.

andy73 · July 26, 2021, 1:54pm

oh I see, yes thanks for that.

andy73 · July 26, 2021, 9:59pm

memory utilization is very good and stable since disabling the ring integration and reboot .

andy73 · July 27, 2021, 8:50am

yes, memory is very stable and flat with no Ring integration…

Marsman · August 3, 2021, 1:34pm

In that case it might be related to this issue Ring Memory Leak/OOM issue · Issue #53506 · home-assistant/core · GitHub

I also have the Ring integration active, so disabled it for now and see how it behaves.

andy73 · August 5, 2021, 3:41pm

it is stable without ring integration, watching the ticket on github but not much activity on it. Issue persists in latest 2021.8.1.

Maybe is related with latest change to reduce the video URL TTL from 45 min to 3 min.