Same for me, I had to Roll back to 0.102.2
I’m still staying at 0.100.2. Trying to upgrade to 0.102.3 today but fail to do so since HA crashes every time.
The log doesn’t provide much info on this
19-12-04 12:51:24 INFO (SyncWorker_5) [hassio.docker.homeassistant] Start homeassistant homeassistant/raspberrypi3-homeassistant with version 0.102.3
19-12-04 12:51:49 ERROR (MainThread) [hassio.homeassistant] Home Assistant has crashed!
19-12-04 12:51:49 CRITICAL (MainThread) [hassio.homeassistant] HomeAssistant update fails -> rollback!
19-12-04 12:51:49 INFO (MainThread) [hassio.homeassistant] Update Home Assistant to version 0.100.2
Anyone with same issue?
I’m seeing something similar after upgrading Hassio to 0.102.3 yesterday. My RAM and swap fill up completely and then the CPU goes to 100%. I’m on a NUC with plenty of resources.
I’m also unable to load the Home Assistant web UI (“504 Gateway Time Out”) and my logs show a long list of time out errors on what looks like every component.
Restarting containers manually doesn’t help, will try downgrading and rebooting the machine next and see if that does it.
edit: a reboot did fix it and so far so good continuing with 0.102.3, will see if it happens again. I couldn’t help but notice my uptime was 370+ days before doing this. Down to <2GB of RAM usage on the VM, where it was previously maxing out it’s 8GB.
For me, a reboot did not fix it. HomeAssistant was running for a couple of hours until the RAM was filled up again and the CPU load went up to 100% again. While being in this state, the HDD LED is lighting up constantly since the Linux is working heavily on the swap file.
While the CPU is at 100%, almost anything is working anymore. It is really hard to connect to the main system over SSH. It takes several minutes to connect and to execute commands. With a lot of patience, I was able to check the performance with the “top” command:
top - 09:51:56 up 14:55, 2 users, load average: 116.85, 135.05, 88.71
Tasks: 378 total, 2 running, 325 sleeping, 0 stopped, 0 zombie
%Cpu(s): 24.7 us, 31.0 sy, 0.0 ni, 0.0 id, 43.8 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem : 8035168 total, 113200 free, 7783184 used, 138784 buff/cache
KiB Swap: 2097148 total, 0 free, 2097148 used. 25556 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
9516 root 20 0 1332188 8892 0 S 100.6 0.1 27:35.67 grafana-server
59 root 20 0 0 0 0 R 99.7 0.0 274:51.01 kswapd0
1715 root 20 0 2386572 18528 0 S 8.0 0.2 23:34.93 dockerd
16466 root 20 0 8252480 6.866g 0 S 2.7 89.6 16:36.29 python3
6437 root 20 0 385560 3264 156 D 1.2 0.0 0:23.55 php7.2
3425 root 20 0 118424 8924 0 S 0.9 0.1 9:56.59 python3
26847 root 20 0 449932 5320 2176 D 0.9 0.1 8:38.62 Xorg
6465 root 20 0 109104 768 0 S 0.6 0.0 1:54.53 containerd-shim
The python3 task of the main HomeAssistant container is consuming almost 90% of the RAM. With that information, I rebooted the system and checked the performance of the HomeAssistant container by using the graphical interface of Portainer. This is how I got the picture I attached in the previous post. I can clearly see that the RAM is filling up abut 150MB every hour. I highly recommend you are observing the actual RAM usage of the HomeAssistant container before it crashes again. Maybe on the weekend, I will be able to test other versions between 0.101.2 and 0.102.3 in order to narrow it down.
1 Like
As a simple data point I am running on a nuc with ubuntu 18.04 with HA 0.102.3. I use docker but not hassio. I have no such symptoms or problems. The python3 process is using about 5% CPU and 3% MEM at a maximum.
Not saying you guys aren’t having problems, just trying to isolate them. Maybe it is hassio, maybe it is some component you have in common. Perhaps if everyone lists the custom components they have?
I’m running HA ON QNAP TS-453 Pro using ContainerStation and since 0.101 right through to 0.102.3 I have found CPU and Ram are being used up at very unusual levels.
I’m just hoping the next release will get this sorted.
Thank you for your inputs. I was actually not thinking about the custom components. I am using the following custom components:
- hacs
- gardena (https://github.com/wijnandtop/wt.pygardena)
- shelly (managed with hacs)
hacs & zigate for me
On the weekend, I was able to do further investigations. The problem is definitely not related to the custom components. I removed all the custom components and had still the memory leak. The memory seems to be Ok in the first hour. After about an hour, the memory increases in steps. The steps get larger every time. It is almost exponential increasing.
I have tested also different versions. 0.101.3 is still working while the error exists in 0.102.0. I have also tested the new beta 0.103.0b0. It looks like the issue is not resolved in the next release.
Since I am not a software engineer, I do not know how to further hunt down the issue. Any help is highly appreciated.
This issue seems to only affect the docker images as my venv install only uses ~21-22% of available RAM.
No it does NOT affect docker images.
Maybe the hassio docker images, but not the “straight” docker images.
thanks you sir yoloyolo
Using a 4GB RPi4 here, I usually see between 2-3% mem usage from HA most of the time.
If HA using more than 1GB you most likely have an issue somewhere.
My HASSIO is using 4Gb …
My install is a docker image & not Hassio Nick.
You’ll need to list all the integrations you are using as well in your bug report.
Have you gone through disabling integrations one at a time to try and narrow it down?
OK lets narrow this down. What architecture is the qnap?
Well I’m not exactly sure what you mean Nick but its CPU specs are: Intel® Celeron® 2.0GHz quad-core processor (burst up to 2.42GHz) d mine has 8GB Ram.
Full specs are detailed here: https://www.qnap.com/en-au/product/ts-453%20pro/specs/hardware
Yeah so a fairly standard amd64 architecture. I assume it is running the same docker image as I run on a NUC.