Looking at 3 days of history causes >100% CPU usage (Ver 0.83.2)


#1

Hi Everyone,

I have noticed that when I go into the history and try to choose “3 Days” or “1 Week” and display it, the CPU usage goes through the roof. It seems that “1 Day” works, somewhat, it still consumes a lot of CPU but it’s navigable. When I just click on “History” and nothing else from the front page, I can watch the duration and date input fields get enabled and disabled without any input. It’s almost like a javascript but is stuck in an infinite loop, I believe this is causing the high CPU load. Switching to “3 Days” or “1 Week” (and choosing a day in the past so I see more data) causes a runaway-like condition and the CPU doesn’t seem to ever recover. I had to restart hass to retain control of the cores.

I have home assistant running on an Orange Pi Plus 2E. It’s 1.6GHz quad core and isn’t beefy but historically this wasn’t a problem. I’m running Home Assistant version 0.83.2. Any ideas?

Thanks,
-Greg


HA slow loading history
#2

Mine takes a while to load history but it does load in the end if I leave it for 1 or 2 mins.

How big is your database? If you have a lot of sensors and data then it can get pretty large.


#3

Mine is currently 647MB, it only stores data for 14 days. I left it running for about 5-10 minutes and it was still railing the CPU so I restarted hass. What kind of CPU/system are you running and what size is your DB?

I use the following recorder block to limit the history to 14 days.

# Keep two weeks of data
recorder:
  # Delete events and states older than 2 weeks
  purge_keep_days: 14

#4

Mine is currently 647MB, it only stores data for 14 days. I left it running for about 5-10 minutes and it was still railing the CPU so I restarted hass. What kind of CPU/system are you running and what size is your DB?

My CPU is probably slower than yours, I’m using a Raspberry Pi 3 with no overclocking.

I also find CPU goes to 100% when clicking history, but it does load eventually, and the CPU usage goes back to normal. My DB is hosted on another system running MariaDB 10. My DB is around 200Mb and I’ve got purge_keep_days set to 5 days.

What I did find to speed things up it to not record every entity from HA. I’m using the include and exclude properties to not log weather entities, for example.

Another thing you can do if you are having issues with the database is to just wipe it completely and start from scratch.

I don’t think your DB is really that big, some people have a DB over 1 GB, so I’m guessing there could be some corruption?

5-10 mins is way too long to view history, try looking at memory usage also, if memory usage is high that would explain why it’s taking so long or freezing.


#5

Thanks for the response. I noticed that I put the excludes list in my history: section instead of in the recorder: section. If I understand it, that means that I was recording everything but only displaying a subset of the sensors.

I’ve moved the excludes, deleted the db file and restarted home assistant. I’ll let it go a few weeks and try it again…


#6

So I’ve been watching the DB file and it’s much smaller than it used to be. So far 255MB. Anyway, after deleting the DB and letting it get recreated I have noticed that the problem of the javascript being stuck in a loop has not disappeared. I’ve captured a couple animated gifs in hopes that it shows more clearly what I’m talking about.

You’ll see in the gif below, when I click on History the CPU usage on the system jumps (as expected). Then the header area of HASS seems to keep submitting keeping the CPU crunching. This problem is exacerbated when looking at a larger amount of time, presumably because each time the fields gray out the data has to be fetched.

I also got a capture of the browser with the debug console enabled so we can see the network requests. You can see a new one get queued up every time the “Show entries for” and “Period” fields get grayed out. I am not doing anything with the keyboard when this happens by the way.

Any ideas?? I like using the history function but this makes it basically unusable for looking at longer periods of time.

Thanks,
-Greg


#7

Can you check the io utilization to understand where is the bottle neck?
I’m usually running
sar 1

Sar is one of the tools from sysstat, if you don’t have the sysstats you will need to install it by
sudo apt install sysstat


#8

Same “refresh loop” issue here, running 0.84.2. Cannot really narrow down which version brought that strange “behavior”, but I didn’t have this issue with 0.76.2.
I have done a gigantic amount of exclusion from recorder/history, but the problem didn’t disappear. Now that I see your post, I guess many people could experience the same issue.


#9

identical problem where… but i don’t need 3 days… 1d is enough…

imagem

version 0.84.6
rpi zero w


#10

I have the same problem. I’m running version 0.85.0 on a rpi3. I moved my DB to a external server running MariaDB, but it didn’t help.


#11

I’m not too surprised re. the Pi Zero W, I have to admit - I guess you’re not running hass.io on it, do you?

Entering SAR 1 in the CLI, what do the values for %iowait look like?

If not installed use: sudo apt-get install sysstat -y


#12

I tried this out and I do see that %iowait bumps up for a single instance seemingly every time the “Showing entries for” and “Period” fields get disabled but the values are less than 1 usually:

07:19:47 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
07:19:48 AM     all      3.26      0.00      1.75      0.00      0.00     94.99
07:19:49 AM     all      9.57      0.00      1.26      0.00      0.00     89.17
07:19:50 AM     all     25.19      0.00      0.76      0.00      0.00     74.06
07:19:51 AM     all      9.80      0.00      1.01      0.00      0.00     89.20
07:19:52 AM     all     14.65      0.00      2.53      0.51      0.00     82.32
07:19:53 AM     all     23.43      0.00      1.76      0.00      0.00     74.81
07:19:54 AM     all     33.59      0.00     11.36      0.25      0.00     54.80
07:19:55 AM     all     37.94      0.00      6.53      0.00      0.00     55.53
07:19:56 AM     all     30.15      0.00      5.53      0.00      0.00     64.32
07:19:57 AM     all     24.56      0.00      3.29      0.00      0.00     72.15
07:19:58 AM     all     23.81      0.00      1.50      0.00      0.00     74.69
07:19:59 AM     all     20.85      0.00      5.28      0.00      0.00     73.87
07:20:00 AM     all     18.56      0.00      8.42      0.00      0.00     73.02
07:20:01 AM     all     25.25      0.00      6.19      0.25      0.00     68.32
07:20:02 AM     all     25.06      0.00      2.51      0.25      0.00     72.18
07:20:03 AM     all     24.87      0.00      1.27      0.00      0.00     73.86
07:20:04 AM     all     23.99      0.00      2.78      1.01      0.00     72.22
07:20:05 AM     all     18.64      0.00      7.30      0.00      0.00     74.06
07:20:06 AM     all     20.35      0.00      5.03      0.00      0.00     74.62
07:20:07 AM     all     22.28      0.00      3.04      0.00      0.00     74.68
07:20:08 AM     all     23.62      0.00      1.76      0.00      0.00     74.62
07:20:09 AM     all     23.87      0.00      1.51      0.00      0.00     74.62
07:20:10 AM     all     19.65      0.00      6.05      0.00      0.00     74.31
07:20:11 AM     all     21.72      0.00      5.05      0.00      0.00     73.23
07:20:12 AM     all     24.37      0.00      1.26      0.00      0.00     74.37
07:20:13 AM     all     24.37      0.00      1.76      0.00      0.00     73.87
07:20:14 AM     all     27.57      0.00      3.26      0.00      0.00     69.17
07:20:15 AM     all     20.20      0.00      5.30      0.00      0.00     74.49
07:20:16 AM     all     21.46      0.00      5.81      0.00      0.00     72.73
07:20:17 AM     all     24.74      0.00      2.30      0.77      0.00     72.19
07:20:18 AM     all     25.32      0.00      1.77      0.00      0.00     72.91
07:20:19 AM     all     26.77      0.00      1.52      0.00      0.00     71.72
07:20:20 AM     all     45.57      0.00      5.32      0.00      0.00     49.11
07:20:21 AM     all     44.30      0.00      6.58      0.00      0.00     49.11
07:20:22 AM     all     47.72      0.00      3.05      0.00      0.00     49.24

Does this mean anything to you?


#13

Not that I am an expert here, but the %iowait value changes don’t seem to indicate a problem with the SD Card or wherever the Orange Pi stores the db.
Interesting, though, that the idle% value stays sign above 50% in most cases.
But I can’t really provide any further recommendations here, sorry.
Maybe somebody else sees something I don’t.


#14

Hate to bring up a thread that’s gone two months with no responses, especially if all i have to offer is a “Me too!” comment.

Docker on Synology:
HA 0.85.1 container

  • recorder includes/excludes configured
  • recorder logging/keeping 28 days
  • database ~2.8GB constant

MariaDB container hosting recorder database

When I try to view the history tab it will load after about 30 seconds but then the date selection and time range will gray out randomly. The data displayed will not change if I select a different date. The memory and CPU usage of the HA container easily spikes to 3GB and about 70% CPU usage. If I leave the history tab and go back to a states page the memory usage will very slowly start to come down and the CPU usage goes back to normal (ie. < 5%) immediately. I usually have to restart the HA container though to get memory back under control. Normally my HA container sits at about 250MB memory usage.

The recorder and history page used to be really fast as in loaded in under 10 seconds and you could select dates and date ranges and they would populate the data quickly. I don’t have an exact estimate when it stopped working properly but I’d put it around 0.80.


#15

Hello squirtbrnr, about a month ago I migrated my home assistant from an ARM version of Ubuntu to hass.io running on a raspberry pi. I no longer see this issue with the refreshing history/logbook pages. I’m not sure why but it was constantly a problem on the other installation. This morning after seeing your post I opened up the history page and let it sit for 5 minutes, no reloads (greying out of the date and period fields). I also did this with the logbook, also no reloads. I’m not sure where this leaves you but it’s another observation. If you have a raspberry pi, you may want to try hass.io. I like it better than manually installing and running home assistant because of it’s portableness - I converted to hass.io around 0.87 and now I’m running 0.89. I haven’t experienced this issue yet using hass.io.


#16

thanks for the update. I’m in the process of updating my HA install from 0.85.1 to at least 0.87.1 and hopefully higher. I held off on the switch to Lovelace so I’m a few versions behind and maybe one of those updates improves the history/recorder functionality. Unfortunately I don’t have a spare RPi at this time. My Synology is an Intel model and plenty fast, so I don’t think it’s a performance issue. I may try blowing away the recorder database as a troubleshooting step too.

EDIT: Just updated to Lovelace and 0.87.1 for starters and my problems almost went away. History loads much quicker and doesn’t grey out the date or range selection. Memory usage still increases a lot while viewing history, but not nearly as bad. I’m calling this “fixed” for me.