Suddenly InfluxDB Became Resource Hog

chairstacker · August 29, 2020, 2:01am

My RPi4 with a Supervised Install just got very sluggish so I looked in Portainer and saw that CPU Usage of the InfluxDB Container was North of 150%.

I updated to the latest add-on version but it didn’t change a thing.
From what I can see with htop there are still multiple influxdb processes running that eat up way more CPU% than they should.

These are the log entries if somebody understands what they mean and can give me any direction how I can fix this - other than just turning the add-on off permanently:

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.,
[s6-init] ensuring user provided files have correct perms...exited 0.,
[fix-attrs.d] applying ownership & permissions fixes...,
[fix-attrs.d] done.,
[cont-init.d] executing container initialization scripts...,
[cont-init.d] 00-banner.sh: executing... ,
,
-----------------------------------------------------------,
 Add-on: InfluxDB,
 Scalable datastore for metrics, events, and real-time analytics,
-----------------------------------------------------------,
 Add-on version: 3.7.5,
 You are running the latest version of this add-on.,
 System: Raspbian GNU/Linux 10 (buster)  (armv7 / raspberrypi4),
 Home Assistant Core: 0.114.4,
 Home Assistant Supervisor: 235,
-----------------------------------------------------------,
 Please, share the above information when looking for help,
 or support in, e.g., GitHub, forums or the Discord chat.,
-----------------------------------------------------------,
[cont-init.d] 00-banner.sh: exited 0.,
[cont-init.d] 01-log-level.sh: executing... ,
[cont-init.d] 01-log-level.sh: exited 0.,
[cont-init.d] create-users.sh: executing... ,
[cont-init.d] create-users.sh: exited 0.,
[cont-init.d] influxdb.sh: executing... ,
[cont-init.d] influxdb.sh: exited 0.,
[cont-init.d] kapacitor.sh: executing... ,
[cont-init.d] kapacitor.sh: exited 0.,
[cont-init.d] nginx.sh: executing... ,
[cont-init.d] nginx.sh: exited 0.,
[cont-init.d] done.,
[services.d] starting services,
[services.d] done.,
[18:51:56] INFO: Chronograf is waiting until InfluxDB is available...,
[18:51:56] INFO: Kapacitor is waiting until InfluxDB is available...,
[18:51:56] INFO: Starting the InfluxDB...,
[httpd] 127.0.0.1 - chronograf [28/Aug/2020:18:52:21 -0700] "GET /ping HTTP/1.1" 204 0 "-" "InfluxDBShell/1.8.2" 473cb933-e99a-11ea-8001-0242ac1e2102 254,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:21 -0700] "GET /ping HTTP/1.1" 204 0 "-" "InfluxDBShell/1.8.2" 473ebbbb-e99a-11ea-8003-0242ac1e2102 60,
[httpd] 127.0.0.1 - chronograf [28/Aug/2020:18:52:21 -0700] "POST /query?chunked=true&db=&epoch=ns&q=SHOW+DATABASES HTTP/1.1" 200 121 "-" "InfluxDBShell/1.8.2" 473cfed1-e99a-11ea-8002-0242ac1e2102 156245,
[18:52:21] INFO: Starting Chronograf...,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:21 -0700] "POST /query?chunked=true&db=&epoch=ns&q=SHOW+DATABASES HTTP/1.1" 200 121 "-" "InfluxDBShell/1.8.2" 473ee6bd-e99a-11ea-8004-0242ac1e2102 166259,
[18:52:21] INFO: Starting the Kapacitor,
,
'##:::'##::::'###::::'########:::::'###:::::'######::'####:'########::'#######::'########::,
 ##::'##::::'## ##::: ##.... ##:::'## ##:::'##... ##:. ##::... ##..::'##.... ##: ##.... ##:,
 ##:'##::::'##:. ##:: ##:::: ##::'##:. ##:: ##:::..::: ##::::: ##:::: ##:::: ##: ##:::: ##:,
 #####::::'##:::. ##: ########::'##:::. ##: ##:::::::: ##::::: ##:::: ##:::: ##: ########::,
 ##. ##::: #########: ##.....::: #########: ##:::::::: ##::::: ##:::: ##:::: ##: ##.. ##:::,
 ##:. ##:: ##.... ##: ##:::::::: ##.... ##: ##::: ##:: ##::::: ##:::: ##:::: ##: ##::. ##::,
 ##::. ##: ##:::: ##: ##:::::::: ##:::: ##:. ######::'####:::: ##::::. #######:: ##:::. ##:,
..::::..::..:::::..::..:::::::::..:::::..:::......:::....:::::..::::::.......:::..:::::..::,
,
2020/08/28 18:52:22 Using configuration at: /etc/kapacitor/kapacitor.conf,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:22 -0700] "GET /ping HTTP/1.1" 204 0 "-" "KapacitorInfluxDBClient" 478da706-e99a-11ea-8005-0242ac1e2102 82,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:22 -0700] "POST /query?db=&q=SHOW+DATABASES HTTP/1.1" 200 121 "-" "KapacitorInfluxDBClient" 478e0f96-e99a-11ea-8006-0242ac1e2102 3912,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+_internal HTTP/1.1" 200 153 "-" "KapacitorInfluxDBClient" 478f05a1-e99a-11ea-8007-0242ac1e2102 2559,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+homeassistant HTTP/1.1" 200 149 "-" "KapacitorInfluxDBClient" 478f9d6c-e99a-11ea-8008-0242ac1e2102 2950,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:52:22 -0700] "POST /query?db=&q=SHOW+SUBSCRIPTIONS HTTP/1.1" 200 225 "-" "KapacitorInfluxDBClient" 479037d8-e99a-11ea-8009-0242ac1e2102 2178,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 48cfd7ee-e99a-11ea-800a-0242ac1e2102 202373,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 48fb5b45-e99a-11ea-800b-0242ac1e2102 17467,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 4905d252-e99a-11ea-800c-0242ac1e2102 14726,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 490e1ede-e99a-11ea-800d-0242ac1e2102 79973,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 4920820f-e99a-11ea-800e-0242ac1e2102 12535,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:24 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 49288188-e99a-11ea-800f-0242ac1e2102 16162,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:25 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 4930fc66-e99a-11ea-8010-0242ac1e2102 11626,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:27 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 4a9845f9-e99a-11ea-8011-0242ac1e2102 30654,
time="2020-08-28T18:52:40-07:00" level=info msg="Reporting usage stats" component=usage freq=24h reporting_addr="https://usage.influxdata.com" stats="os,arch,version,cluster_id,uptime",
time="2020-08-28T18:52:40-07:00" level=info msg="Serving chronograf at http://127.0.0.1:8889" component=server,
[18:52:41] INFO: Starting NGinx...,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:45 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 55a83e75-e99a-11ea-8012-0242ac1e2102 15731,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:52:58 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 5d1165f1-e99a-11ea-8013-0242ac1e2102 10715,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:02 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 5f77a1c8-e99a-11ea-8014-0242ac1e2102 8835,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:14 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 66989351-e99a-11ea-8015-0242ac1e2102 10105,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:16 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 67dbc729-e99a-11ea-8016-0242ac1e2102 13228,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:53:22 -0700] "GET /ping HTTP/1.1" 204 0 "-" "KapacitorInfluxDBClient" 6b504b75-e99a-11ea-8017-0242ac1e2102 74,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:53:22 -0700] "POST /query?db=&q=SHOW+DATABASES HTTP/1.1" 200 121 "-" "KapacitorInfluxDBClient" 6b5079bc-e99a-11ea-8018-0242ac1e2102 2760,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:53:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+_internal HTTP/1.1" 200 153 "-" "KapacitorInfluxDBClient" 6b511a5d-e99a-11ea-8019-0242ac1e2102 2388,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:53:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+homeassistant HTTP/1.1" 200 149 "-" "KapacitorInfluxDBClient" 6b51a8fa-e99a-11ea-801a-0242ac1e2102 1217,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:53:22 -0700] "POST /query?db=&q=SHOW+SUBSCRIPTIONS HTTP/1.1" 200 225 "-" "KapacitorInfluxDBClient" 6b522dd6-e99a-11ea-801b-0242ac1e2102 1342,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:28 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 6ef5cc75-e99a-11ea-801c-0242ac1e2102 30968,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:30 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 7008a3b4-e99a-11ea-801d-0242ac1e2102 7636,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:32 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 715ce12c-e99a-11ea-801e-0242ac1e2102 5578,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:34 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 729085ea-e99a-11ea-801f-0242ac1e2102 7990,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:39 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 754e8660-e99a-11ea-8020-0242ac1e2102 30511,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:41 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 76845ff4-e99a-11ea-8021-0242ac1e2102 8873,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:53:53 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 7daa6ebc-e99a-11ea-8022-0242ac1e2102 23113,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:05 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 84ceff83-e99a-11ea-8023-0242ac1e2102 119911,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:09 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 87319386-e99a-11ea-8024-0242ac1e2102 6694,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:11 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 8865ce8d-e99a-11ea-8025-0242ac1e2102 4810,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:15 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 8ac9b859-e99a-11ea-8026-0242ac1e2102 4796,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:21 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 8e527933-e99a-11ea-8027-0242ac1e2102 4258,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:54:22 -0700] "GET /ping HTTP/1.1" 204 0 "-" "KapacitorInfluxDBClient" 8f138722-e99a-11ea-8028-0242ac1e2102 489,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:54:22 -0700] "POST /query?db=&q=SHOW+DATABASES HTTP/1.1" 200 121 "-" "KapacitorInfluxDBClient" 8f13d88e-e99a-11ea-8029-0242ac1e2102 1725,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:54:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+_internal HTTP/1.1" 200 153 "-" "KapacitorInfluxDBClient" 8f14545a-e99a-11ea-802a-0242ac1e2102 1681,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:54:22 -0700] "POST /query?db=&q=SHOW+RETENTION+POLICIES+ON+homeassistant HTTP/1.1" 200 149 "-" "KapacitorInfluxDBClient" 8f14cdcd-e99a-11ea-802b-0242ac1e2102 1101,
[httpd] 127.0.0.1 - kapacitor [28/Aug/2020:18:54:22 -0700] "POST /query?db=&q=SHOW+SUBSCRIPTIONS HTTP/1.1" 200 225 "-" "KapacitorInfluxDBClient" 8f1545cd-e99a-11ea-802c-0242ac1e2102 12760,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:28 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 92a5ff8c-e99a-11ea-802d-0242ac1e2102 10691,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:39 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 99237b6b-e99a-11ea-802e-0242ac1e2102 190865,
[httpd] 172.30.32.1 - homeassistant [28/Aug/2020:18:54:45 -0700] "POST /write?db=homeassistant HTTP/1.1" 204 0 "-" "python-requests/2.24.0" 9cbb0139-e99a-11ea-802f-0242ac1e2102 4689,

fversteegen · August 29, 2020, 6:27am

Hmm… Since about July my Influxdb database size started growing very rapidly. It has been steady for years at around 3GB, but since July it grows with about 3GB per week. I do not have enough knowledge of Influxdb to determine the reason why, but maybe it is the same issue as you are having.

tom_l · August 29, 2020, 9:28am

Strange, my InfluxDB is still growing at a fairly constant linear rate.

chairstacker · August 30, 2020, 1:44am

It seems like the InfluxDB is less than 2GB when I use the sensor you suggested here @tom_l

But starting the addon immediately takes to load of my Pi4 from below 1 to well above 3: