Influxdb High CPU/memory usage every few days

Installation Home Assistant OS
1

Starting around 3-4 weeks ago my hass.io installation grinds to a halt and needs to be power-cycled every 2-3 days. At those times influxdb is using a lot of CPU and memory and the load average is in excess of 20. I’ve attached a graph showing the regular nature of this event

Peaky%20CPU%20Load
Peaky%20CPU%20Load1885×510 62.9 KB
.

The gap in data between 20-24th September is when I turned off influxdb to see if it was masking any other issues - it wasn’t as far as I can tell, and all ran smoothly.

I upgraded from 0.95.4 to 0.97.2 on the 19th August and that ran problem free until the beginning of September. I updated from 0.97.2 to 0.99.2 on the 20th September but not change in behaviour. Unfortunately, I don’t have details on when exactly I updated the influxdb and grafana addons but I am running the latest versions. I’m guessing I upgraded to the previous version 3.2.1 of the influxdb addon around the same time my issue started appearing, but I can’t be certain - I’m currently running 3.3.0.

Anyone else experiencing this?

1 Like
2

I’m running influx independently on a Linux host. I had similar issues until I increased the amount of RAM influx has to use. I think at one point there’s too much data and influx can’t shift it around the shards or delete it if you have rentention policies in place. Unfortunately I can’t help you how to do this in HASS.io

3

Thanks. I added more swap a week ago and it now stays up longer even when it started managing its data. However, after a day or so doing that I go this:

runtime: out of memory: cannot allocate 8192-byte block (538836992 in use)
fatal error: out of memory
runtime: out of memory: cannot allocate 8192-byte block (538836992 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0xc8fe90, 0xd)
        /usr/local/go/src/runtime/panic.go:608 +0x5c
runtime.(*mcache).refill(0x76fbc360, 0x304130b)
        /usr/local/go/src/runtime/mcache.go:124 +0xe8
runtime.(*mcache).nextFree.func1()
        /usr/local/go/src/runtime/malloc.go:749 +0x24
runtime.systemstack(0x31303c0)
        /usr/local/go/src/runtime/asm_arm.s:354 +0x84
runtime.mstart()
        /usr/local/go/src/runtime/proc.go:1229

So around 530MB of RAM in use and it runs out of memory. I tried restarting influxdb but it crashes again instantly with the same error. Memory looks like:

$ free -m
              total        used        free      shared  buff/cache   available
Mem:            927         170         475           3         281         674
Swap:          1023         287         736

So I’m a little stumped as to what pool of memory is running out.

4

I really haven’t used HASSio so I can only speculate:

  1. Maybe your influxdb container has a memory limit and that’s the memory being exhausted
  2. The free output might have been taken after influxdb has crashed, and as a result the free memory was previously used by influxDB and you really ran out of memory.

I’d suggest - taking a look at the influxdb.conf file. I have no idea how to do that with hassio. In regular install it’s in /etc/influxdb/influxdb.conf. With container it’s somewhere in the volumes, if 1. there’s a dedicated configuration volume, 2. you mounted the configuration volume.

I’d really advise you against running influxdb on a raspberry pi, unless you’re collecting 2-3 metrics every 10-15 minutes or so.

5

This post has the info on how to edit the docker-container configuration for influxdb:

Have the high-CPU-load issue also, but the ram is not exhausted on mine.
root@hass-io:~# free -m
total used free shared buff/cache available
Mem: 3826 1247 1247 75 1332 2560
Swap: 99 6 93
root@hass-io:~#