Storing your very long history - RRD Recorder

Tags: #<Tag:0x00007f3258011cb0>

Hi!

Sometimes we want to store long histories of our sensors (e.g. Network, Energy, Weather, etc) and the default sqlite is not the best place, nor a SQL database (MySQL, Postgres) due to performance issues and sheer amount of space it occupies. If you are like me, you have setup InfluxDB which is a specialised DB for time series, yet it is a new daemon you must install (probably through the Add-on) and lots of complexity just to answer trivial curiosity questions such as how hot was the summer last year. IMHO there should be a better way to solve these trivial questions.

To solve this issue, I’ve implemented the “RRD Recorder” which stores information into RRD database files, that have the particularity of having a fixed size and storing information using aggregation functions very efficiently. Using an RRD database one might store every event occurring in the last day, but only the hourly average of the sensor for the last week, and a daily average for the last year.

The current implementation can be installed through HACS (you must add a custom repository for now) and creates and updates the RRD files (recorder functionality) and renders fully configurable graphs (through a virtual camera) using rrdtool graph functionality (check documentation).

7 Likes

Wow I’ve thought for a while that an RRD would be a great addition to the HA toolkit. Well done.

Thanks for the feedback :slight_smile:

I was not that sure if bringing a 90’s network tool would be of use to anyone :smiley:

I am trying to install into HA, which runs in Docker containter (it is running in Alpine Linux). But packages librrd-dev and libpython3-dev are not available in Alpine. Therefore during app start there is a error message:

Error: Unable to compile the binary module. Do you have the rrdtool header and libraries installed?

I tried to install DEB packages manually by DPKG, but there are too much dependencies.

Sorry, I’m still fighting how to make this component available for those running hassio.

The solution is to create a python wheel package of rrdtool. But I haven’t researched enough how to do it and provide that through HACS.

I’ve updated the repository with a “camera” component that can be used to graph rrd databases.

1 Like

Networking tools was where I first encountered RRD tools.

It seams that rrdtool-dev is available

https://pkgs.alpinelinux.org/packages?name=rrdtool-dev&branch=edge

$ apk add rrdtool-dev ?

It looks like rrdtool-dev is not same as librrd-dev. Or some env variable or path is missing?

~ # apk add rrdtool-dev
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/armv7/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/armv7/APKINDEX.tar.gz
OK: 211 MiB in 183 packages

~ # pip install rrdtool
Collecting rrdtool
  Downloading rrdtool-0.1.15.tar.gz (21 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w21m2_pl/rrdtool/setup.py'"'"'; __file__='"'"'/tmp/pip-install-w21m2_pl/rrdtool/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-py3js4ew
         cwd: /tmp/pip-install-w21m2_pl/rrdtool/
    Complete output (1 lines):
    Error: Unable to compile the binary module. Do you have the rrdtool header and libraries installed?
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Probably missing the compiler…

I successfully got rrdtool installed on an Alpine image using these steps:

$ apk add build-base rrdtool-dev python3-dev py-pip
$ pip install rrdtool
1 Like

I would like to share my implementation of final version of this RRD recorder component. I am using it for power conseption metering of whole flat.

I love dark theme, so I updated parameter to be able to generate dark charts.

Data are readed measured by Shelly EM, which is installed directly in flat electric distribution box.

I am using custom Dockerfile, because RRD recorder component has some libs as dependencies.

FROM homeassistant/raspberrypi4-homeassistant:dev
RUN apk add --no-cache build-base rrdtool-dev python3-dev py-pip ttf-dejavu

Whole project was helpful. Thanks @dgomes.

1 Like

I think you inspired me to do something similar :slight_smile:

I’ve been using RRD to measure my internet connection, but energy is also a very good idea.

Why don’t you share your RRD recorder + camera configuration with us ?