Storing your very long history - RRD Recorder

Share your Projects! Custom Integrations
1

Hi!

Sometimes we want to store long histories of our sensors (e.g. Network, Energy, Weather, etc) and the default sqlite is not the best place, nor a SQL database (MySQL, Postgres) due to performance issues and sheer amount of space it occupies. If you are like me, you have setup InfluxDB which is a specialised DB for time series, yet it is a new daemon you must install (probably through the Add-on) and lots of complexity just to answer trivial curiosity questions such as how hot was the summer last year. IMHO there should be a better way to solve these trivial questions.

To solve this issue, I’ve implemented the “RRD Recorder” which stores information into RRD database files, that have the particularity of having a fixed size and storing information using aggregation functions very efficiently. Using an RRD database one might store every event occurring in the last day, but only the hourly average of the sensor for the last week, and a daily average for the last year.

The current implementation can be installed through HACS (you must add a custom repository for now) and creates and updates the RRD files (recorder functionality) and renders fully configurable graphs (through a virtual camera) using rrdtool graph functionality (check documentation).

10 Likes
2

Wow I’ve thought for a while that an RRD would be a great addition to the HA toolkit. Well done.

1 Like
3

Thanks for the feedback :slight_smile:

I was not that sure if bringing a 90’s network tool would be of use to anyone :smiley:

1 Like
5

I am trying to install into HA, which runs in Docker containter (it is running in Alpine Linux). But packages librrd-dev and libpython3-dev are not available in Alpine. Therefore during app start there is a error message:

Error: Unable to compile the binary module. Do you have the rrdtool header and libraries installed?

I tried to install DEB packages manually by DPKG, but there are too much dependencies.

6

Sorry, I’m still fighting how to make this component available for those running hassio.

The solution is to create a python wheel package of rrdtool. But I haven’t researched enough how to do it and provide that through HACS.

1 Like
7

I’ve updated the repository with a “camera” component that can be used to graph rrd databases.

2 Likes
8

Networking tools was where I first encountered RRD tools.

1 Like
9

It seams that rrdtool-dev is available

https://pkgs.alpinelinux.org/packages?name=rrdtool-dev&branch=edge

$ apk add rrdtool-dev ?

10

It looks like rrdtool-dev is not same as librrd-dev. Or some env variable or path is missing?

~ # apk add rrdtool-dev
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/armv7/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/armv7/APKINDEX.tar.gz
OK: 211 MiB in 183 packages

~ # pip install rrdtool
Collecting rrdtool
  Downloading rrdtool-0.1.15.tar.gz (21 kB)
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w21m2_pl/rrdtool/setup.py'"'"'; __file__='"'"'/tmp/pip-install-w21m2_pl/rrdtool/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-py3js4ew
         cwd: /tmp/pip-install-w21m2_pl/rrdtool/
    Complete output (1 lines):
    Error: Unable to compile the binary module. Do you have the rrdtool header and libraries installed?
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
11

Probably missing the compiler…

I successfully got rrdtool installed on an Alpine image using these steps:

$ apk add build-base rrdtool-dev python3-dev py-pip
$ pip install rrdtool
1 Like
12

I would like to share my implementation of final version of this RRD recorder component. I am using it for power conseption metering of whole flat.

rrdtool
rrdtool620Ă—541 47 KB

I love dark theme, so I updated parameter to be able to generate dark charts.

Data are readed measured by Shelly EM, which is installed directly in flat electric distribution box.

IoT Mereni Spotreby 800
IoT Mereni Spotreby 800800Ă—460 157 KB

I am using custom Dockerfile, because RRD recorder component has some libs as dependencies.

FROM homeassistant/raspberrypi4-homeassistant:dev
RUN apk add --no-cache build-base rrdtool-dev python3-dev py-pip ttf-dejavu

Whole project was helpful. Thanks @dgomes.

3 Likes
13

I think you inspired me to do something similar :slight_smile:

I’ve been using RRD to measure my internet connection, but energy is also a very good idea.

Why don’t you share your RRD recorder + camera configuration with us ?

14

Just to give some additional information to interested users:

If you are using either Home Assistant OS or Home Assistant Container, you already have a librrd wheel available. You can easily add the supporting dependencies using an automation:

Add to automations.yaml:

- alias: Install librrd
  trigger:
  - platform: homeassistant
    event: start
  action:
  - service: shell_command.install_librrd
  initial_state: true

Add to configuration.yaml:

shell_command:                                      
  install_librrd: "apk add librrd ttf-dejavu"

Please note, that you will have to restart HA once for the new installed dependency to take effect

1 Like
15

Thank you
If I understand correctly, this tool cannot be used on HA in Docker.

Can I use it as a separate tool, for example, as a separate Docker container, to connect to MariaDB and reduce data?

Thanks
Leonid

16

This tool is now obsolete, the long-term statistics of HA provide a very similar functionality.

See Long- and short-term statistics | Home Assistant Data Science Portal

1 Like
17

Admittedly rrdtool doesn’t get many updates. But it’s hardly obsolete. And it produces way better graphs than almost anything else, and is easily customisable.
And it’s not a confusing mess of contradictions, unlike a lot of software.

1 Like
18

rrdtool python binding library has been deprecated by it’s author, so my custom integration doesn’t work anymore :frowning:

19

It’s easy enough to call the rrdtool binary itself.