Long term sensor data

1

Hi,
i’m looking at a method to store long-term sensor data.
I have been looking at different options, but for now haven’t find the right solution.
What I would like to have is for short-term (e.g. several days) have all ‘high-res’ sensor data available, for some of my sensors this is 1 datapoint per second so this quickly becomes much data.
For the long term I would like to have a min/max/average per sensor per day, so I can see seasonal changes.
Thoughts that came to mind are:

  • filter-sensor: data is ‘by-default’ stored into the same database the high-res data is in, and don’t know if it’s possible to purge the default database only for the high-res data and leave the min/max/avg in place.
  • other possibility: some sort of automation that processes the high-res data into 1 datapoint per day and store that into another database
  • run some python script and export sensor data to e.g. google-sheets or some other online platform and do further processing there.
  • also looked at the jupyter addon by with data-detective by @robmarkcole @frenck , this really looks awesome, but haven’t figured out if I can use this to create my long-term sensor data.

But as I said I haven’t really figured out how to set this up correctly, any thoughts on how to do this ?
What I’m looking at is basically what Domoticz uses in their sensor graphs.
Regards,
Herman

2

Any of the suggestions you made are worth a try

3

ah, yes, I forgot to mention that I actually did give them a try :wink:
The problem is that I’m not a software guy and only have very limited knowledge of python, e.g. for the last option I have been trying to copy/paste some code together to make e.g. plots of the current data in the hass-database, that already cost me several hours.
I don’t have a clue where to start when I actually want to start doing calculations on the data like min/max/average and store that again into some sort of database.
More or less the same goes for the other option, the ideas are there, but I lack the proficiency to actually implement it :frowning:

4

You can configure the recorder to only store data you care about. For studying your data checkout the jupyter addon

5

I use influxdb for long term metric storage. It can ‘downsample’ data after a certain point but I haven’t configured that yet (only been using influxdb for a couple weeks now). It’s a lot faster than storing all the metrics in SQL and by combining it with Grafana it allows you to easily make nice dashboards. It sounds like that is exactly what you want and it’s rather easy to setup. I made some dashboards in Grafana and include them as iframe in home-assistant using Grafana’s kiosk mode, so it integrates cleanly :slight_smile:

6

This is the page that describes Influxdb downsampling, which I think is what you are after. I have to admit, that I have just kept the default retention policy and haven’t looked in to it much.

https://docs.influxdata.com/influxdb/v1.7/guides/downsampling_and_retention/

7

Thanks @mvdm and @gpbenton, I wasn’t aware of the influxdb addon, that adds yet another option to my list. Just spend a few minutes configuring it and I must say this seems to be the most easy setup till now.
Without any programming I could just make nice plots of my sensor data and will check also the downsampling option for the long-term data.

8

This is a great idea. Push metrics to InfluxDB (very low retention), downsample and then integrate long term data via influxdb sensor.

9

Is there experience how to downsample all HA influxdB addon data in one continuous query all in one go?

Happy to get a guide/template.