In my home, I have a number of “custom” sensors based on Arduino, ESP32, etc. For example I record the water levels in three different sumps (along with temp and humidity).
Right now I’m using Node-RED, and sending sensor values every 2 seconds into InfluxDB. I have kept data for a few years now.
This doesn’t actually take up a lot of disk space, but still I prefer to use “Continuous Queries” in InfluxDB, which essentially downsample the data as you prefer… right now I downsample a bit after 30 days, a bit more after 6 months, then a bit more after a year.
I’d like to move this over to Home Assistant, and the History / Recorder is the obvious way to do this.
Does Recorder have the ability to keep data on a per-entity level for different amounts of time? Or is the purge for all entities?
Does Recorder support downsampling (or will it)? The reason is that I would prefer to sample everything and all the time, than to only select a few things I really need.
As you can imagine, the more we automate our homes, the more data we can keep, and it gives us insight into things like power saving. For example I just installed a 16 channel energy monitor, each giving various stats about power, apparent power, voltage, current, total energy used… that would be: 16 channels * 5 data types per second - that’s 2.5 billion values per year…
It looks like the bright blue line on the chart is a downsampled / averaged version of the darker shaded blue area around that line. So-- is some downsampling happening already - or is that just a Google Graphs thing?
Hi hazymat,
did you ever find a solution that can solve your problem?
Just like you, I’ve got a handful of entities that update very frequently (about once a second) and a lot of entities that update a little less frequently (about 1-5 times per minute) and the need to keep this data for a long time.
This has lead my database currently sitting at 25 gigs, which makes handling backups challenging.
My current solution is to create filter entities to downsample data for every entity which I need to keep for a long time.
I then use a yaml file to configure how long I need each entity (or entity glob) in the database before data gets purged.
This file gets read by a python script which calls the recorder.purge_entities() service and purges all entities as per the config file. Every entity not in the config file gets purged after a default amount of days. Once every fortnight, the script also calls the recorder.purge() service with keep_days unset and repack set to true. I call the python script once every night.
This works reasonably well, but requires me to create a lot of filter entities and lacks the option to further downsample data after some time passes.
