Excludes / Includes for Recorder purge

killsystem · December 19, 2016, 8:32pm

Allow the user to define the data that should be purged through the ‘purge_days’ with exclude or include statements.

Exlude: would hinder the purge to delete the specific entries.
Include: would define only which entries should be deleted.
Multiple entries of ‘purge_days’ would allow you to keep some data longer than other.
So you can keep all those climate data, but get rid of the motion sensor triggers in the database.

This entry may be a little similar to Exclude option for the Recorder component but would filter the data on purge, not before.

BlinkyLights · May 2, 2017, 6:16pm

I agree, at this point I really do want to monitor door and light events over maybe two weeks, however I would like to keep device tracker home and away information for maybe a few months, and temp/energy data for maybe two years to be able to compare year over year data.

I think putting a full include / exclude under multiple purge top level items of various lengths would be highly useful.

RobDYI · May 2, 2017, 9:08pm

You can automate a shell command to run simple sql command to delete entries matching your criteria. I did it for not needed entries for logbook / history to work.

HerrieMan · January 17, 2019, 11:37am

Hi,
To bring up an old topic, has this been addressed yet in the meantime ?
I have a similar request, would like to e.g. keep the ‘high-res’ data for my Temp/Hum/Baro sensors for 1 week, and then have a statistics sensor process this into e.g. 1 min/max/avg data-point per day and keep that data for e.g. 5 years or so.
Would this be possible ? any suggestions on how to configure this would be great.
Thanks,
Herman

RobDYI · January 17, 2019, 4:00pm

I would try other software that specialize in the graphs and collection that you need and link the webpage. HA could automate input of the data you want into a cloud database or your mysql database. I collect my solar heater temps and pool temps in a mysql database (same mysql but different than HA database) and use a browser graphing html page to display the graphs I want.

https://jpgraph.net/

HerrieMan · January 17, 2019, 6:03pm

Hi,
your suggestion looks good, might be a better option than to store everything on HA itself.
Was thinking about somehow pushing the data to my google-drive account and just use sheets to make some graphs, but haven’t really figured out how to make that work

HerrieMan · January 17, 2019, 7:52pm

just found this link: Quick Start to Home Assistant Data Science
That also looks promising, will check some out and see what works best for me.

CarlosGS · February 14, 2021, 9:26pm

For anyone reaching this: It’s just better to set up influxdb and only “include:” the desired sensor entities.

How to set up influxdb with the official plugin:

github.com

hassio-addons/addon-influxdb/blob/f4f35590a29e79ae7db39bb58f1cf9b558c240f6/influxdb/DOCS.md

# Home Assistant Community Add-on: InfluxDB

InfluxDB is an open source time series database optimized for high-write-volume.
It's useful for recording metrics, sensor data, events,
and performing analytics. It exposes an HTTP API for client interaction and is
often used in combination with Grafana to visualize the data.

This add-on comes with Chronograf & Kapacitor pre-installed as well. Which
gives you a nice InfluxDB admin interface for managing your users, databases,
data retention settings, and lets you peek inside the database using the
Data Explorer.

## Installation

The installation of this add-on is pretty straightforward and not different in
comparison to installing any other Home Assistant add-on.

1. Search for the "InfluxDB" add-on in the Supervisor add-on store and install it.
1. Start the "InfluxDB" add-on.
1. Check the logs of the "InfluxDB" to see if everything went well.

This file has been truncated. show original

And if you already have a large home-assistant_v2.db file you can migrate your data using this script:

(In particular I used this fork of the script invoked as python homeassistant2influxdb.py -t SQLite -d home-assistant_v2.db, after following the readme that is)

Cheers!

magiva · April 9, 2022, 8:47am

what is suggested here (orignally) it to have a purge-exclude.
i would want to record my sensor data for the normal 10 days as that is useful but when purge happens i want to “simply” exclude my power and maybe the weight/fitness scales as i want that data for years.

having to specify just what you want to include to store or purge is impossible to maintain

BlinkyLights · April 9, 2022, 6:42pm

Influx is a workaround to HA not managing long term data and also then requires something in addition to visualize the data.

With the addition of more power tracking the original request as a native feature makes sense.

HA and it’s built in DB and Graphing need to improve how they handle long term and short term data.

Power tracking, water use, weather data, hvac all possibly benefit from annual data comparison. Where as I don’t care past maybe 30 days if my door or blinds open and closed.

CarlosGS · April 10, 2022, 1:55pm

Well they have been working on that In this time HASS implemented “long term statistics” and the new energy dashboard feature that allows to store&visualize the long term data for power sensors. That way the historic data is kept separately from the state database.

Note the state database (recorder) can get corrupt upon a power outage and there is no easy way to backup/restore its data. If you need to keep valuable data for years, do look into the “long term statistics” feature or set up InfluxDB/Grafana for more granularity.

BlinkyLights · April 18, 2022, 5:31pm

The SQLlite or if you set it up external MySQL DB is VERY EASY to back up, I take nightlys. Saying it is hard to back up so you should another set of redundant tools is stilly.