Best choice for keeping a longer history for sensors

Configuration
1

I’ve been using HA for about 8 months now, and love it. I’ve tried to keep my install as generic as possible. Other than the many cool components out there I have not installed any “extras” like nodered, appdaemon, etc. I wanted to get a firm understanding of how things were working before taking that step. Overall I have not found much I could not do, that I wanted to do, that required any extras. I’m running HA in a python VENV on ubuntu, at the moment.

Sorry for the intro before my actual question.
I’ve started to add a few sensors that I think I’d like to actually be able to keep a longer history. Also maybe something a bit more stable for storing the states of input_selects, booleans, text, etc. Once in a while the recorder component fails after a restart/upgrade and I lose my values and have to redo them.

Looking for something stable, but not overly complex. Something I 'm not going to have to upgrade and reconfigure every few months. Hopefully more of a set it and forget it kind of thing.

I’m not a developer, but I can get around in unix and yaml pretty well.

Thanks

Edit: Should have included some hardware stuff. I’m running on an older I3 with 4gb ram on an SSD.

1 Like
2

Have you looked at the MariaDB addon (https://www.home-assistant.io/addons/mariadb/) or MariaDB / MySQL external to your HA install to store your history etc?

The basics are covered in the addon link above and expanded a little here https://www.home-assistant.io/components/recorder/#custom-database-engines

I currently have mine set to hold data for 367 days ( allowing me to compare today to a year ago) and performance is acceptable for me on a Pi 3. You can also add your own databases and use the SQL Sensor (https://www.home-assistant.io/components/sensor.sql/) to query them.

Not sure if this will fit what you are looking for.

1 Like
3

The influxdb component was designed for this purpose.

1 Like
4

ok, so here is probably an obvious question about influx or MariaDB (or any external DB). Is it transparent to me as a user in the UI. As in, If I click on history, it just works? Or an entity graph, etc.

Any other tricks, issues, changes to my startup routine, etc.

1 Like
5

InfluxDB isn’t for holding history accessible to Home Assistant, you’d use other tool (like Grafana) to use that.

If you want longer history in Home Assistant, then yes, MariaDB or a number of others can be used transparently. Other than updating your config to point recorder to the new database, there’s nothing to do.

2 Likes
6

Can you explain where you would want to use Influx vs any of the other database options?

Maybe I have two things I’m trying to do, and both it what I need.

#1 - I want to have a bit more stability for keeping my input_text, Input_date_time, etc after a reboot.
#2 - I’d also like to have longer history for some/all of my sensors, entities. As long as there is not a substainal slow down. At the moment it’s more of a fund to do thing. Of course as it evolves, I can see uses of having a longer history.

7

because HAs DB is growing real big (even if you use something else as DB for HA) i use appdaemon to store my sensor values.
there is no real way i have seen to just save only the basic sensor state like:

  • timestamp;state
  • timestamp;state
  • timestamp;state

and every other way to save data makes a DB grow big very quick when you want to save more then a few sensors.

i have saved data from over 2 years now and for over 60 sensors (some of them changing state several times a minute) and i am 180 MB data now.

any other way to save data then just ts;state would increase the amount of data into GB area
saving that kind of data with HA would need a complete harddrive.

8

Sorry for the dumb question. Are you saying you use app daemon with Influx or just saying that using appdaemon is another option for what I am asking?

9

i say that using appdaemon gives you the option to look at the data and decide what to do with it.

it is for example quite easy to collect data from a sensor the updates 6x a minute and then take an average from every 5 minutes and save that every 5 minutes.
or just save the state and timestamp only like:

  self.listen_state(self.savestate,"sensor.anysensor")
def savestate(self,entity,new,old,attribute,kwargs):
  self.savenow(datetime.datetime.now(),new)

in the function savenow you could save the data to a database, or a file or any way you like.

3 Likes
10

Help me understand here. I only need to add 2 or 3 days to my history duration. Do I really need to install MariaDB or another addon to do this? Is the SQL built into HA so terrible and unstable that adjusting its capacity by just a few days would make the whole thing topple over?

11

There 2 aspects… MariaDB or any DB other than native MySQL might be more robust solution, more resilient to data corruption (especially while running on SD card). Also moving DB from SD card (in case of RPi) will largely enhace life expectation of card. Personally when I was running HA on RPi with relatively complex setup and carefully crafted to keep history of entities I really need to keep, I experienced 3 data corruptions requireing fresh installation on new SD card (it become unusable). After that I moved HA to VM on ESXi and DB to MariaDB on Synology NAS and never again experienced such issues, despite my installation grow and now i’m keeping more history (1 month).
So to answer your question; everything depends on your config and how much data you write (from transactions performed by DB engine it does not matter for how long you keep data, what matters is volume of data you write or delete from DB. So I’d start from ensuring you regularly do backups to restore system in case of disaster and then only start thinking about any other DB solution, if any data lost…

1 Like
12

Awesome explanation, thanks! So I don’t use SD cards, Like you, I am running HASS.IO on a powerful ESXI server with a dual redundant SSD RAID array. I found a very quick solution to increasing my history to a month. I can’t believe no one knew about this haha, Simply add this to your configuration.yaml:

recorder:
  purge_keep_days: 32

works like a champ!

5 Likes
13

fantastic thread, exactly what i was wondering! :slight_smile: