This is definetely a good workaround to the problem and is a good thing in itself. It should really be implemented, but I also think making it possible to specify individual retention policies in the recorder: yaml section should exist too. The workaround should be easier to implement.
3 Likes
+1
I really need a system that can keep the history for different days depending on the entities
1 Like
I must say, the documentation around this is pretty rubbish… I’ve posted new threads, looked at old threads and just keep going around in circles - most of the threads I find have few if any people answering, so I’m now scattergunning in threads that do have answers…
I don’t see Recorder in the settings search, and given that HA is supposed to be moving most things into UI this would seem like a first point of call. At the very least the overall retention policy should be settable within the UI, but even better granular control at the entity level…
I do not, and really should not need to, start looking at external solutions like MariaDB or Grafana or whatever just to track entity data longer than 10 days
Recorder isn’t even in my configuration.yaml - at the very least shouldn’t this be pre-populated at setup (it says required in the docs but it’s clearly not…?)
My immediate question…
I have a self created Input Number (input_number.daughters_achievement_points) helper which gets updated at maximum once per day, typically once per week. I want to set the policy to keep this data indefinitely…
How on earth do I do this?
I would also like to keep longer data more broadly (30days) - I do not see any ui selection for this? Why do other thread say not to set it more than 30 days?
I’m only using 10% of my storage on my setup, I just want to store things longer
1 Like
With HA “out of the box” you don’t.
I share your frustration. IMHO Recorder is a weak spot in, one might even say an embarrassment to, the otherwise fantastic HA.
This isn’t really a thread for posting workarounds to the requested features. That said, I’ve found that the easiest way to manage long-term data is to save it to a text file and analyze it outside of HA.
Thanks, but even that seems overkill… I don’t even need to analyze it… I just want it to be stored longer!
I agree, Recorder should have more flexibility. But HA is a great product otherwise, and our only options are to use it as-is, or learn how to fix it ourselves.
Honestly, I don’t do a lot of analysis. I keep data on the operation of my heating and cooling system, and occasionally pull it up in a spreadsheet to compare how this season is going, compared to previous ones. I have heard of people who use statistical analysis software, but for my needs that would be overkill. For me, just knowing all the data is saved long-term, in case I need it some day, is enough. There’s really no reason for it to be saved in the HA Recorder database.
2 Likes
I think there’s quite a balancing act here for the developers: HA is designed for integrating and controlling home automation (devices). This requires a database and storage that is fast, extremely fast. Or your devices and automations will react too slow and users will start complaining. Those type of databases are not quite suitable for keeping data for the long run. They become buggy, searches won’t work, they can’t aggregate data etc. Thus adding a second database for long term data storage is necessary. I’s actually how most modern applications work: short-term working storage within the application and a long-term-storage in a separate database. It’s a pity this is not how HA is working out-of-the-box. I agree recorder needs more flexibility, however a second database (or file export) is inevitable IMHO.
1 Like
Very good point. I think adding the new statistics tables was a huge step in the wrong direction. Those should have been a separate database or used some other storage structure. Adding them with no purge options made it even worse. I hope you’re right that fixing this is inevitable. The rest of HA is so well designed that it’s odd to have this one neglected but critical component at the very core of the system.
1 Like
I personally have mine set to 30 days, and would really like the ability to keep some for 365+ days, doing for everything will build up but if its per entity then no need for them other database apps for most people, I would really love this feature.
I keep track off how many minutes my heating is on with HA, having long winter data is really cool
1 Like
Exactly, no need for fancy tools, just the ability to set different periods for different data. ATM we can use flat files (a.k.a. notifications) but then it’s hard to re-introduce the data into HA.
1 Like
Thanks, and welcome to the forum!
Don’t forget to go back up to the top and vote if you think this feature would be helpful.
1 Like
+1 that feature must be a ‘must have’
sad that two years have passed
3 Likes
+1.
It is also very harmless!
I implemented keep_days as part of the recorder.purge_entities service. There is an example automation in the below linked PR. I choose this design because:
- It can be modified without having to restart which makes it much easier to test
- External addons can call the service (ie appdaemon, node-red, etc)
- You can control when the purge happens
- It was a not significant change in design to add it and it avoids making the recorder more complicated
- Its possible to setup multiple automations for more granular control
- The target audience is advanced users
- A blueprint can easily be added for this
4 Likes
That’s a very nice addition, thank you!! More flexibility is always a good thing.
I still think the original FR should stand, since that would allow - even encourage - beginner users to manage what is retained in the Recorder database, right from the start. It’s unlikely anyone is going to even know that they need to set up automations calling the recorder.purge_entities service when they start adding entities to their new HA install.
We can certainly leave this open but I don’t expect we will implement a UI based control since it would be too easy to overload the system with ungrouped purge jobs.
Thank you. But I’m not sure I understand. This FR wasn’t for more purge jobs. Just the ability to set the retention duration by entity.
The problem is that we’re already overloading the system (specifically, the Recorder database) with many unwanted events and state changes. No reasonable person would want to retain every one of these for the same duration as the ones they care most about. Some we want to keep for a long time, some for shorter time, and others not at all. This has been discussed many times.
The current defaults are a trap for new users, especially those who run HA on a RPi with an SD card. The database quickly gets out of hand. New users may not be familiar with database administration and just assume all is well by taking the defaults.
This FR doesn’t have to be the solution. If there’s a better way to allow users to achieve a reasonable database retention regimen, right from the UI when they first start adding events, I’d love to hear it.
2 Likes
You would need a different query for every duration and entity combination that was configured. You end up another purge job for each entity + duration combination to find the rows that need to be purged. We could make it a bit smarter and group the queries with the same durations but it still be quite easy to generate too many combinations.
Can you give some more color here?
I’m surprised you would see an issue with the default of 10 days unless you have a misbehaving integration. Even at my very large installs with 3000+ entities, I don’t see my database growing to more than 1GB with the defaults. (And not more than 600MB with 2023.4.x/dev)
I took notice and started doing database maintenance when my database file reached over 3G. This was very early on in my experience with HA. I’ve seen many other references to databases that large and even larger on this forum. It’s been an ongoing issue which repeatedly impacts new users.
I’ve ruthlessly excluded everything I can, and do some manual purges when needed, especially prior to updating. I’ve also set my purge to 3 days. There are data I’d like to keep longer, so I just save them off to a text file, which I copy off the HA host to use elsewhere. It’s far from ideal, but it works for me. New users who don’t have DBA experience would be unlikely to be satisfied with this solution, nor with creating automations to run the purge service.
Again, the specific solution chosen isn’t that important. The ideal would be some way for new users to identify how long to keep data from each entity, as it is being set up. Just as today they are prompted to add the entity to a group.
From your questions, I assume there’s just one query run against the database daily to do a purge, and it’s fairly simplistic. Sort of like “everything greater than x days.” I see how doing multiple queries could get out of hand. Although, in the end, we’d be deleting the exact same number of records. They all get purged eventually.
That said, I could see that limiting the purge_days selection to just a handful of options, rather than any number of days, would be kinder to the database.