As the title says. Log retention for only 10 days?
There needs to be a better way to handle logs than yaml files.
You’ve confused three unrelated things
- History is kept for 10 days by default - and that can be changed. History is for tracking entity history.
- YAML isn’t used for logs
- The log isn’t limited to 10 days
1 Like
pardon me.
What I did mean was History yes.
Also… I had to go to and add
recorder:
purge_keep_days: 365
Hence the mention of yaml.
Unless you have restricted which entity’s state-changes and events are permitted to be recorded, you might regret setting the purge cycle to 365 days (for all entities and events).
Keep a close eye on the database’s growth over the next few weeks. A very large database impacts performance.
FWIW, long-term data storage is handled another way in Home Assistant.
6 Likes
Hence the WTH post.
Why do I need to go balls deep into the dev documents to get this basic functionality for my Power monitoring, network monitoring etc.
2 Likes
You don’t have to go to any depth.
Home Assistant will automatically record long-term stats for suitability configured entities. It’s the responsibility of an integration’s author to determine which entities are automatically enrolled for long-term retention (i.e. never purged).
For example, I have 3 years worth of data for various sensors (mostly dealing with energy consumption and operating time) and my database’s purge cycle is the default 10 days.
You made an assumption that increasing the purge cycle, to once a year, is the optimal strategy for long-term storage (it’s not) and then disliked the fact that the setting is only available in YAML. Increasing this setting is almost never necessary, and not even advisable unless mitigated by excluding entities and events, so that’s why it’s not exposed in the UI (where it’s too easy to unwittingly cause performance problems).
tl;dr
Your WTH is based on a misunderstanding of how long-term data is recorded.
3 Likes
Okay… fair…
I’ve only used the platform for little over a year. I honestly don’t know everything.
If I open my History tab, select my Shelly power monitoring Device, and select time table “This year” I only get past 10 days. If I check it 2 days later, I still get only those 10 past days.
This is not intuitive.
I got it “more intuitive” via before mentioned step (yaml configuration) . Now I can see past 10 days in my History graphs. What am I doing wrong in that case?
Why can’t my WTH be - make it simpler?
Nothing. The integration doesn’t support Long Term Stats - you should raise a WTH about that integration not supporting it.
2 Likes
Been using HA for two years …
WTH are long term stats !?!?!?
I agree with the OP, while you can raise an issue with every single integration that you want history stats… Why can’t we use the main tool… Home Assistant… To easily change a field for each entity or device? The average user isn’t going to want to dive into yaml or learn how to address an issue with the integration developer. Do we want to gatekeep people who don’t have the time/skills to change their history, or make the product easier to use?
2 Likes
Totally agree. I still don’t know how to store long term history for specific entities. The log retention is use case specific: integration developer has no idea about my use case and hence most of my sensors don’t support longer retention. This should be easy to configure from the UI on per entity basis.
1 Like
Totally agree that the default retention leads to horrible experience
The default should be at least 100 days - that would let us, when a new month starts, to compare the last month vs. the previous two months so we can check for anomalies or trend
Blaming tons of different devices is ridiculous - if home assistant knows how to store 10 days of stats, it should know how to keep them around longer…
1 Like
As said before, HA knows how to do this. But there’s a downside to keeping a lot of data in a database you want to access often. And we want HA to be able to run on simple devices like Rpi with storage that is limited in speed and size. I think the main problem is not the default recorder time. It’s the fact that the user has no influence (except through workarounds) on which entities are stored in the long-term database and which are not
1 Like
If we rephrase this WTH to “why is there no per-entity and per-integration setting for retention in long-term statistics” , I would vote for it. It would not be as simple as a check box of course, since you need to assign a measurement class if the integration doesn’t provide it, but it would put the power with the user instead of the integration developer.
1 Like
@allard77 well - if we want to support Rpi with small and slow storage, have 100 days as default for everyone and recommend using a different number in Rpi install guide!
Or maybe add out-of-the-box alert if disk space is less than a few GB etc - there are many solutions that don’t give negative experience out-of-the-box.
Regarding “per-entity, per-integration settings” - don’t force all users to micromanage the internal storage space. Have great defaults first (this WTH), than sure have ability to super customize - but only if really needed. Comparing this month to the previous month or two is just super basic feature in my opinion.
By default long term statistics are stored forever. So this comparison with numerical sensors is already available regardless of the 10 day retention period.
@petro “By default long term statistics are stored forever” this is true but 100% irrelevant.
Most entities are not compatible with long-term statistics internal requirements. So in practice, out-of-the-box, all users can only see 10 days of history for most of their things 
I am not asking for 10 years of retention by default, just a small 100 days so everyone can compare the current month with previous two without reading multiple discussions on how to wrap each entity with another entity that has the correct metadata to convince HA to store its stats (still haven’t figured it out, if that is not obvious, seems like a massive time sink, for now I tried to increase my default to 100 days).
1 Like
That’s not True. That may be true if you use random custom integrations. However most internal UI entities support LTS. Most yaml entities as well, however you need to know to add state_class to them.
100 days would be 10-20 gb of data with most setups. Just buck up and add a longer default for yourself. It’s rather simple.
# Add to configuratoin.yaml as-is
recorder:
purge_keep_days: 100
2 Likes
This is not true. Most of my sensors that I would ever care for are stored forever, automatically. Those include devices using the Shelly integration, pretty much everything related to energy for example. Not sure why someone is having such problem with theirs Shelly.
So you guys have some examples of which exactly sensors from which devices are not automatically stored in the long term statistics? Maybe it’s some bug.
2 Likes
Did you check the energy sensor of that device? Because I just checked, my shelly does store energy sensor data long term, but the power only in short term. I think it makes sense in general, unless you want to somehow be able to check peak power consumption over a long period of time or something similar. Still should be possible to set it up, but not that straightforward.