HA Data Retention - Lost data

Configuration
1

Hi All,
I hope you can enlighten me! It’s probably me being dumb!
My HA system went live(ish) around middle of August, this year, with various integrations added since, but nothing significant since the middle of September.
I have the ‘Recorder’ retention set to 1000 days.
So, for example, when I click on entities, such as my Solis solar data and other energy entities, I have data going back to the middle of August.
This morning, I clicked on a couple of entities from my Solcast integration and noticed that the data now only goes back to the beginning of Oct. On further investigation, much of my data also seems to have disappeared prior to this date, but not all. Certainly, data history for my custom sensors also now only starts at the beginning of October.
Have I done something to cause this, or is it that I don’t understand how HA data retention works?
Help! …

2

I can’t answer your question specifically, but why do you want retention of 1000 days? Do you really need to know when you turned a light on 3 years ago? The data that is useful to keep long term is probably already in long-term statistics, which are kept forever regardless of the retention setting. And if something is not being stored in long-term statistics you can always create your own template sensor to do so.

My database is 2.5GB with 10 days retention - 100 times bigger would be a nightmare for backups etc.

1 Like
3

Hi Michael, thanks for that.
Specifically, regarding Solcast data, I wanted to retain their forecasts over a period of at least a couple of years to see how their accuracy improves, or not. As I said, all of a sudden, that data now only goes back to 1 Oct. Is this something dictated by Solcast or is there an HA process which culls data on sensors which are not classified at LTS?
This being the case, if I want to keep that data, do I need to copy the sensor in question to a custom sensor and add a state of ‘total ascending’ - does that sound right?

Presumably if I reduce my purge setting down to say 30 or 60 days, then my existing db is going to shrink at some point? When does the purge take place?

Sorry - so many questions!

4

I believe the two requirements for long-term statistics are to specify state_class and unit_of_measurement. state_class can be one of measurement, total, or total increasing - so given you have forecasts, it would seem like measurement is most appropriate.

I’ve seen some people suggest you can use “customize.yaml” to put a state class on an existing entity to get long-term statistics. It might be worth trying that first, and falling back to copying the state to a template sensor. You can check long-term statistics are working by waiting at least an hour (for the first to be created) and then using the Statistics tab under Dev Tools to see if the entity is there.

You can also use the recorder.purge service call in Dev Tools any time you like with a custom “days to keep”. According to the documentation, an automatic purge occurs at 04:12 local time every day, assuming it is enabled (which is the default). To get the benefit of the reduced size you would probably also have to use “repack” in the service call.

1 Like
5

Thanks again Michael.

Three last questions…

  1. I’m really struggling with the definition of the 3 device states. Measurement, I thought I understood, until you suggested that Solcast’s kWh forecast should be classified as such :-). I assumed measurement would be for things like speed, temp, humidity, levels, etc
    Very unclear as use of total and total_increasing. Are you able to give me specific examples for these.

  2. Any idea why, as of now, some of my (short-term) stats now only go back to somewhere between 1-12 October, when they use to go back to early September?

  3. What dictates whether the history explorer card shows data as either line or vertical bar?

Again, sorry for the questions, but HA’s like that - you understand one thing and that uncovers a whole load more stuff you don’t understand!

6

Ah my bad. I didn’t know about SolCast and a very quick google made me think it’s weather forecasting. So yes, total increasing for kWh.

Sorry, no idea where your history went. And i don’t use the history explorer card - i prefer apex charts.

1 Like
7

Here you go: