Long term statistics: sensors purged after 10 days although state_class set: attribute order a problem?

Thanks for the clarification; I remember it didn’t work when I tried it (a long time ago) but clearly it does work so I must have misremembered the test case and/or results. :man_shrugging:

Anyways, state_class doesn’t appear to work for MQTT sensors so manual customization can be a YMMV situation.

1 Like

I see; have no experience with mqtt, cannot test unfortunately.

If you ever want to try it, apparently version 2024.9.0 makes it easier than ever (to install).

Taras, thanks a lot for the link.

1 Like

That is totally fine for me as thread opener. Anything to be learned about this topic is welcome. :slight_smile:

What I have learned for now:
Using customize to set state_class might help or might not help. For my sensor ist does not help. Also not for some (modern?) Utility Meter Sensors. I will post an example today or tomorrow.

It looks like I have to create around 50 duplicates template sensors to get LTS. That will be a great work for the weekend. :roll_eyes:

Here is the concrete example, that not only MQTT sensor have problems with customized state_class:

I have a self created template trigger sensor which works perfect (as all template sensors). It has a state_class and LTS is working like a charm for this sensor:

template:
 - trigger:
   - alias: pv
     id: pv
     entity_id: sensor.sunny_home_manager_2_metering_total_absorbed
     platform: state
     not_to:
     - unknown
     - unavailable
   sensor:
   - name: vk_pv_wr_bezug_gesamt
     unique_id: vk_pv_wr_bezug_gesamt
     unit_of_measurement: "kWh"
     state_class: "measurement"
     state: >
       {{ states('sensor.sunny_home_manager_2_metering_total_absorbed') }}

I installed the Utility Meters integration to get to know the daily changes of this sensor:

utility_meter:
  vk_pv_wr_bezug_gesamt_daily:
    unique_id: vk_pv_wr_bezug_daily
    name: "vk_pv_wr_bezug_daily"
    source: sensor.vk_pv_wr_bezug_gesamt
    cycle: daily

As this utility meter sensor was not showing LTS data, I added state_class (and later for a test also the already existing unit_of_measurement) to the customize section:

homeassistant:
  customize:
    sensor.vk_pv_wr_bezug_daily:
      unit_of_measurement: "kWh"
      state_class: "total_increasing"

Still, LTS does not work. Just not to miss again to give information: it it less than 10 days ago I added the customize section for this utility meter sensor. So maybe it is going to work in a few days because HA only can store data long term that was generated after I added the customize section. BUT: I hoped the same for the MQTT sensors and there it did not work out. :wink: Also the typical “Source” information is missing in the “statistic graph card”. So I guess it will stay as it is. The data will be purged after 10 days.

So I’m pretty sure that the problem that customize does not work to enable long term statistics, is not only bound to MQTT sensors.

P.S.
Just in case somebody has the same idea as me:
Adding the state_class to the utility_meter sensor will be denied by home assistant. It is not starting anymore. :wink:

I wonder why the issue came up from the beginning - in my practice all utility_meter sensors were already with some state_class right after creation, i.e. no “customize” was needed.

I did a test an created the same sensor again:

  vk_pv_wr_bezug_gesamt_daily_3:
    unique_id: vk_pv_wr_bezug_daily_3
    name: "vk_pv_wr_bezug_daily_3"
    source: sensor.vk_pv_wr_bezug_gesamt
    cycle: daily

And you are right. The state_ class does exist:

Nevertheless my first sensor with the same configuration was purged after 10 days. I only started using customize with state_class after I observed that the values are purged and somebody told me I should use customize. I will wait another 12 days for this new sensor and see what happens. But I doubt that it will work as the first one did not work either.

What I just realized:
The source sensor “vk_pv_wr_bezug_gesamt” does have the state class “measurement” despite measuring kWh. That should of course be “total_increasing” instead. It does not harm the long term statistic of thios sensor at all. But maybe the LTS for the Utility Meter sensor does not work because the source sensor has a “wrong” state_class? :thinking: The Utility Meter is using the correct “total_increasing”. So it has “overruled” the value from the source sensor.

Not sure whether I can change the state_class of source sensor “vk_pv_wr_bezug_gesamt” without destroying my LTS for that sensor. I think it will not work because of different database columns used for the state classes.

EDIT:
I created a new source sensor for the Utility Meter Sensors with state class “total_increasing” and a new utility meter sensor referring to the new source sensor (without customize) .

So now I have to wait 10+ days to see whether different state_classes were the problem for the utility meter sensors regarding the LTS.

You seem to be confusing the recorder with statistics.

Every entity, those with LTS or not, records every state change in the recorder. These are always purged after your purge_keep_days interval.

Entities with LTS have an additional database table called long term statistics that are generated every hour. These are not purged.

You don’t have to wait 10 days for anything. If you want to see what statistics are being collected for your sensors, plot them on a statistics graph card. If you see data on the card, then statistics are being collected. This should start within 1 hour of creating your new sensor.

I’m well aware of that fact. Nevertheless I only see 10 days in the statistics graph card.

Unfortunately that seems not to be true. Yes, I do see my problem-sensors in the Developer Tool / statistics section and in the statistics graph cards as I have written already a few times. But I only see 10 days.

Example for a working MQTT sensors where the source integration sets the state_class itself:

You can see that data is shown for the last 30 days.

And now the same statistics graph for a MQTT sensor where the source integration does not set the state_class but I added it per customize:

You can see: despite 30 days are configured there is only data for 10 days. I’m pretty sure that setting state_class via customize was way before 26th August. But now I’m asking myself whether I have maybe switched it forth and back for testing purposes. :thinking:

Let’s wait
While writing this post I notice: With all these changes I have done the last weeks I’m kind of lost, what I did when. Whether I turned something back and if yes when. So I would propose that I wait for 10+ days now and see how the MQTT sensors and the new Utility Sensor behaves after that. I will come back with result.

Sometimes waiting really helps. All my MQTT sensor I added a state_class via customize show now long term statistics data for more than 10 days. So customize work for these entries. :+1:

I’m very sure I added it 17th August. So I do not know why it is working now. But maybe I did some tests in-between which caused the 10 days to start again.

My Utility Meter sensor still only show 10 days. But I think the root cause is the fact, that the source sensor has state_class “measurement” despite unit “kWh”. I created for both - the sourse sensor and the Utility Meter sensor - new instances yesterday. It looks very promising, as when hovering in the graph it shows the “Source:” information which I only observe for working LTS sensors.

I need to wait another 10 days for the Utility Meter Sensor and will come back with the result.

It seems the problem is not home assistant but sits 30cm in front of the keyboard. :wink: Sorry for the confusion and thanks a lot for your input which helped me to find the different root causes.

Today I also can announce, that my Utility Meters are also working correctly now.

So let me recap this thread:

I had problems with

  1. MQTT sensors have been purged after 10 days although I added a state_class via customize
  2. Utility Meters sensor have been purged after 10 days although they had a state_class set by the Integration itself

Both is solved. The solutions are:

  1. For the MQTT sensors I did either not wait long enough or played around too much. Setting state_class via customize finally worked. Special thanks to @Ildar_Gabdullin for pointing out that customize for non-legacy sensor still should/could work. :slight_smile:
  2. For the Utility sensor the root cause have been, that the source sensor has set unit_of_measurement to “kWh” but state_class to “measurement” which does not really fit together. The Utility Meter uses state_class “total_increasing” for kWh. After changing the source sensor to “total_increasing” (and waiting 10 days :wink: ) everything works fine. :slight_smile:

Sorry for bothering you with my misconfiguration and thank you for all the help! :slight_smile:

1 Like