Long term statistics: sensors purged after 10 days although state_class set: attribute order a problem?

Which integration produces this entity?

sensor.wolf_wp_cha_0x3_270048_verdichterfrequenz

If it’s the MQTT integration, is the entity configured via YAML or MQTT Discovery? Post its configuration (if available).

The question is a bit confusing… :persevere:

When you create a new sensor, data will be purged after 10 days (or whatever you’ve set purge_keep_days to).

If the sensor state_class is measurement, total or total_increasing, sensor data will also be saved to long term statistics. This happens once an hour, so you’ll get 24 entries a day.

The two are different, and are saved separately. Short-term data are still purged; long-term statistics are never purged.

The data is created by a stand alone application (ism7mqtt) which sends data to MQTT. I guess MQTT is set to auto discovery. I never did anything to add a new sensor. They appear automatically.

Sorry for that. :face_with_peeking_eye:

Regading your answer:

I know that that short term and long term two different statistis (even DB tables). But as you can see in my screenshot the state_class is set correctly to measurement.

So why it is not working? Why do I see long term statistics for some sensor and not for others with exactly the same configuration?

In that case, I suggest you contact the author of the custom integration (ism7mqtt) and report the issue.

What comes into my mind:
It is possible, that the state_class attribute of all not working sensor are added by customize in configuration.yaml by myself. I have to cross check that tomorrow.

Maybe “just” the customization does not work when trying to add state_class for enabling LTS. :thinking:

You overlooked to mention this important detail in your first post.

That trick works for Template Sensors configured using legacy format which don’t natively support the state_class option. It’s not known to (always) work for entities created by other integrations.

The state_class attribute you added via customize.yaml is being treated like any other attribute (i.e. not being used for long-term statistics).

Contact the author to add a proper state_class option to the entity.


FWIW, the same thing happens to users who attempt to add friendly_name to a sensor entity via customize.yaml. It appears in the entity’s list of attributes but it is not handled as the entity’s friendly name. EDIT See Ildar_Gabdullin’s post below.

2 Likes

The way I understand it is that it happens every 5 minutes and is down-sampled to hourly data after the purge_keep_days value.

1 Like

I am using a Starline integration.
A “mileage” sensor did not have “state_class” defined.
I manually made this sensor “LTS-able”:

homeassistant:
  customize:
    sensor.auto_xxx_mileage:
      state_class: measurement

and got this:
image

i.e. it is possible (may be for some integration) to define “state_class: measurement” for not-template sensors.

(only in 2024.9 this sensor got a native “total_increasing” class)


I have a sensor with this default “friendly_name” provided by a Traccar integration:
image

After this customization:

homeassistant:
  customize:
    sensor.battery_traccar_iphone_12:
      friendly_name: my super battery 

it looks like

i.e. customizing “friendly_name” via “customize” is possible for sensors at least for some integrations.

1 Like

Try it with an MQTT Sensor.

Do not have any mqtt; may be it is mqtt-specific…

I think that calculating of statistical data happens every 5 minutes; these “5minute-data” are stored as “short-term statistics” and purged after “purge keep days” interval.
Every hour an “hourly data” are calculated and stored as “long-term statistics” and never purged. I.e. that “downsampling” happens every hour immediately after a sensor is created (not “after the purge_keep_days”).

Sorry for that. I was not aware of it anymore. I tried so many different approaches that I forget that “detail”.

Thank you soooo much for that piece of information! :slight_smile: That is really helpful knowledge. Now I know that I do not have to go in the customize direction anymore where I already spent a lot of time trying out LTS enabling.

I read a lot threads about LTS problems. And it is often stated that you should use customize in case the state_class is missing. So there is a lot misinformation out there. Maybe it was right at the time it was written but tough to know without the background knowledge of legacy and new sensors.


For me it is a general a problem in the home assistant architecture that I’m not able to decide on my own, which third party sensor shall be in the LTS and which not. The developers of integrations cannot know which sensors the users like to have in their LTS. And each user will have a different opinion. So the user shall be able to configure for each sensor whetehr it is included in LTS or not.

But ok. It is what it is and I will have to duplicate all needed sensors with trigger sensors and enable LTS for the duplicates.

I have the problem also for sensors of the Utility Meter integration. Even though the sensor Utilty Meter is using as source includes state_class and has a working LTS. Adding state_class to the Utility Meter Sensor via customize does not work/help for enabling LTS.

So the problem seems not to be bound to MQTT but to be a more general one. Maybe your example sensors are legacy ones?

For me the behavior of HA feels like a bug. But it might also be a feature.

P.S.
I’m using HA version 2024.8.3

See my example above - I can define “state_class” for a sensor of some not-template integration.
Also, I defined “state_class: none” in some cases - for example, some GPS position integration (Traccar server) defines “battery_level” sensors with “state_class: measurement”. Imho this is useless - so I made it “none”.
Repeat, defining “state_class” in “customize” does work for SOME not-template integration, can say nothing about mqtt and all other integrations.

1 Like

Adding state_class via customize.yaml is a “hope for the best but expect the worst” situation.

Well, this turns to be a situation similar to this:
“customize” - is it going to be deprecated?
i.e. causes doubts like “smth is working, have never been declared as unsupported - and may stop working some day”.
Are there any official info in Docs that users should not use “customize” for defining “state_class”? For defining “friendly_name”? I do not recall this info in Docs. As a result - users are confused.

The documentation for Manual Customization is skimpy and the ability to perform any of it via the UI was removed several versions ago. My conclusion is that this is a feature the development team has purposely demoted.

What I mentioned above was learned empirically. For example, I tried adding a friendly_name attribute to a modern format Template Sensor and it’s visible but not handled as the sensor’s friendly name. EDIT It does work; see post below.

Hope it will be “legacy” (i.e. at least a working solution).

This would be an off-topic here since the thread is about LTS, but anyway:

input_number:
  test_sensor_with_friendly_name:
    min: 0
    max: 10
    step: 1
    mode: slider

template:
  - sensor:
      - name: test_sensor_with_friendly_name
        state: >-
          {{ states('input_number.test_sensor_with_friendly_name')|int(default=0) * 2 }}

      - name: test sensor with friendly_name (2)
        state: >-
          {{ states('input_number.test_sensor_with_friendly_name')|int(default=0) * 4 }}

homeassistant:
  customize:
    sensor.test_sensor_with_friendly_name:
      friendly_name: Hello world
    sensor.test_sensor_with_friendly_name_2:
      friendly_name: Goodbye world


Here modern template sensors have customized “friendly_name”.
Note that just in case the 2nd sensor was declared with some “nice name” which was then slugified to create “entity_id”; I usually use a “name” option to define “entity_id” explicitly.

2 Likes

Thanks for the clarification; I remember it didn’t work when I tried it (a long time ago) but clearly it does work so I must have misremembered the test case and/or results. :man_shrugging:

Anyways, state_class doesn’t appear to work for MQTT sensors so manual customization can be a YMMV situation.

1 Like

I see; have no experience with mqtt, cannot test unfortunately.