Long term statistics: sensors purged after 10 days although state_class set: attribute order a problem?

[kruste]

Hi,

I have the problem with several sensor from different sources:

1. The state_class is set correctly
2. The sensor is shown in the Development tools / statistic section
3. Nevertheless, the data is purged after 10 days

This happens mostly to MQTT sensors, but also e.g. for Utility Meter sensors. What I observe (might be a coincidence): the long term data is always missing in case the state_class attribute is not shown as very first attribute in the Development tools / statistic section.

Example of a sensor which has the attribute not as first attribute in Development tools / state:

It is shown with no issue in the Development tools / statistic section

But the data is still purged after 10 days and the “Source” information is missing when hovering:

The same is true when I use the statistic graph card.

---

For a different sensor from the same MQTT topic and even the same device everything works fine:

For this sensor I can see all data unpurged and also the typical “Source: Long term statistics” information when hovering in the graph.

---

I have this problem for a lot of sensors (also for different units, measurement, total etc.). And not even only MQTT but also Utility Meter as source.

What I observe: for the MQTT sensors the problems always seems to occur in case the state_class attribute is not shown as the first one in the attribute list. That might be a coincidence but is the only difference I see (yes the units are different but the problem also occurs for other units).

Attribute order or not… something is not working despite a correct set state_class. When I create my own sensor which simply copied the values from the original one, my copy works. It has exactly the am attributes but is shown in different order.

Has anybody an idea what the problem could be or how to finds the root cause (e.g. checking with SQL queries whether the data is in the LTS table or not)?

[kruste]

@kruste

When I create my own sensor which simply copied the values from the original one, my copy works. It has exactly the am attributes but is shown in different order.

To make this statement a bit clearer:

The second sensor in the following screenshot is the original one, for which the LTS is not working. The first one is my “copy” with exactly the same attribute. This one is fine, the data is not purged.

They have the same attributes. Nevertheless one is purged and the other not.

The copied sensor is created with a template trigger:

template:
  - trigger:
    - alias: vk
      id: vk
      entity_id: sensor.wolf_wp_cha_0x3_270048_verdichterfrequenz
      platform: state
      not_to:
      - unknown
      - unavailable
    sensor:
    - name: vk_cha_0x3_270048_verdichterfrequenz
      unique_id: vk_cha_0x3_270048_verdichterfrequenz
      unit_of_measurement: "Hz"
      state_class: "measurement"
      icon: "mdi:sine-wave"
      state: >
        {{ states('sensor.wolf_wp_cha_0x3_270048_verdichterfrequenz') }}

[123 (Taras)]

Which integration produces this entity?

sensor.wolf_wp_cha_0x3_270048_verdichterfrequenz

If it’s the MQTT integration, is the entity configured via YAML or MQTT Discovery? Post its configuration (if available).

[jackjourneyman (Jack)]

The question is a bit confusing…

When you create a new sensor, data will be purged after 10 days (or whatever you’ve set purge_keep_days to).

Recorder

If the sensor state_class is measurement, total or total_increasing, sensor data will also be saved to long term statistics. This happens once an hour, so you’ll get 24 entries a day.

Long- and short-term statistics | Home Assistant Data Science Portal

The two are different, and are saved separately. Short-term data are still purged; long-term statistics are never purged.

[kruste]

@123

Which integration produces this entity?

sensor.wolf_wp_cha_0x3_270048_verdichterfrequenz

If it’s the MQTT integration, is the entity configured via YAML or MQTT Discovery? Post its configuration (if available).

The data is created by a stand alone application (ism7mqtt) which sends data to MQTT. I guess MQTT is set to auto discovery. I never did anything to add a new sensor. They appear automatically.

@jackjourneyman

The question is a bit confusing…

Sorry for that.

Regading your answer:

I know that that short term and long term two different statistis (even DB tables). But as you can see in my screenshot the state_class is set correctly to measurement.

So why it is not working? Why do I see long term statistics for some sensor and not for others with exactly the same configuration?

[123 (Taras)]

In that case, I suggest you contact the author of the custom integration (ism7mqtt) and report the issue.

GitHub - zivillian/ism7mqtt

[kruste]

What comes into my mind:
It is possible, that the state_class attribute of all not working sensor are added by customize in configuration.yaml by myself. I have to cross check that tomorrow.

Maybe “just” the customization does not work when trying to add state_class for enabling LTS.

[123 (Taras)]

@kruste

are added by customize in configuration.yaml by myself

You overlooked to mention this important detail in your first post.

That trick works for Template Sensors configured using legacy format which don’t natively support the state_class option. It’s not known to (always) work for entities created by other integrations.

The state_class attribute you added via customize.yaml is being treated like any other attribute (i.e. not being used for long-term statistics).

Contact the author to add a proper state_class option to the entity.

---

FWIW, the same thing happens to users who attempt to add friendly_name to a sensor entity via customize.yaml. It appears in the entity’s list of attributes but it is not handled as the entity’s friendly name. EDIT See Ildar_Gabdullin’s post below.

[tom_l]

@jackjourneyman

If the sensor state_class is measurement, total or total_increasing, sensor data will also be saved to long term statistics. This happens once an hour, so you’ll get 24 entries a day.

The way I understand it is that it happens every 5 minutes and is down-sampled to hourly data after the purge_keep_days value.

[Ildar_Gabdullin]

@123

That trick works for Template Sensors configured using legacy format which don’t natively support the state_class option. It’s not known to work for entities created by other integrations.

I am using a Starline integration.
A “mileage” sensor did not have “state_class” defined.
I manually made this sensor “LTS-able”:

homeassistant:
  customize:
    sensor.auto_xxx_mileage:
      state_class: measurement

and got this:

i.e. it is possible (may be for some integration) to define “state_class: measurement” for not-template sensors.

(only in 2024.9 this sensor got a native “total_increasing” class)

---

@123

the same thing happens to users who attempt to add friendly_name to a sensor entity via customize.yaml. It appears in the entity’s list of attributes but it is not handled as the entity’s friendly name.

I have a sensor with this default “friendly_name” provided by a Traccar integration:

After this customization:

homeassistant:
  customize:
    sensor.battery_traccar_iphone_12:
      friendly_name: my super battery 

it looks like

i.e. customizing “friendly_name” via “customize” is possible for sensors at least for some integrations.

[123 (Taras)]

Try it with an MQTT Sensor.

[Ildar_Gabdullin]

Do not have any mqtt; may be it is mqtt-specific…

[Ildar_Gabdullin]

@tom_l

The way I understand it is that it happens every 5 minutes and is down-sampled to hourly data after the purge_keep_days value.

I think that calculating of statistical data happens every 5 minutes; these “5minute-data” are stored as “short-term statistics” and purged after “purge keep days” interval.
Every hour an “hourly data” are calculated and stored as “long-term statistics” and never purged. I.e. that “downsampling” happens every hour immediately after a sensor is created (not “after the purge_keep_days”).

[kruste]

@123

You overlooked to mention this important detail in your first post.

Sorry for that. I was not aware of it anymore. I tried so many different approaches that I forget that “detail”.

@123

The state_class attribute you added via customize.yaml is being treated like any other attribute (i.e. not being used for long-term statistics).

Contact the author to add a proper state_class option to the entity.

---

FWIW, the same thing happens to users who attempt to add friendly_name to a sensor entity via customize.yaml. It appears in the entity’s list of attributes but it is not handled as the entity’s friendly name.

Thank you soooo much for that piece of information! That is really helpful knowledge. Now I know that I do not have to go in the customize direction anymore where I already spent a lot of time trying out LTS enabling.

I read a lot threads about LTS problems. And it is often stated that you should use customize in case the state_class is missing. So there is a lot misinformation out there. Maybe it was right at the time it was written but tough to know without the background knowledge of legacy and new sensors.

---

For me it is a general a problem in the home assistant architecture that I’m not able to decide on my own, which third party sensor shall be in the LTS and which not. The developers of integrations cannot know which sensors the users like to have in their LTS. And each user will have a different opinion. So the user shall be able to configure for each sensor whetehr it is included in LTS or not.

But ok. It is what it is and I will have to duplicate all needed sensors with trigger sensors and enable LTS for the duplicates.

[kruste]

@Ildar_Gabdullin

Do not have any mqtt; may be it is mqtt-specific…

I have the problem also for sensors of the Utility Meter integration. Even though the sensor Utilty Meter is using as source includes state_class and has a working LTS. Adding state_class to the Utility Meter Sensor via customize does not work/help for enabling LTS.

So the problem seems not to be bound to MQTT but to be a more general one. Maybe your example sensors are legacy ones?

For me the behavior of HA feels like a bug. But it might also be a feature.

P.S.
I’m using HA version 2024.8.3

[Ildar_Gabdullin]

@kruste

And it is often stated that you should use customize in case the state_class is missing

See my example above - I can define “state_class” for a sensor of some not-template integration.
Also, I defined “state_class: none” in some cases - for example, some GPS position integration (Traccar server) defines “battery_level” sensors with “state_class: measurement”. Imho this is useless - so I made it “none”.
Repeat, defining “state_class” in “customize” does work for SOME not-template integration, can say nothing about mqtt and all other integrations.

[123 (Taras)]

Adding state_class via customize.yaml is a “hope for the best but expect the worst” situation.

[Ildar_Gabdullin]

Well, this turns to be a situation similar to this:
“customize” - is it going to be deprecated?
i.e. causes doubts like “smth is working, have never been declared as unsupported - and may stop working some day”.
Are there any official info in Docs that users should not use “customize” for defining “state_class”? For defining “friendly_name”? I do not recall this info in Docs. As a result - users are confused.

[123 (Taras)]

The documentation for Manual Customization is skimpy and the ability to perform any of it via the UI was removed several versions ago. My conclusion is that this is a feature the development team has purposely demoted.

What I mentioned above was learned empirically. For example, I tried adding a friendly_name attribute to a modern format Template Sensor and it’s visible but not handled as the sensor’s friendly name. EDIT It does work; see post below.

[Ildar_Gabdullin]

@123

My conclusion is that this is a feature the development team has purposely demoted.

Hope it will be “legacy” (i.e. at least a working solution).

@123

For example, I tried adding a friendly_name attribute to a modern format Template Sensor and it’s visible but not handled as the sensor’s friendly name.

This would be an off-topic here since the thread is about LTS, but anyway:

input_number:
  test_sensor_with_friendly_name:
    min: 0
    max: 10
    step: 1
    mode: slider

template:
  - sensor:
      - name: test_sensor_with_friendly_name
        state: >-
          {{ states('input_number.test_sensor_with_friendly_name')|int(default=0) * 2 }}

      - name: test sensor with friendly_name (2)
        state: >-
          {{ states('input_number.test_sensor_with_friendly_name')|int(default=0) * 4 }}

homeassistant:
  customize:
    sensor.test_sensor_with_friendly_name:
      friendly_name: Hello world
    sensor.test_sensor_with_friendly_name_2:
      friendly_name: Goodbye world

Here modern template sensors have customized “friendly_name”.
Note that just in case the 2nd sensor was declared with some “nice name” which was then slugified to create “entity_id”; I usually use a “name” option to define “entity_id” explicitly.