Need for state_class "delta" that is neither "measurement" nor "total"

1

I have a sensor that tells me the amount of water consumed since the last sensor query.
The device_class is presumably volume since the amounts consumed between readings are ‘volumes’.

However, the state_class is not total (or total_increasing`) since the amounts read are deltas (since last measurement) rather than totals.

Also, the state_class is not measurement since it is not necessarily gallons per minute (unless the reading is exactly once per minute).

It seems like there should be a 3rd type of state_class of type delta – in that sense it is a bit more like a total than a measurement since ultimately the statistics are quantity-like, however, unlike total (or total_increasing) you would need to add the delta to the previous ‘sum’ to get the new value of sum

This seems like a very common use case yet, surprisingly it doesn’t seem to exist.
For example, if I wanted to see how much gas my car has consumed, I would input number of gallons added each time I fill up a the gas station. This is neither a cumulative ‘total’ nor a ‘gallons-per-minute’ measurement. And the appropriate statistic to track would be the ‘sum’ of cumulative fills.

More generally, ‘delta’ could be positive or negative of course.

Now of course, one could generally convert from total to delta and vice-versa but that shouldn’t be necessary and HA should allow one to use the ‘raw’ form of the data.

1 Like
2

I don’t see what is wrong with using measurement with units gallons. And you can feed that into a utility_meter helper (with the delta_values option enabled) and you’ll get an entity that tracks the sum.

2 Likes
3

I believe I got an error when combining gallons with measurement saying they were incompatible – indeed, the docs say that:

“The state represents a measurement in present time , not a historical aggregation such as statistics or a prediction of the future. Examples of what should be classified measurement are: current temperature, humidity or electric power.”

Also, why do I need to use utility_helper to get an entity that tracks the sum.
It seems to me that “delta” is itself a fundamental measure just like ‘total’ or a ‘measurement’ variable.

If I wanted to create a derived entity, I could do that by manipulating the sensor data itself…

4

Even if there were a state_class called delta you’d still need another sensor to calculate the sum. Since the data you have is delta data, you have to manipulate it to get the sum. There’s no way around that.

gallons is definitely valid with the state_class set to measurement. In fact I don’t think there are any units of measurement supported by HA that are incompatible with measurement.

I don’t think it’s a terrible idea to have a state_class that indicates a value is a delta value, but my point is that HA would do nothing different between measurement and delta. The statistics would be calculated the same way, the supported units of measurement would be the same, etc. Which means the usefulness of that additional state_class is questionable. It doesn’t hurt anything, but it doesn’t really help either. It becomes merely an indicator to a human of what the value represents. HA could care less.

5

Well, it’s actually already in the docs as to how delta should be handled. I was incorrect, you shouldn’t set it as a measurement. You should set it as total with it resetting on every state change.

  • The sensor’s state is reset with every state update, for example a sensor updating every minute with the energy consumption during the past minute: state class total, last_reset updated every state change.
6

Ahhh. That seems like a bit of a hack but I see how it works as a delta becomes a total if you consider the total to be reset after every reading.
But would this require some type of automation to set last_reset to last_updated_ts for every new state?
If so, that seems pretty klugey for something that seems very natural.

Also, would the statistics sums be correct then?

(BTW thanks for all the helpful advice here)

7

My suggestion was that for a new state_class of delta, the statistics module would know to add previous sum to current state to get the new sum – just like statistics module knows to calculate avg/min/max for measurements vs. sum for totals, it would know how to calculate sums for deltas. In fact, the notion of sum in some way makes more sense for delta since for a total the sum is synonymous with the total itself (unless a reset occurs) and is not very meaningful as a separate statistic.

The actual coding changes for this should be pretty trivial.

8

That should be handled by whatever integration created the sensor, assuming the integration is properly coded.

If it is an MQTT sensor, the definition is documented here.

If it is a template sensor, I haven’t tried, but it should be possible to just define last_reset as an attribute. I may try this at some point to see if it works.

OK I understand now. It wasn’t apparent to me that you wanted the statistics; I was thinking you wanted the total available as the state of a sensor. I think, and again this is conjecture, but I think the statistics should be calculated correctly for this “delta” scenario if the last_reset attribute is updated properly.

9

Oh cool! And thanks for the link.
Yes it is an MQTT sensor.
I tried adding last_reset_value_template for the relevant MQTT sensor definition (as per the docs) but it fails to update either last_reset or last_reset_ts.
Specifically I publish the following to my Mosquitto broker.

{
  "name": "Water",
  "unique_id": "water-monitor",
  "state_topic": "+/MySensors/sensors/waterplug/water",
  "value_template": "{{ value_json.gallons | is_defined }}",
  "device_class": "volume",
  "unit_of_measurement": "gal",
  "state_class": "total",
  "last_reset_value_template": "{{ now().isoformat() }}",
  "device": {
    "identifiers": [
      "watermonitor"
    ],
    "name": "WaterMonitor",
    "model": "Water Monitor",
    "sw_version": "0.5.0"
  }
}

I expected the template line to set the last_reset to the current time each time a new state is entered.

But in retrospect, not sure how that would even work since last_reset and last_reset_ts belong to the table schemas for statistics and statistics_short_term and not the states table so not sure how updating this as part of a sensor update (which affects the states table) would make its way to the statistics tables.

Said another way, last_reset would only appear once per hour in the statistics table while I actually need to reset it for every sample in the states table which could occur as frequently as once per minute.

I fear I am misunderstanding how to use this variable.
Is it possible that this variable won’t work with the MQTT broker but needs to be configured via yaml directly?

10

last_reset exists as an attribute, and therefore you’ll find it in the state_attributes table along with all the other attributes that you’re setting with that same block of code.

Are you sending that code to the discovery topic? If so, you should expect errors because your value_template will render into a boolean, and you’ve set a unit_of_measurement so HA is expecting a numeric value. Check your logs for errors.

I haven’t done what you are attempting to do, but based on the docs I’d expect what you have to work, provided you fix the unrelated bugs.

11

I did some testing with a trigger-based template sensor:
Setting an attribute called last_reset to the current datetime worked as desired: it made the statistics generate a sum that calculated like the value being entered was a delta value.

In the statistics_short_term table, the last_reset column is always empty, but the last_reset_ts does get populated.

image
image1067×57 8.71 KB

For reference, this is how I set up the template sensor:

Click Here for template sensor details 
- trigger:
    - platform: event
      event_type: set_delta_sensor
  sensor:
    - name: Delta Sensor
      unit_of_measurement: "gal"
      device_class: volume
      state_class: total
      state: "{{ trigger.event.data.get('value', this.state) | float(this.state) }}"
      attributes:
        last_reset: "{{ now() }}"

I also tried all these attribute formats, and they all worked also:

          last_reset: "{{ now().isoformat() }}"
          last_reset: "{{ trigger.event.time_fired }}"
          last_reset: "{{ trigger.event.time_fired.isoformat() }}"

I confirmed when the last_reset attribute was not configured in the template sensor, the sensor ignored decreases and only added increases; and the last_reset_ts in the statistics table remained empty.

12

OK - I think I spoke too soon yesterday.
Because on looking on the statistics data today it is indeed working exactly as you demonstrated.

  • last_reset_ts is appropriately populated in the 2 statistics tables
  • sum is cumulative as desired

(I think what tripped me up is that the first couple of statistics entries after restarting didn’t capture the last_reset_ts, presumably because there was no new state)

Also, I didn’t realize that last_reset would be stored as an attribute in states_attributes but indeed it too is *exactly& as you described. Thanks!

So, it does seem to be working just fine now and I believe it does exactly what I needed (though I still don’t love the fact that states_attributes now gets cluttered with a new attribute for every state entry for that sensor whereas it would be so much simpler if one could just define a measurement_class for delta)

THANKS SO MUCH FOR ALL THE PATIENT HELP!

13

Interesting, I am using MQTT so I used the format that I quoted above where I don’t directly set the last_reset attribute but instead set:

"last_reset_value_template": "{{ now().isoformat() }}",

I imagine that the MQTT integration then converts that to a last_reset attribute which in turn is used by the statistics tables.

Note I also tested and confirmed that when multiple states (with different last_reset attributes occur within one statistics interval that the corresponding statistics table shows the last last_reset_ts in the given interval and the sum appropriately adds all updates that occured during that interval.
So, indeed it seems to be working fine.

14

I do have one outstanding question regarding how this will display in the history dashboard.

My understanding is that the history dashboard uses the following sources of date to populate the historical value of a sensor (in descending order of priority)

  1. states table
  2. statistics_short_term table
  3. statistics table

Now what will happen in the case of my water sensor where:

  • states table shows deltas
  • statistics_short_term and statistics tables show sums

It seems like there will possibly be a discontinuity when the states table delta values are no longer available and the history dashboard starts pulling from the statistics sum values.

More generally, it would be nice if the history table displayed the sum all the time as that is what I am interested in (I know I can create a new template sensor of course to represent the sum, but I would prefer not to).

15

I am not seeing any log errors.
I even restarted ha core to see if the ‘unknown’ state would generate an error.

16

The statistics tables (both short term and long term) track both state and sum. However I don’t know what state actually is, in that context. I think it is the last state in that 5-minute (or 1-hour) window. It’s not very useful, but there shouldn’t be a discontinuity.

17

If you want this on a dashboard, you can coos a statistics graph card, and you can select to delay the sum property.

18

But it does seem like there will be a discontinuity – since last state is not the same as sum…

I guess I will know for sure in about 29 days (I retain 30 days)

19

My point is that the history graph should show the following in order of descending priority:

  1. state from states table (recorder database, which has every state change, stored until your recorder purge settings)
  2. state from short_term_statistics table (which are every 5 minutes, stored until your recorder purge settings)
  3. state from statistics table (long term statistics, which are hourly, stored indefinitely)

It shouldn’t display the sum at any point, so there shouldn’t be any discontinuity.

However, if you do want to see the sum, you can use a statistics graph card, and with that card set to show the sum, you will see the sum data from whichever source provides the data from the “Period” you select. For example if you select a period of 5 minutes, you can see the data from the short-term statistics table, and if you select a history longer than your recorder purge settings, it won’t show any data. If you then change to “hourly” or a longer period, you will get data all the way back to whenever that sensor was created in HA because it is coming from the statistics table which is the long-term statistics captured indefinitely.

You can also use the statistics card to view the hourly state data if you want to get a preview of what the history view will show when you are outside your purge settings. No need to wait 29 days, you can just view it now.

20

Ahhhh… that makes total sense.
I forgot that when has_sum is true, that both the state and the sum are recorded vs. when has_mean is true only the mean/min/max are recorded but not the state itself…

I assume that the state recorded when has_sum is true is the last state before the end of the relevant statistics time window.

Thanks again for bearing with me and helping me through all this!!!