Binary Statistics `average_step` Weirdness

[jeffeb3]

• I have a binary sensor binary_sensor.heating. The state changes to On when it starts heating and Off when it stops. Sometimes, it is unknown.
• I want statistic measurements that tell me more about how much heat I am using. So I have been using binary statistics.

The config looks like this:

sensor:
  # Heater
  - platform: statistics
    name: "Heating over last 24 hours"
    entity_id: binary_sensor.heating
    state_characteristic: average_step
    max_age:
      hours: 24
    sampling_size: 1000
  - platform: statistics
    name: "Heating count over last 24 hours"
    entity_id: binary_sensor.heating
    state_characteristic: count_on
    max_age:
      hours: 24
    sampling_size: 1000

I haven’t had any heating in the last 24Hrs, so my states currently look like this:

1. The count over 24h is correct. But the percentage is not. There haven’t been any On or Off values in the last 24hrs, so I think that is why the percentage is unknown. It isn’t looking at the previous message to know if it has been on this entire time, or off. So maybe that is not a bug, but it is a missing feature. Ok…

Let’s look at the history for the last week:

The count (on the bottom) looks correct.

2. The percentage (in the middle, the top graph) is wildly incorrect though. The actual amount of heat I am using (in this springtime climate) is less than 5%. The percentage goes as low as that sometimes, but then has these steady periods of 30-40%. And when the heat turned off for more than a day, it shows it stuck at 100% (is this just a case where the “unknown” is being misinterpreted by the history graph?).

3. The current value is “unknown”, when it should be 0%. I could make a template sensor to correct that, basing it on the current value of heating and the buffer_usage_ratio. That seems a little clunky.

The thermostat is my own arduino/esp32 creation. I am guessing I could make this a lot easier on the statistics module if I periodically (every 6 mins) publish the on/off. IIRC, it is coming via MQTT.

Ultimately, I want these statistics to lead to smarter automations, and long term logging (comparing this year’s heating % to last year’s).

Anyone have any tips? Should I report this upstream to HA as an issue? Am I misusing the statistics module? Is there a better alternative?

[unfug-at-github]

I also had similar issues when using the statistics. I also had a look at the code and as far as I see it, there is some room for improvements there.

As soon as values are stable for a long time (longer than your sampling interval), you have no values on which you can base your computation. The statistics function only uses values that were changed in your sampling interval. In this case you will get an undefined value (none).

Depending on which statistic function you use, you need a different number of inputs. E.g. you don’t need any value to do a count of elements, you will need at least one value for a min or max value and in some cases, you’ll even need more than one.

Sometimes you can avoid undefined values by using the option “keep last sample”, so that you will have at least one value to do your computation. Technically, this value is not belonging to your sampling interval then (e.g. the value may be two minutes old while your sampling interval is one minute only).

In the case of “average_step” the computation will only be done when there are at least two values (meaning two changes or one change and one kept via option). I don’t see why you wouldn’t compute the average of a single value (yielding the value itself) and avoid the undefined value in this case.

Another problem with the way this is computed is that it doesn’t accurately compute a weighted average for the specified interval. I’ll give you an example:

You want to compute the average electrical current provided by a sensor over 10 minutes. The current is stable (meaning not changing at all, lets say at 5A) for almost the whole time. For a few seconds though someone is switching things on and off repeatedly (lets say for six seconds). During these six seconds the current is changing several times. The statistics function will now only look at the values of those six seconds. It will ignore the stable phase before and after the six seconds and compute an average based on the six second interval only. If you look at the full ten minutes the average current would be close to the 5A (which is what most people would expect). The statistics function however could give you values significantly lower (e.g. 1A) depending on how long the device was on and off in the six second interval.

The “average step” computation is okayish when you have a situation where you have a large number of value changes in your interval, so that ignoring the borders (the time before the first and the time after the last change) will not have much of an effect.

For everything else, it would be a lot better to have a computation that is considering the full sample interval, which would automatically mean you always have to keep the latest change to know which value was measured at the beginning of your interval.

What is really annoying is that the statistic values are undefined frequently when you have sensors that change very little. I had to build templates that would switch between the computed average value when it was there and a stable input value when the statistics were undefined.

I guess the very least thing that should be done is allowing an average of a single value. In combination with the “keep last sample” option this would avoid the undefined values at least.