Statistics Sensor Issue

ttaidapos · April 7, 2020, 1:57pm

Ok this one is bugging me for a while and can’t troubleshoot the sucker. I have a bunch of statistics sensors and they’re all calculating perfectly fine, any feedback on what i’m doing wrong will be greatly appreciated!

Environment
Rpi4, Raspbian, 107.5

Issue
I’m interested in the “change” attribute in the stats sensor only, it’s not calculating correctly no matter what sampling size i’m using, as you can see below i’m using 20k, originally set at 3600. The simple arithmetic doesn’t make sense. All other stat sensors are calculating correctly. Mind you this is only for deaths not calculating the change correctly, the confirmed change value is correct.

Sensor Source:
Corona-virus integration from UI

Greece Coronavirus deaths
sensor.greece_coronavirus_deaths

Greece Coronavirus confirmed
sensor.greece_coronavirus_confirmed

Statistics Sensor

- platform: statistics
  name: cov19_gr_confirmed
  entity_id: sensor.greece_coronavirus_confirmed
  sampling_size: 20000
  max_age:
    days: 1
- platform: statistics
  name: cov19_gr_deaths
  entity_id: sensor.greece_coronavirus_deaths
  sampling_size: 20000
  max_age:
    days: 1

Actual data

Example problem

Summary
The confirmed shows a change of 20 which is correct. For deaths I’m seeing 5 but the answer should be 8. Seems that I’m always off by 3!

jocnnor · April 7, 2020, 2:09pm

You are telling it to only compare 1 day worth of values. (max_age 1 day)

Looking at the graph, it says it was 73 deaths on April 6, 8:55 AM.

Also looking on the graph, it looks like it’s pretty close to 8:55 AM the next day (can’t tell exactly). The next value from 73 is 76. 81-76 = 5.

The problem is probably that the history graph shows a little more than 1 day while the stats sensor is calculating exactly one day.

Here is the code for the ‘change’ attr. It’s literally the last value minus the first value. It shouldn’t be wrong.

self.change = self.states[-1] - self.states[0]

ttaidapos · April 7, 2020, 3:24pm

You’re absolutely right, I believe the gap of 3 was just a recurring coincidence. The graph is probably not as precise as the stat sensor. The thought behind my setup was that since the graph only shows 24 hours, I would have the stat sensor match the same time-frame. Do you happen to know the exact time frame of the graph? On the UI side, the graphs seems to be 24hrs exactly.

jocnnor · April 7, 2020, 3:35pm

Well, now I’m curious.

I’m assuming the touch graph is just this one…with a default of 24 hours.

And when I pull up one of my graphs, it shows exactly 24 hours.

I think the problem is more related to how often the sensor the stats integration it’s tracking is updated and there being some room for error there. I’m not exactly sure how it gets the 1 day thing. If it only polls every hour or so, then I could see a 26 hour window in the sensor being possible.

I’d have to look at the code more to find out.

ttaidapos · April 7, 2020, 4:21pm

Really appreciate you looking into this. This is obviously nothing critical just a gap I noticed I’ve played around with the sampling number a bit and the 3600 seems to be a common setting for any sensors that update every 30sec which definitely covers the 2,880 precision of updates in a given 24hr period.

ttaidapos · April 7, 2020, 4:38pm

Using the screens below, value of change should have remained at 5 (it was five a bit of time ago) and now it’s 2. I’m wondering if it has anything to do with hass using UTC vs. local timezone? Looking at the stat sensor, it’s not going back 24hrs but only 20hrs? The other idea is how max_age calculates dates.
The greece coronavirus values don’t update as often as say other sensors so this would explain why the other sensors i’m using are calculating correctly. Should I attempt to change sampling to hours vs. days?

min_age: 2020-04-06T17:26:29.346408+00:00
max_age: 2020-04-07T13:51:12.258641+00:00

jocnnor · April 7, 2020, 4:38pm

Looking at the code, it actually handles purging in 2 ways.

Every sensor update it will purge old data (now() - age > max_age).
After an update, calculate the time when it will expire and schedule an automatic update at that time. This is in case sensors aren’t updating, it will still purge old data.

The age is the state.last_updated time, so it should all be pretty accurate.

There is some console debugging that will show the purge dates. If you turn on debug for this integration, you should be able to see when the next purge dates are.

ttaidapos · April 7, 2020, 8:11pm

Not sure if you noticed the above post… Looks like other sensors are having the same time-issue. Basically, any sensor that doesn’t change the value, will fall into this same problem. If the value changes, then the issue is not as prominent.

jocnnor · April 7, 2020, 11:08pm

Oh, that makes sense. Couldn’t find anything wrong in that code

mookkari · August 25, 2021, 4:25pm

I noticed this same issue in my statistics sensors that track energy consumption during last 24h period. Issue is that amount of change of first sample is always ignored.

Let me try to explain using same images as last posted be user ttaidapos.

There are two samples during last 24 hours:

First sample at 17:26:29 when 3 people died (value increased from 76 to 79)
Second sample at 13:32:04 when 2 people died (value increased from 79 to 81)

We agree that totally 5 people has died during 24h period.

Statistic sensor will show value of change as 2 because code will take latest sample and decrease it from the end value of first sample.

self.change = self.states[-1] - self.states[0]
= self.change = 81 - 79 = 2

I think change should be 5 in this case, first deads increased from 76 to 79 and then from 79 to 81, total change from 76 to 81 = 5. But first change is ignored in code.

You can also test this with creating new statistic sensor and add only one sample with change, e.g. value is changing from 10 to 20. Change value is shown as 0, but I think it should be 10. This is because initial value 10 has been set outside period and therefore not registered. Also other value like Mean, Median, Min, Max are incorrect if you compare sensor values during same period.

Also problem with statistics sensor is that most statistic values will go to “unknown” state if there are no samples during max_age period. This happens when statistics sensor is created and also when time pass to period without any samples.

I was looking the code and it seems that state object does not know previous value (same way as trigger knows from_states and to_states values).

I propose as code fix for issues above to get previous (=current) value of source sensor (entity_id) outside period and and save is as beginning value of period. This value could be set to other values like Mean, Median, Min, Max as well to prevent “unknown” values when there are not samples during period.

“”“Initialize the Statistics sensor.”""
self._beginning_value = [Current value of sensor entity_id]
self.min = self.max = self.mean = self.median = self._beginning_value

Change code in async_update(self):
self.change = self.states[-1] - self._beginning_value

In purge self._beginning_value should be updated from latest purged sample.