History Stats Bug! Please Help!

Configuration
1

I have a history stats template

  - platform: history_stats
    name: Berkan evde Aylık
    entity_id: person.berkan_sezer
    state: "home"
    type: time
    start: '{{ now().replace(day=1).replace(hour=0).replace(minute=0).replace(second=0) }}'
    end: '{{ now() }}'

This template starts from the first day of the month and goes until the end of the month.

I have also deleted my database last night and started with a fresh one.

This template was created today.

The result comes as :

Ekran Resmi 2022-06-12 19.03.15

283,05 hours. This is not possible!!! Where is HA taking that data from ?

No matter what I do it shows my “presence at home” duration as 283,05 which makes early 11 days which also means that I haven’t actually left home this month!!! This is not possible. Please help.

2

You say you deleted your database. I guess the sensor assumes you are home if there is no data in history. But I agree it should have been unavailable up until the time it started recording data again. Do note that the default purge time for the recorder is 10 days. So unless you change that, you will never have enough data for a month.

3

I changed that too. Here is my recorder settings:

recorder:
  purge_keep_days: 60
  exclude:
    domains:
      - updater
    entity_globs:
      - sensor.weather_*
    entities:
      - sun.sun # Don't record sun data
      - sensor.last_boot # Comes from 'systemmonitor' sensor platform
      - sensor.date
      - group.motion_sensors
      - automation.lonely_antre_off
      - automation.lonely_bedroom
      - automation.lonely_dressing_off
      - automation.lonely_kitchen_off
      - automation.lonely_living_room_off
      - automation.lonely_office_off
      - automation.lonely_bathroom
      - automation.unavailable_entities_notification
    event_types:
      - call_service # Don't record service calls

İt is set to 60 days!

4

I would look back in a couple of days to see if the increase is consistent with the value for those days. If so, the problem will be gone next month, as long as you do not delete the database again.

Also check the device trackers linked to your person. I recently had my ipad worm its way in there, and the iPad is always home. It kept me toggling between home and away every few minutes.

5

My ipad is also linked to my device trackers. Thanks for the advice. I have deleted it from the persons entity as well. But still it does not explain where the 283 days data came from ? although I have deleted the database yesterday. Anyway I will check that later in a few days. Thank you very much.

6

By the way I don’t think the problem lies with ipad because I have a HA alarm activated whenever I leave home. If ipad was to give home data to HA, I should also receive Alarm Disabled notification.

7

Not 283 days but but 283 hours = 11.7 days as you noted before. That is indeed pretty much the entire month until now.

I think I’ve seen a feature request saying statistics sensors should not assume that the first value in the history for the period is indeed the value at the start of the period. It can be considered a bug. If enough data is available it is not that much of an error, but in absence of a lot of data such as in your case, it becomes very evident. So you may want to put in a bug report for this.

8

Sorry for the typo yes it is actually days.

I think I go for another way. I will create a boolean for the times I spent at home. So history stats should take the boolean on status for the data. That I see as a much convenient way.

9

Edwin it is very clear now that this is a bug. A friend of mine who has no usage of history stats, tried the same thing and got the number 283 as well. I think for now it is wiser to use booleans instead of directly pointing out to trackers.

10

I’ve got 160 or so hours (data available from the 5th) so it is doing something though, and seeing the value not counting 1 through 4 as present. When I changed the start day to 4 the value did not change, so that is confirmed. If I zoom in on two days I can easily calculate I get the right values too. For me it looks fine.

I am using mariadb as the recorder database, I do not know if that makes a difference. It shouldn’t though.

What is the history graph saying about the entity you supplied?

11

When I choose the dates from the 1st of this month till today, it only shows the last couple of hours. As I said the database has just been created today. So normally it should only give the values from the time it was created. Clearly a bug.

Ekran Resmi 2022-06-12 23.04.13
Ekran Resmi 2022-06-12 23.04.133324×1198 228 KB

12

It’s not a bug, it was a design decision to assume that the first state in your period counts as the duration from the start of your period.

13

So will it turn into normal after a period ? How am I going to track my real monthly duration I spent at home ?

14

It wont fix itself this month because you don’t have history that extends beyond today. If you want to track this months@home, set your datetime to start when you started collecting data. Otherwise it’s going to always add 11 days of your first state to the duration, if that’s the state you’re counting.

You need history for it to calculate, otherwise it makes assumptions. You don’t have the history, so temporarily set it to start on the 12th and then next month switch it to the 1st.

15

Do you know why the 12 missing days are added? I have purge set to 7 days, and when I tried it the purged days were not added. It made sense to me that if the data wasn’t there, that I also wasn’t home. Or was that because by chance I wasn’t home as the first state? It isn’t true this can be extrapolated backwards.

(I know I shouldn’t do this with a period longer than purge settings, but I an curious)

16

Yes, when you start up the history stats sensor looking for X state, but you the very first state in your database is X, it’s going to assume X was the state from the beginning of your configured time window.

v-- start                    v-- X (first in database)            current time--v
|                            |                                                  |
\__________time added________/
17

That does make sense for a new database or new sensor, but not so much for when a database is purged. In that case the first known value represents a change. For numerical is is not a bad assumption per se the value before was quite similar. But for binary sensors it is 100% sure wrong. In both cases unknown is more accurate. That is what also happens if Home Assistant is down for some time.

18

Unless you’re searching for off.

Keep in mind that history stats only knows about the state it’s looking for, nothing else. So there’s only so many ways it can count or not count that duration. And everyone wants something different.

19

I don’t get that. If you assume the first value you find was also there before, and the first value you find is off, then the value before was either on or unknown. Because off was stored as result of a change, and there’s nothing else it could have been. Afaik the database does not sture states unless there’s a change, so because of that the value before was not the value you found as the first.

The fact is that the data needed is not there. So the quick solution would be count it as unknown. The perfect solution would be for the history to either use the last value before the period, or unknown if that does not exist. But that comes at a performance hit, as you cannot optimise the search on the start if you do.

I do not believe every one wants something else, I think people want what is correct. There already is a feature request that amounts to the same that I am saying:

20

Sure, but history stats is old. I have no idea what was going through the dev’s mind when he first implemented it. I can guarantee there was a reason.

It does that. It doesn’t do that when nothing exists before hand, which is the situation we are in.

History_stats is not the same as statistics.

1 Like