History stats stops working for app. 1 week

Configuration Frontend
1

Hi

I posted a similar post earlier but without any response - now I try once more with more information.

I am running a presence simulation that turns lights on/off as last week.
For this purpose, I use history stats to generate a sensor.

I works perfectly apart that for some random periods (typically one week duration) it stops working.

The image below shows my repetition sensor of two of my lights together with the actual light situation one week earlier. As can be seen, at some point in time the repetition sensor starts to only turn on very shortly and then off again.

image
image502×724 21.4 KB

Here is a cut of my sensor yaml-code:

  - platform: history_stats
    name: "Vacation Light3"
    entity_id: sensor.light3_filtered
    state: "on"
    type: count
    start: "{{ (as_timestamp(now()) - 604800) }}"
    duration: "00:01:00"

I run on the internal SQLight database of Home Assistant and consider moving to MariaDB.

2

Not sure why that is happening but did you know there is an integration to do this?

3

If you do want to debug it, I’d start with looking at the graph of the history_stats sensor, and then also show whatever code you’re using to turn the light on based off that sensor

4

Thanks for your reply.
The debug information is what I show on the graphs inserted above.
The output of the history stats is the orange graph and the input is the blue.

The orange graph shall follow the blue graph, and it typically does except for some periods (as shown above). The blue graph is offset by one week (-7d) to clearly show the faulty output of the history stats.

5

Thanks for your reply.

My implementation is older than this integration and this is the first reason why I do not use it :wink:
But the most important reason is that my lights are often “unavaliable” due to being switched off by a power switch. In the presence simulation this state need to be converted to “off”. I did not find this feature supported by the integration.

6

That could be what is messing up History Stats.

7

No, because for the entity “sensor.light3_filtered” shown below the “unavailable” state has been converted to “off”.

The filtering is done as shown below - it can only be in state “on” or “off”:

      - name: "Light3 Filtered"
        state: >
          {% if state_attr('light.flowerpot', 'brightness') == None %}
            off
          {% elif state_attr('light.flowerpot', 'brightness') > 20 %}
            on
          {% else %}
            off
          {% endif %}

And note, that this is running OK most of the time :wink:
And that the error occurs during the same period for several independent lights.

8

Please show the home assistant history graph for sensor.light3_filtered for the time range in your graphs above.

9

I hope the picture below shows it.

The “Vacation Light3” is set to the number of “on” states of “Light3_filtered” 7 days ago.

My light 3 is typically turned on every night and therefore shows almost a periodical behavior. But starting from 7th of october, it only seem to count when the rising edge of “filtered” (endge from off to on) is within the one-minute window of the history stats.

Typically it will work correctly again in a couple of days (it is typically out for 1 week)

image
image1643×640 43.5 KB

  - platform: history_stats
    name: "Vacation Light3"
    entity_id: sensor.light3_filtered
    state: "on"
    type: count
    start: "{{ (as_timestamp(now()) - 604800) }}"
    duration: "00:01:00"
10

And looking back in history, this is the 4th occurance since august 1st, this year. Everytime almost for 1 week.

image
image1645×557 46.1 KB

11

This picture shows that it happens to all 17 lights at the exact same time. This indicates to me that it is not related to the state of the individual light but a general issue with the history stats.

image
image1639×660 129 KB

12

The time frame for your history stats is very short. It has been a while since I las tried, but my experioence is that history stats on a binary sensor is not reliable when there are no states recorded in the timeframe that the history stats measure or in the period before.

And since the recorder does it best lately not to record anything if the state does not change, it is very likely that history stats has no states in the duration that you supply.

Another thing: did you check the recorder settings to see how long states are kept, and of these periods coincide with purge of history? There must be history to work with, long term starts are not used.

13

Thanks for your reply.

I get your point but I am struggeling to understanding why this only happens from time to time and during some windows that are synchronized between all 17 sensors.

Do you think it would make any changes if I updated to a numerical sensor instead of a boolean sensor?

My thoughts were more in the direction that maybe the subtraction of 604800 seconds from the timestamp ended up in some underflow/overflow situation internally in the history stats implementation. I don’t know if there is a relation between the app. 7 days duration of the outage and the 7 days delay of the history stats (604800 seconds).

I initially used 10 s. duration and changed to 1 min. but that didn’t change anything back then.

  - platform: history_stats
    name: "Vacation Light3"
    entity_id: sensor.light3_filtered
    state: "on"
    type: count
    start: "{{ (as_timestamp(now()) - 604800) }}"
    duration: "00:01:00"

I have 14 days of history which should be sufficient:

recorder:
  purge_keep_days: 14
14

If your recorder settings are set too low, then a purge of the database removes your history, and that happens periodically. So that may very well be it.

15

I have 14 days of history which should be sufficient:

recorder:
  purge_keep_days: 14

I expect that this guarantees a minimum of 14 days history?

16

Is there variation in how long the lights are on for, or the brightness? Maybe how long the history stats take into account to look for previous states factor in.

Maybe put the current minute in the attribute of the filtered value when it is on, to force state updates to the recorder and see if that fixes it for ine light?

17

Like previous occations, everything starts working as inteded after app. 7 days. Nothing is changed in the meantime.

Are there anyone in here with knowledge about the implementation of statistics? Or is there some official place to report such situation?

image
image1624×669 46.7 KB

18

Thanks for the hint.
I have upgraded some of my lights to attach currrent timestamp as an attribute.
Now I just have to wait till the next outage window to see if this makes any difference.

19

The problem is that your duration is asynchronous to your light activation.

So the light might turn on 0.1s before the end of your specified 1 minute duration then at the end of the duration the count resets. So you only get a 0.1s spike.

The best way around this would be to forget the history stats sensor and use an SQL sensor to read the past values instead. More info and examples here:

20

You can report issues on GitHub but I think you’ve got some work to do yet before you have anything that someone can actually investigate.

Graphs of a weeks’ worth of data are good to show that something is wrong, but you need to dig down into the exact timestamps for when something happened or didn’t happen and explain what you expected compared to what the history shows. Pick a moment where the history stats sensor turned on only for a brief moment: list out the timestamps for exactly when it turned on and when it turned back off, and then go back a week prior and show what the status of the light was for the one minute on either side of those timestamps.

That would be the bare minimum to start investigating this issue.