History Stats and Long Term Storage - Tracking Visits to places and time (long term)

[bryancp31]

I am trying to enable long term storage of certain sensors (do not purge after the default time), and am having trouble getting them to stick around.

I am using history stats to track my visits to places as well as how many times I visit. Previously I was keeping my entire HA history for 365 days, and this worked for the most part, but my DB got huge and the system started to run sluggishly. I’ve reverted that setup and now use the default purge.

Is there something else I need to do to keep certain stats/sensors for an indefinite amount of time?

Sample of some of the entries in my sensor.yaml file:

- platform: history_stats
  name: Place 1 Trips Total
  entity_id: person.me
  state: "Place 1"
  type: count
  start: '{{ 0 }}'
  end: '{{ now() }}'
- platform: history_stats
  name: Total Time Spent at Place 1
  entity_id: person.me
  state: "Place 1"
  type: time
  start: '{{ 0 }}'
  end: '{{ now() }}'

[Ildar_Gabdullin]

My code

sensor:
  - platform: history_stats
    name: "ildar Home today"
    unique_id: ...
    entity_id: person.ildar
    state: "home"
    type: time
    start: "{{ now().replace(hour=0, minute=0, second=0) }}"
    end: "{{ now() }}"

creates a sensor with state_class=measurement:

i.e. the sensor is kept forever.
Means - these HS sensors have this state_class by default.

[bryancp31]

I have similar code as you, but the values keep being purged after the default time.

I have sensors for trips per month, per year, and ever, and I can’t get them to hold values for more than 7 days.

[Ildar_Gabdullin]

Post a screenshot from Dev tools → State for these history_stats entities which are purged.
Also, are you sure that they are purged? Why do you think so?

[bryancp31]

Thanks for helping me trouble shoot this. Here is a capture from my Dev Tools screen.

I may be using the word purge wrong, but they are all being reset back to 0 after 7 days.
This worked when my ‘purge’ was set to 365 days (although the data dropped off after a year).

Here is the code for that section:

- platform: history_stats
  name: Hollywood Studios Trips This Month
  entity_id: person.me
  state: "Hollywood Studios"
  type: count
  start: '{{ now().replace(day=1).replace(hour=0).replace(minute=0).replace(second=0) }}'
  end: '{{ now() }}'
- platform: history_stats
  name: Time Spent at Hollywood Studios This Month
  entity_id: person.me
  state: "Hollywood Studios"
  type: time
  start: '{{ now().replace(day=1).replace(hour=0).replace(minute=0).replace(second=0) }}'
  end: '{{ now() }}'
- platform: history_stats
  name: Hollywood Studios Trips This Year
  entity_id: person.me
  state: "Hollywood Studios"
  type: count
  start: '{{ now().replace(month=1).replace(day=1).replace(hour=0).replace(minute=0).replace(second=0) }}'
  end: '{{ now() }}'
- platform: history_stats
  name: Time Spent at Hollywood Studios This Year
  entity_id: person.me
  state: "Hollywood Studios"
  type: time
  start: '{{ now().replace(month=1).replace(day=1).replace(hour=0).replace(minute=0).replace(second=0) }}'
  end: '{{ now() }}'  
- platform: history_stats
  name: Hollywood Studios Trips Total
  entity_id: person.me
  state: "Hollywood Studios"
  type: count
  start: '{{ 0 }}'
  end: '{{ now() }}'
- platform: history_stats
  name: Total Time Spent at Hollywood Studios
  entity_id: person.me
  state: "Hollywood Studios"
  type: time
  start: '{{ 0 }}'
  end: '{{ now() }}'

[Ildar_Gabdullin]

OK, seems that these sensors have a proper state_class.
But you have not answered this:

@Ildar_Gabdullin

Also, are you sure that they are purged? Why do you think so?

Why do you think that there is no LTS permanently stored? Have you tried adding a Statistics graph? Also, a History page will show you an “augmented” history (last 10 days of normal history + LTS for ancient times).

[bryancp31]

First, thank you for your continued support while I try to figure this out!

Maybe it’s coming down to how I am trying to display this information.
Previously I was using a standard entity card and it would display everything cleanly.

I may need to look into a Statistics graph as it is not something I had used.

Here’s an example of the current entity card:

type: entities
entities:
  - entity: sensor.animal_kingdom_trips_this_month
  - entity: sensor.time_spent_at_animal_kingdom_this_month
  - entity: sensor.animal_kingdom_trips_this_year
  - entity: sensor.time_spent_at_animal_kingdom_this_year
  - entity: sensor.animal_kingdom_trips_total
  - entity: sensor.total_time_spent_at_animal_kingdom
title: Animal Kingdom Visits

Am I over complicating this? Could this all be done with one sensor and then I could use the statistics card/graph to extrapolate the data?

[Ildar_Gabdullin]

@bryancp31

current entity card:

This is Entities card, not Entity card.
Both display a current state, not any historical / statistical data.

@bryancp31

this_month

You have 2 types - “count” & “time”, for 3 periods - “this month”, “this year”, “total”.
For this you need 6 history-stats sensors.

But your post is not related to the previous discussion, here is what you said:

1. I have some HS sensors.
2. Their history is purged after 10 days, I want to see the history forever.

Everything what was written earlier was about “Why my sensors do not have long history? - Are you sure that they do not have?”.

[bryancp31]

You are correct. I think I need to rethink how this is setup. Thank you for talking through this with me.

I think I was mixing up how the entities card was displaying those sensors. In the past my purge history was set to 365 days, which allowed those entities cards to ‘work’. I was not calling any history stats in with those. This also caused my HA database to become bloated and sluggish.

I think I should create new sensors to track my location/‘state’ and instead of tracking each time frame individually (month, year, total) I can use the statistics card to display that information. Is this the reasonable approach?

Again, THANK YOU for pitching in to help me figure out my misunderstanding of this.

[Ildar_Gabdullin]

General:

1. Any entity’s history is kept in database (DB) within a “purge interval” (10 days by default). If an entity is excluded from Recorder - then it’s history is not stored at all.
2. Any numerical sensor entity may have LTS which is kept forever. To make a sensor “LTS-able” - a “state_class” should be set for this sensor.
3. Some integrations define “state_class” for their sensors by default, a user may not care about it. For template sensors - a user may define “state_class” manually as a part of definition.
4. HS-sensors (history-stats) have “state_class = measurement” by default.
5. To get HS-data for some period - this period should be defined as a part of HS-sensor definition. In your case it will cause creation 6 sensors - 3 periods * 2 types, where “types” stands for “time” & “count”.
6. BUT - there is a Statistic card which has a “period” option. I have never used this card - may be you can create 1 “time” sensor for “whole period” (instead of 3) and then select a particular period in that “option”. Just a guess…

[bryancp31]

Thanks for this explaination. I manually created these history_stats sensors in my sensor.yaml file a while ago (not fully well knowing what they did). I tracked the entities via the card and everything was working as expected while I had the recorder purge interval set to 365 days.

Looking into #6 now and I think it will fit what I’m looking for. This should reduce the amount of history stats I need to keep. I can keep just one timer and one count for each state (instead of splitting out by place in time).

Thanks!