testing chart. I think/hope from tomorrow stats max and group max to be aligned but group last will be 0
fixed bug in chart
Long term statistics (LTS) are hourly min, max and average, which are kept forever for any sensor with the state_class of measurement.
The history stats integration uses entity state values, not LTS. State history is only kept for the recorder purge_keep_days setting (10 days by default). You don’t want to keep more than a month or so as it can affect performance.
I see. That’s an issue for both the weekly and monthly heater usage I want to track. I don’t need statistics really. Just to store one data point per week or month.
I’m thinking now that this is not possible with the path I am going down.
Works great for the daily view though!
Are there any options that I can look into for my weekly and monthly views without changing how long state data is held?
Failed experiment as setting the start time one minute earlier did not result in the reset happening before midnight. Not sure why. Will try again with a 5 minute time delta (so reset at 11.55pm) and report back tomorrow.
I might also setup a test history stat so I can play around with it today.
Hello @tom_l - I have been doing a lot of work/thinking about the issue you raised in relation to LTS and my history_stats sensor…
My idea was to create a template sensor with the state_class measurement and use the state of the history stat sensor to populate it. code below. I converted time to energy as I know the power output of my heater.
I think this should work but will need to wait 10+ days to confirm. So I went and created a chart to track this and noticed that the data in the history stats sensor goes back way more than 10 days. So now I am confused. See chart below.
So I am asking how can I have the data going back 2-3 months if it is not LTS?
- sensor:
- name: "Gas Heater Daily Energy"
unique_id: gas_heater_daily_energy
state_class: measurement
unit_of_measurement: "kWh"
icon: "mdi:home-lightning-bolt"
state: >
{% set heater_hrs = states('sensor.gas_heater_daily_usage') | float %}
{{ ( heater_hrs * 6.2 ) | round (1) }}
attributes:
heater_switch: "{{ states('switch.gas_heater') }}"
heater_power: "6.2kW"
heater_hours_today: "{{ states('sensor.gas_heater_daily_usage') | float }}"`
Very strange.
Have you previously changed the recorder purge keep days setting?
Does the new sensor you created have the same entity id as a previously existing sensor with a measurement state class?
sensor.gas_heater_daily_energy
The history stats sensor definitely states:
No I haven’t changed purge_keep_days. Where is this configured so I can check?
Interestingly, when i use the entity history viewer (see below) the data only goes back 7 days but when I use apexcharts and set the period back further - the data is there!
Also I think i have confused you (sorry) with the chart as I added a new series that has energy calculated using the time x power output of the heater.
The power sensor was only created today so has very limited data at the moment.
The history chart is using the state values.
Your apex chart is using statistics (LTS) isn’t it?
Yes but does this new sensor entity id match a previous one (now deleted) that had the same entity id?
LTS are associated with entity ids and LTS are never deleted. So if you had a sensor with the same entity id previously that is where the data is coming from.
Otherwise Apex Charts seems to be making it up as it goes along.
In your configuration.yaml file. If you are using default_config: you may not have a recorder: integration entry in your configuration.yaml file as it is included in default_config.
Ok well I am using default_config so that confirms I haven’t changed anything.
Below is a better chart that uses the same entity but one is using group_by and the other statistics. looks to me that LTS are being maintained for the the entity. Would you agree?
Yes, you must have previously had a sensor called sensor.gas_heater_daily_energy. Your new sensor is appending LTS to this previous sensor’s LTS data.
In other news/update, I did some testing and when I use a specific time with today_at such as
{{ today_at('10:00') - timedelta(minutes=3) }}
The event happens faster - 15secs rather than 30-90 secs.
So I have now changed my reset time for daily as below - also increased delta from 1min to 3.
"{{ today_at('00:00') - timedelta(minutes=3) }}"
So tomorrow I will see if I have fixed the sensor as it should reset well before 12am.
Just FYI, these two should be equivalent:
today_at('00:00')
today_at()
Midnight is the default if no time is specified.
Are you saying the one with the time `00:00’ specified updates at a faster polling interval?
At the end of the day, this is good news as I needed the history_stats sensor to hold LTS which it seems to be doing.
And I have found the transform function in apexcharts so I can calculate energy.
If it all works, I can delete the new ‘energy’ LTS sensors as they are redundant
Yes, I think so. I haven’t tested ‘00’00’ yet. But i did create some history stat sensors during the day today with other times set e.g. 10:00 and they only took 10 seconds
Over the next few days I will update this thread as it might help others with a similar issue.
I’m confident that HA will now reset the sensor just before midnight and as result group_by max and statistics max will match up
Bummer but it still didn’t work. I think the issue is that the event can’t fire earlier than 12am no matter what you do with timedelta as it is needs to wait for ‘today_at’ and that can’t occur before 12am.
My next experiment is to try to set the end time for the sensor to be at 11.59pm. Nope. Didn’t work
My other idea is to use statistics max together with group_by max (on the same entity in apexcharts) and see if that helps with how the data is displayed Nope. didn’t work
hey @cd84 and @tom_l - just wanted to say thanks again and to update you on this. I have found the issue and a solution…
The issue is ‘known’ in that it is reported in core for history_stats - see link below where I also found a solution but it it a work-a-round. Tested and working fine!
In summary, changing start and end times for the senor/entity does not work no matter what you do. End time stops the recording of data but retains the last state value until it resets just after 12am.
The solution is to use a python script (another HA integration - see link below) to reset the state for the sensor to 0 just before midnight at 23:59:59 (it might even work if you set this to 00:00:00 but I haven’t tried that). The python script is triggered with an automation. @cd84 - happy to share all the details if this helps you. I don’t see why you couldn’t use the solution on the sensor(s) you are having trouble with.
Happy Days!
1 Like