Manually cleaning statistics is probably useful if it happens about once in a blue moon. However, if it happens more often than that, it becomes practically impossible to keep up unless you want to spend an hour every day hunting down the bad values.
What I would like to see is something like a default outlier filter easily applied to sensors of a certain type. A regular nuisance is that the zwave-based energy meters suddenly and momentarily think that the consumed energy is >200,000kWh. I have >100 sensor entities that I’m using in various scenarios, creating a filter definitions for them all will produce a huge configuration.yaml file (let alone the fact that it duplicates the number of entities available in dropdown etc). Something like this should do it:
Temperatures must not change more than X degrees between two reports.
Accumilated consumed/produced energy must not change more than X kWh between two reports.
Humidity must not change more than X percent between two reports.
And so on. But these are the most important ones for me… Simple and effective, should require just a configuration of X for each type sensor type - no config, no filtering.
Instead of going around and fixing the effects of the bad data, why don’t we avoid storing it in the recorder in the first place?
Not really a universal answer, since this button often doesn’t exist.
None of the entities that I care about (all of my room temperatures & humidity sensors) even show up in the statistics page in developer tools.
All of them are MQTT sensors, with appropriate device_class set.
Just yesterday I stumbled upon the statistics page within the developer tools.
There are hundreds of “FIX ISSUE” buttons just leading to an annoying message box:
Entity no longer recorded
We have generated statistics for this entity in the past, but state changes of this entity are no longer recorded, therefore, we can not track long term statistics for it anymore.
You probably excluded this entity, or have just included some entities.
If we cannot delete the statistics, why show the issues at all?
Trying to delete the entries via recorder.purge_entity service (and restarting HA afterwards) leads to nothing.
Using the fix on the dev-tools → Statistics fixes the Energy Dashboard data but the sensor itself still shows the wrong data and is therefore still exported wrongly for example to prometheus. Is there a way to fix all data?
I want this exact thing too, having the same problem with sensors throwing weird values every now and then. It feels so ineffective to make double entities of all of these just to make this work properly… Even worse is that I use influxdb and I can fix the ones quickly in ha, but then I would have to do the same in influxdb to get those graphs show correctly. Just not doable. You got my vote!
That’s all garbage, not only is state_class_options made up, but you can’t use state_class with legacy template entities. You have to use the new template style to use state_class.
This is why posting things from that certain bot are strictly prohibited. So please delete it and never post code from that bot again. Thanks.
In the process of migrating from Domoticz and although HA is easier and more intuitive than domoticz the latter had a great gimmick: In the graph you could click on an invalid point while holding shift and domoticz would ask if you would like to delete that specific point
Is there a uservote / functionality request option for HA?
This kind of functionality would be great.
My Energy-Dashboard is totally worthless for this year due to wrong statistic date I can’t remove.
Unbelievable that this discussion started one year ago…
I made myself a Node-Red flow to automatically detect and delete those bad values.
But not from the internal Sqlite db, as I use InfluxDB and Grafana for energy dashboarding.
Nonetheless: Maybe it helps someone to create a similar solution for his needs instead of manually deleting bad Zwave values every few weeks from his history.
Today, my DSMR gas sensor apparently had a hiccup; it reported 0 m3 for a minute or so. Now, today’s gas consumption is way off (I think it exceeds Tata Steel daily consumption).
Since this is not a ‘statistic’, it seems like this is an unrecoverable error? There is no way to remove a faulty data point that I could find.
I sorely miss this functionality (as well as being able to mass-delete non-existent devices)
Is there perhaps an add-on that would allow me to repair the data?
I had to remove this value manually in the following way: I searched with the tool DB Browser (SQLite) for the faulty value (“284.05”) in the STATES database and noted down the state_id. Then, I deleted this single value with DELETE FROM "states" WHERE state_id='1278727'; and the chart looks properly again:
Just discovered HA’s long term stats existed today after I saw the 2023.12 posting that talks about them. Unfortunately I also found today that most of my sensors collected outliers over the last year or so, which renders the long term stats totally useless, see for instance
I looked into a random sample of 12 sensors (all of which have outliers) and not a single one of them shows the “fix” icon described above, supposed to be in Development Tools > Statistics list. Not sure what is the criteria but these are all sensors from standard temp / humidity / etc zwave devices.
We really need a feature in the History Graph so that we can click on the outlier point (which is always easy) and then be offered the option to “Delete this point”. Without this, long term stats have been useless every time I tried to use them.
Edit: it would also be useful to have a “reset sensor stats” type button to target all data or a specific period of data, for a specific sensor.
