Why can’t I get rid of bad sensor data?

[kuchta (Steve Kuchta)]

I would like to be able to easily remove or edit a bad data point from a sensor. Here is an example of a humidity sensor that has a random reading that throws off the history graphs which I like to use.

It seems like the only real way to do this now is to edit the recorder database directly. There was an update in 2022.4 that allows for correcting aggregate statistics using Developer Tools | Statistics, but I don’t think that helps in this particular situation. For one, this sensor isn’t even in this section, but I think that is because I don’t have all of the necessary properties configured in this MQTT sensor. But it also appears this is only for updating aggregate values (e.g. resetting the summation from a point in time). I don’t have any statistics being calculated on this sensor (at least not that I know of); it just retains the data for months.

For me, this is also unrelated to the Energy Dashboard, which it seems like many other people are referring to when looking for similar help. And I see a couple other WTH posts about the Energy Dashboard as well, but I think they are different than what I am asking for.

Ideally, I would be able to edit or delete this data point directly in the History view or at least get to it from the History view.

I think that the aggregate calculations make this request more complicated, so I’m not sure if there is a way to only allow it on non-aggregated sensors? Thanks for considering and all of the hard work from the developers! If there is already an easier way of fixing this type of issue without editing the database directly, I’d love for someone to point it out.

[e-raser]

Small note: guides (two really great!) and former feature requests in first post of duplicate WTH topic:

WTH can't we fix recorder data easily using the UI (dev-tools section) - single data points + mass- changes?

“If I would have one wish free, that would be… Allow correcting database entries using the UI” The current option of fixing single statistic entities values is nice, but not enough at all. [grafik] Your total energy usage sensor had a hiccup and substracted/added some megawatts a few days ago and now your energy dashboard is totally broken? Well, have fun with one of the guides listed below. Grab some coffee and say goodbye to your friends and family for the next few hours (better to reserv…

[kuchta (Steve Kuchta)]

Great notes in your post! I, too, spent hours digging through the database in order to fix a couple data points when the energy dashboard first came out. It was both painful and worrisome that I would make things worse! I hope that we can get a solution that makes it easier to edit this non-energy-dashboard bad data in a more sensible and safe way!

[Dave2 (Dave)]

Occasionally, a temperature sensor goes haywire and reports it’s 400°+ outside. That screws up all of my charts showing historical trends with that sensor. I wish there was an easy user interface to delete values out of range for a given entity without SQL.

[petro]

There is, developer tools → statistics, then click this button
next to your entity.

[swa72]

Will that change in data propagate to e.g. data from the Energy Dashboard? Or data recorded in influxdb? I doubt it - but it would be nice.

[petro]

Energy yes, influx

[sophof]

omg, since when when! I have been wishing for this for ages and never realized

[petro]

a year now

[Dave2 (Dave)]

I see what you’re talking about but this icon doesn’t appear to have that icon next to the entry I’m having an issue with. Maybe I’m just relying on the primary database of events rather than offloading, high and low temperatures for the sensor to a statistic which might be different?

[cargsl (Carlos S)]

This would be an incredibly useful feature. I run my recorder DB on postgres and I’ve had to jump in and manually remove rows from the statistics table to fix this. For many (most?) of my entities the edit button doesn’t appear on the Statistics panel in dev tools.

Also the same capability would be super useful for states

[metbril (Robert 🇳🇱🇪🇺)]

Cleaning the statistics is working just fine, but how about the recorder database itself?

[Jonas_Anden (Jonas Andén)]

Manually cleaning statistics is probably useful if it happens about once in a blue moon. However, if it happens more often than that, it becomes practically impossible to keep up unless you want to spend an hour every day hunting down the bad values.

What I would like to see is something like a default outlier filter easily applied to sensors of a certain type. A regular nuisance is that the zwave-based energy meters suddenly and momentarily think that the consumed energy is >200,000kWh. I have >100 sensor entities that I’m using in various scenarios, creating a filter definitions for them all will produce a huge configuration.yaml file (let alone the fact that it duplicates the number of entities available in dropdown etc). Something like this should do it:

• Temperatures must not change more than X degrees between two reports.
• Accumilated consumed/produced energy must not change more than X kWh between two reports.
• Humidity must not change more than X percent between two reports.

And so on. But these are the most important ones for me… Simple and effective, should require just a configuration of X for each type sensor type - no config, no filtering.

Instead of going around and fixing the effects of the bad data, why don’t we avoid storing it in the recorder in the first place?

[JWilson (James Wilson)]

Not really a universal answer, since this button often doesn’t exist.

None of the entities that I care about (all of my room temperatures & humidity sensors) even show up in the statistics page in developer tools.
All of them are MQTT sensors, with appropriate device_class set.

[Ghafla]

Just yesterday I stumbled upon the statistics page within the developer tools.
There are hundreds of “FIX ISSUE” buttons just leading to an annoying message box:

Entity no longer recorded

We have generated statistics for this entity in the past, but state changes of this entity are no longer recorded, therefore, we can not track long term statistics for it anymore.

You probably excluded this entity, or have just included some entities.

See the recorder documentation for more information.

If we cannot delete the statistics, why show the issues at all?
Trying to delete the entries via recorder.purge_entity service (and restarting HA afterwards) leads to nothing.

[Roemer (Roman)]

Using the fix on the dev-tools → Statistics fixes the Energy Dashboard data but the sensor itself still shows the wrong data and is therefore still exported wrongly for example to prometheus. Is there a way to fix all data?

[Mattie (Mattias)]

I want this exact thing too, having the same problem with sensors throwing weird values every now and then. It feels so ineffective to make double entities of all of these just to make this work properly… Even worse is that I use influxdb and I can fix the ones quickly in ha, but then I would have to do the same in influxdb to get those graphs show correctly. Just not doable. You got my vote!

[teskanoo]

@JWilson

None of the entities that I care about (all of my room temperatures & humidity sensors) even show up in the statistics page in developer tools.
All of them are MQTT sensors, with appropriate device_class set

I also have the same experience with being unable to get my climate (humidity and temp) sensors into statistics

They also have the correct device classes (temperature or humidity) and state class (measurement) set but are not picked up

[petro]

That’s all garbage, not only is state_class_options made up, but you can’t use state_class with legacy template entities. You have to use the new template style to use state_class.

This is why posting things from that certain bot are strictly prohibited. So please delete it and never post code from that bot again. Thanks.

[RobLatour]

I would like to see a fix for this too. My sessor entity does have the ramp icon beside it.