Why can’t I get rid of bad sensor data?

Month of "What the heck?!"
#1

I would like to be able to easily remove or edit a bad data point from a sensor. Here is an example of a humidity sensor that has a random reading that throws off the history graphs which I like to use.

image
image892×675 26.6 KB

It seems like the only real way to do this now is to edit the recorder database directly. There was an update in 2022.4 that allows for correcting aggregate statistics using Developer Tools | Statistics, but I don’t think that helps in this particular situation. For one, this sensor isn’t even in this section, but I think that is because I don’t have all of the necessary properties configured in this MQTT sensor. But it also appears this is only for updating aggregate values (e.g. resetting the summation from a point in time). I don’t have any statistics being calculated on this sensor (at least not that I know of); it just retains the data for months.

For me, this is also unrelated to the Energy Dashboard, which it seems like many other people are referring to when looking for similar help. And I see a couple other WTH posts about the Energy Dashboard as well, but I think they are different than what I am asking for.

Ideally, I would be able to edit or delete this data point directly in the History view or at least get to it from the History view.

I think that the aggregate calculations make this request more complicated, so I’m not sure if there is a way to only allow it on non-aggregated sensors? Thanks for considering and all of the hard work from the developers! If there is already an easier way of fixing this type of issue without editing the database directly, I’d love for someone to point it out.

WTH Can I not delete an obviously erroneous reading from a sensor without SQL access?
WTH can't we fix recorder data easily using the UI (dev-tools section) - single data points + mass- changes?
WTH why do we need to restart HA core so much?
#2

Small note: guides (two really great!) and former feature requests in first post of duplicate WTH topic:

#3

Great notes in your post! I, too, spent hours digging through the database in order to fix a couple data points when the energy dashboard first came out. It was both painful and worrisome that I would make things worse! I hope that we can get a solution that makes it easier to edit this non-energy-dashboard bad data in a more sensible and safe way!

1 Like
#4

Occasionally, a temperature sensor goes haywire and reports it’s 400°+ outside. That screws up all of my charts showing historical trends with that sensor. I wish there was an easy user interface to delete values out of range for a given entity without SQL.

#5

There is, developer tools → statistics, then click this button image
next to your entity.

2 Likes
Is there any way to edit a sensor's history?
#6

Will that change in data propagate to e.g. data from the Energy Dashboard? Or data recorded in influxdb? I doubt it - but it would be nice.

#7

Energy yes, influx :man_shrugging:

#8

omg, since when when! I have been wishing for this for ages and never realized :smiley:

#9

a year now

#10

I see what you’re talking about but this icon doesn’t appear to have that icon next to the entry I’m having an issue with. Maybe I’m just relying on the primary database of events rather than offloading, high and low temperatures for the sensor to a statistic which might be different?

1 Like
#11

This would be an incredibly useful feature. I run my recorder DB on postgres and I’ve had to jump in and manually remove rows from the statistics table to fix this. For many (most?) of my entities the edit button doesn’t appear on the Statistics panel in dev tools.

Also the same capability would be super useful for states

#12

Cleaning the statistics is working just fine, but how about the recorder database itself?

#13

Manually cleaning statistics is probably useful if it happens about once in a blue moon. However, if it happens more often than that, it becomes practically impossible to keep up unless you want to spend an hour every day hunting down the bad values.

What I would like to see is something like a default outlier filter easily applied to sensors of a certain type. A regular nuisance is that the zwave-based energy meters suddenly and momentarily think that the consumed energy is >200,000kWh. I have >100 sensor entities that I’m using in various scenarios, creating a filter definitions for them all will produce a huge configuration.yaml file (let alone the fact that it duplicates the number of entities available in dropdown etc). Something like this should do it:

  • Temperatures must not change more than X degrees between two reports.
  • Accumilated consumed/produced energy must not change more than X kWh between two reports.
  • Humidity must not change more than X percent between two reports.

And so on. But these are the most important ones for me… Simple and effective, should require just a configuration of X for each type sensor type - no config, no filtering.

Instead of going around and fixing the effects of the bad data, why don’t we avoid storing it in the recorder in the first place?

1 Like
#14

Not really a universal answer, since this button often doesn’t exist.

None of the entities that I care about (all of my room temperatures & humidity sensors) even show up in the statistics page in developer tools.
All of them are MQTT sensors, with appropriate device_class set.

#17

Just yesterday I stumbled upon the statistics page within the developer tools.
There are hundreds of “FIX ISSUE” buttons just leading to an annoying message box:

Entity no longer recorded

We have generated statistics for this entity in the past, but state changes of this entity are no longer recorded, therefore, we can not track long term statistics for it anymore.

You probably excluded this entity, or have just included some entities.

See the recorder documentation for more information.

If we cannot delete the statistics, why show the issues at all?
Trying to delete the entries via recorder.purge_entity service (and restarting HA afterwards) leads to nothing. :roll_eyes:

1 Like
#18

Using the fix on the dev-tools → Statistics fixes the Energy Dashboard data but the sensor itself still shows the wrong data and is therefore still exported wrongly for example to prometheus. Is there a way to fix all data?