There have been several times when I have been experimenting with a sensor or circuit and caused some sort of unusual reading. This messes with graphs on the dashboard so would be good to have a way to edit or delete values - without deleting all history.
There are two places this needs to be addressed.
-
Long term statistics can currently only be adjusted for entities with a total or total_increasing state class, not measurement.
-
State history. Though this does disappear after your recorder purge keep days setting (10 days by default).
Yeah, would also love to see this.
Maybe a new service call that takes in data ala āfrom: 2024-12-01 10:00AMā and āto: 2024-11-20 03:00PMā or something. And clears that data out or can replace it with a specific value.
Every time I replace freezer temperature sensor battery, I have peak in graph that also changes min/max values. The same when I need to (re)calibrate power measuring plugs/sockets.
Sometimes I just remove sensor and re-add it back later hoping that the sensor name will be the same. But it is easy to forget that.
1 Like
As a not so SQL-gifted person this would help me greatly, right now I always dread having to do this manually in the database (and I forget how every time).
I consider the Need of Managing Recordings/Historical Data is a Task almost every User will face sooner or later. By now it guess every user at some Point abadons Statics, sometimes containing Values of many Years.
Just an, as i think, really common Usecase:
Got a Replacement for a Temperature Sensor. Hooked it up, and let it run some Days parallel to the old one. Then put the new one beneath some others, and found out the Room where i want to replace the sensor is shifted 0.6Ā° in average.
What i now need to do:
- Export all Statistics for the old one
- Shift the Values by 0.6Ā°
- Identify and Delete wrong Data for both Devices, where i had them in the Hand for changing batteries, carry aroundā¦
- Import the cleaned old Data into the new cleaned up Devicesā LTS
As far as i am aware of, you can only edit specific Datapoints one by one. There is a function to completely delete Stats for an Entity, if it is no longer beeing recorded. Thats it.
What i regulary (1-2x per Year) would need:
- Delete Statistics/Recorder Data for a given Time Interval (ādelete everything between 02-12 12:00 and Nowā)
- Import Statistics/Recorder from another entity/from a csv, with the Option to overwrite everything / Import up to the Point where data starts / overwrite existing data at Destination, keep existing data when not present in Source
- Export all Statistics/Recorder to a csv (this is already available i think)
There are more Tasks, like modify Data externally, but with an Option to export/import everything, that can all be done. Maybe the Options from Pt. 2 can be narrowed down as well.
Most probably a full Restart will be required after such operations. Further, before actually writing Data i guess Recorder needs to be stopped. This could be minimized by only previewing the data operations in edit mode, but queeing the actual commands. then, when hitting run, HA stops Recorder, performs the operations, and automatically restarts - either recorder, or Homassistant in Total.
1 Like
I think this is a fundamental must-have !!
Data is the new oil.
Dear @Alsoff, my WTH WTH can't we comprehensive manage Historical/Recorder Data got merged with yours, as its longer up and has more Votes already - great!
However, my WTH goes beyond the Demand to āedit or delete values - without deleting all historyā.
Is your WTH only about recent readings, or Historical Data as well?
If it is meant to adress both Recorder and LTS, would you mind extendning your Headline? I would be disappointed, if at some Point that would be āsolvedā, but only adresses recent Readings.
Same, if there would come a Feature to edit one specific Datapoint each with āDoneā - i think bulk replace / erase / complete is more handy than edit selective some.
My AI on the edge for gas meter is confusing numbers sometimes and is producing wrong data (see image)
Is there a world where we can edit from UI historical data and it propagates across other statistics?
Would also appreciate tips and tricks into how to fix this in my case
And yes, I understand that I need to fix my source of data but IMHO this needs a way to fix from HA because one such bad reading and your averages, statistics and other potential metrics are skewed and you canāt really do anything about it
For more clarity:
I am using a esp32 cam with this GitHub - jomjol/AI-on-the-edge-device: Easy to use device for connecting "old" measuring units (water, power, gas, ...) to the digital world and pushing the data via MQTT integration with discovery.
Every 5 minutes thereās a reading where it takes a photo, and tries to parse analog gas meter numbers into digits. In the process of converting image to numbers it wrongly might detect 3 as 9 or 2 as 7 and so on
Conceptual HA question:
Is my understanding of the philosophy right here, HA will ingest all the data regardless of the quality, but then we must then decide ourselves to map/filter/reduce data to something that makes sense? This explains the reason why HA cannot natively edit historical data and would not implement this
1 Like
I hope I understand you fully on your exact issueā¦
A sensor is randomly reading an outlier or spiking for no reason?
Or is AI trying to create this data off of your home assistant info?
would this happen to help at all?
sensor:
- platform: filter
name: āFiltered Gas Meterā
entity_id: sensor.gas_meter
filters:- filter: outlier
window_size: 4
radius: 2.0 - filter: time_simple_moving_average
window_size: 10
- filter: outlier
Otherwise If youāre using the Recorder integration, you can purge and repopulate specific ranges of data.
Let me know if any of that helps your specific scenario?
What hardware is at play?
1 Like
Thanks for the reply, I updated the post to clarify the setup and sensor data acquisition. I am yet to play with the filter, but it seems that it should do the job. Do you know if this can be configured somehow from UI or is it a config.yaml only way?
In the meantime you can use a a filter entity to smooth or discard the broken data. See Filter - Home Assistant. With the way your data spikes and reverts it looks like there would be several options for the specific filter, outlier would be my starting point but low-pass may also work well.
Edit: Haha I see someone else had the same thought while I was typing!
2 Likes
Itās a yaml-only sensor unfortunately, and requires a restart to pick up the first one. Once you have it created, you can reload just the filter entities from the Developer page.
2 Likes
Some of my Z-Wave switches have weird power blips too. Instead of Developer tools -> Statistics -> search -> click the Adjust Sum icon -> Outlies, it would be nice to just click the 3 dots (in the upper right of the screenshot in the original post), and select Fix Outliers.
2 Likes
Worse (IMHO) is that the āAdjust sumā button is only on a very select few stats. Why limit the fixing to only specific data types? Hmm, maybe Iāll make a WTH post on that specificallyā¦
1 Like
Thank you, " AI on the Edge Device: Digitizing Your non-digital meters with an ESP32-CAM" looks like an excellent use of AI for home assistants, I havenāt gotten far enough for implementing any energy usage stuff.
I only recently started making plans and looking at water usage devices thatās what I was so curious about when I saw you implementing gas meters into home assistance. If you ever call a service person out for a gas appliance ask for someone who can clock a meter and then share this data with them.
Glad to hear my filter idea was on somewhat of a correct path, I couldnāt say if it is even going to acknowledge the integration(I find Iām constantly looping integrations around(hope that makes sense))
Also to use low-pass or not is out of my range lol. Ask ai?
Can we stick to the WTH Topic? 
If I could just add WTH is there no easy way to move long term statistics to a new entity when replacing it e.g power meter
1 Like
Absolutely! I created a WTH for this, but unfortunately was too late, so it got merged: