I’m brand new to HA and trying to wrap my head around how it handles data and configurations. Can you all help me understand how the filters integration works?
TL;DR: Are filtered sensors treated as new sensors and data recording only begins at the moment they are created/configuration reloaded? Is there any way to retroactively apply the filter to historical data for debugging and tweaking changes to the filter?
My Setup:
I have two sensors that use the jomjol AI on the edge (AIOTE) digitizers to provide MQTT interface for older dumb utility meters. The AIOTE sensors sample about once per five minutes.
This all works great, but the data from the electricity meter is rather difficult to look at. I’d like to smooth this out using the filter integration.
It appears that when I reload the filters yaml, the filter only applies to the filtered sensor data from that point forward. They filtered sensors to be treated as new data and only show the changes going forward. This is extremely hard to debug with slow data that arrives in only a few points per hour. Every change I make, I have to wait 1-2 days to see if I get the results that I want. It also means that I need to keep adding new filters so I can do A:B comparisons.
My Questions
Is my understanding of this correct?
Is it possible to apply a filter to historical data? This feature request from 2020 appears to ask the same question, but with no answer.
Are there alternative ways of displaying/filtering historical data without using the filter integration? I know I can pull down the data in CSV and do all the filtering in spread sheet, but that’s not ideal.
@tom_l: Is there any way to use the current data and run it through the filter outside of the card display? Maybe some sort of sandbox where I can feed a data stream through the filter algorithms and see the output?
Only idea which springs to mind working with past data and filtering is from within apexcharts.
You neeed to record a day of data (since it does need the data) prior using this, since if there’s no data you can’t warp back in time.
And then you simply compare the current data with the data -1d (1 day back)
And apply (surely limited) filters using the func: function.
quick hack done by setting up this code using one of my temp sensors I’m aware it was running in the past.
the orange line is the empearute of the current day (past
the blue simply 24hrs back and flattened using median 3h
I do hope I got it right what was asked for.
I did understand it like this. Aaron was looking for an easy way to play with collected data (from the past) to set up a filtering to get a more suitable representation of the the data.
Means once he found the satifying filtering he’s done and could concentrate upon filtering while collecting data? Correct?
Specifically, what I’m looking for is a way to apply more complex filters to existing data. I want to take data from last week, add a low pass filter to it, look at the output chart, change a few settings and compare the output until I dial in exactly what I’m looking for.
My current solution is to bake in several variations of the same filtered sensor, wait for 3-5 days and then compare the output. I’ll take the best of those variations, and create yet another batch of filtered sensors with slightly tweaked settings, wait 3-5 days, rinse, repeat until I get what want.
I run seperate HASSs for experimentation and create simulators to feed data. The apexcharts is a good suggestion, it’s an excellent charting package and allow you to apply filters to historical data.
Since the query must return 1 result at most, I don’t think you can do processing after the query. So you’re probably stuck with aggregation functions. I’m not sure what’s available, although I found count on the Alchemy website, and round and sum are shown in examples on the sensor webpage.
But simple smoothing might be entirely possible. Maybe check if median is supported.