Understanding And Applying Sensor Filters

Configuration
1

I’m brand new to HA and trying to wrap my head around how it handles data and configurations. Can you all help me understand how the filters integration works?

TL;DR: Are filtered sensors treated as new sensors and data recording only begins at the moment they are created/configuration reloaded? Is there any way to retroactively apply the filter to historical data for debugging and tweaking changes to the filter?

My Setup:

I have two sensors that use the jomjol AI on the edge (AIOTE) digitizers to provide MQTT interface for older dumb utility meters. The AIOTE sensors sample about once per five minutes.

This all works great, but the data from the electricity meter is rather difficult to look at. I’d like to smooth this out using the filter integration.

image
image876×438 13.9 KB

configuration.yaml

sensor:
    - platform: filter
      name: "electricity rate per time unit lp sma"
      entity_id: sensor.electricity_rate_per_time_unit
      filters:
        - filter: lowpass
          time_constant: 10
          precision: 2
        - filter: time_simple_moving_average
          window_size: "00:20"
          precision: 2

    - platform: filter
      name: "sma filtered electricity rate of change"
      entity_id: sensor.electricity_rate_per_time_unit
      filters:
        - filter: time_simple_moving_average
          window_size: "00:20"
          precision: 2

    - platform: filter
      name: "lp filtered electricity rate of change"
      entity_id: sensor.electricity_rate_per_time_unit
      filters:
        - filter: lowpass
          time_constant: 10
          precision: 0

My Understanding

It appears that when I reload the filters yaml, the filter only applies to the filtered sensor data from that point forward. They filtered sensors to be treated as new data and only show the changes going forward. This is extremely hard to debug with slow data that arrives in only a few points per hour. Every change I make, I have to wait 1-2 days to see if I get the results that I want. It also means that I need to keep adding new filters so I can do A:B comparisons.

My Questions

  1. Is my understanding of this correct?
  2. Is it possible to apply a filter to historical data? This feature request from 2020 appears to ask the same question, but with no answer.
  3. Are there alternative ways of displaying/filtering historical data without using the filter integration? I know I can pull down the data in CSV and do all the filtering in spread sheet, but that’s not ideal.
2

Yes new sensors, and no there is no way to apply the filter to previously recorded data.

3

@tom_l: Is there any way to use the current data and run it through the filter outside of the card display? Maybe some sort of sandbox where I can feed a data stream through the filter algorithms and see the output?

1 Like
4

Nope.

5

@tom_I how about some alternative methods/integrations/plugins/magic goats that can be used for visualizing data?

6

You mentioned spreadsheets. That’s what I sometimes use to develop linear or higher degree equations from data points. Nothing else that I know of.

1 Like
7

Only idea which springs to mind working with past data and filtering is from within apexcharts.
You neeed to record a day of data (since it does need the data) prior using this, since if there’s no data you can’t warp back in time.

And then you simply compare the current data with the data -1d (1 day back)
And apply (surely limited) filters using the func: function.

quick hack done by setting up this code using one of my temp sensors I’m aware it was running in the past.

type: custom:apexcharts-card
graph_span: 1d
span:
  start: day
header:
  show: true
series:
  - entity: sensor.esp8266dach_bme280_temp

  - entity: sensor.esp8266dach_bme280_temp
    offset: '-1d'
    group_by:
      func: median
      duration: 3h

image
the orange line is the empearute of the current day (past
the blue simply 24hrs back and flattened using median 3h

I do hope I got it right what was asked for.
I did understand it like this. Aaron was looking for an easy way to play with collected data (from the past) to set up a filtering to get a more suitable representation of the the data.
Means once he found the satifying filtering he’s done and could concentrate upon filtering while collecting data? Correct?

1 Like
8

@justone

Thanks for this. I’ll have a play with it.

Specifically, what I’m looking for is a way to apply more complex filters to existing data. I want to take data from last week, add a low pass filter to it, look at the output chart, change a few settings and compare the output until I dial in exactly what I’m looking for.

My current solution is to bake in several variations of the same filtered sensor, wait for 3-5 days and then compare the output. I’ll take the best of those variations, and create yet another batch of filtered sensors with slightly tweaked settings, wait 3-5 days, rinse, repeat until I get what want.

9

I run seperate HASSs for experimentation and create simulators to feed data. The apexcharts is a good suggestion, it’s an excellent charting package and allow you to apply filters to historical data.

2 Likes
10

Have you by any chance looked into using the SQL - Home Assistant sensor type?

It explicitly allows you to look up the state of a sensor at some time in the past

Since the query must return 1 result at most, I don’t think you can do processing after the query. So you’re probably stuck with aggregation functions. I’m not sure what’s available, although I found count on the Alchemy website, and round and sum are shown in examples on the sensor webpage.

But simple smoothing might be entirely possible. Maybe check if median is supported.

UPDATE: You can probably use whatever functions are available in your db backend (e.g. sqlite). If median is not built-in, you may be able to do something like this: How can I calculate the median of values in SQLite? - Stack Overflow