Outlier filter shows decreasing values

Configuration
1

I’m monitorting the energy consumption of my heatpump. The heatpump sensors work via MQTT, unfortunately the graph shows outliers once in a while. Sometime an increate of 500.000kwh and sometime it goes to a very low value. So I checked the documention and found the outliers filter which should to the trick in this case. The outliers are actually filtered, however, I see some behavior which I don’t understand.

For example.
I created the following filter for the total energy consumption:

sensor:
  - platform: filter
    name: "Energy wp total kwh"
    unique_id: energy_wp_total_kwh
    entity_id: sensor.enviline_boiler_nrgconstotal
    filters:
      - filter: outlier
        window_size: 10
        radius: 5.0

As of my understanding this should filter out increases of more dan 5kwh in one update.

If I look at the original sensor the history graph looks as follow:

original
original1529×611 26.7 KB

The filtered sensor shows the following graph:

filtered
filtered1531×543 23.5 KB

There are several decreases of the values while there weren’t any outliers in the original sensor at that moment. I’m trying to understand the outlier filters but I’m a bit stuck here. What am I missing?

2

those are larger than your radius, increase the size of your radius to something like 100.

As a sidebar, a KWh of 12000 seems a bit large, I’d start to investigate why it’s in the 1000’s

3

Thanks for the quick reply, I’m trying to understand how it should work so I can also use it with different kind of sensors. What do you mean with “those are larger than your radius”?

In the documentation I see the following for the outlier filter:

distance = abs(state - median(previous_states))

if distance > radius:
    median(previous_states)
else:
    state

In my understanding the radius means the maximal increase for each step, but if I read your post I’m probably wrong :slight_smile:, or does it means total of the total window size instead of each update?

The heathump has an increasing sensor starting at day one when the heatpump was installed. These 12.000kwh are over 3,5 years, so it should be accurate, it also matches with a youless energy meter in front of the heatpump.

4

abs = absolute value, i.e. it works on negative numbers as well.

5

I still don’t understand it but I will further experiment with it. Thanks for the help!

6

It’s a math function. Absolute value of a negative number is that same number but positive. I.e. abs(-3948) is 3948

7

This is a bit old thread, but google got me here while trying to solve my own problem with outlier-filter. Still don’t know what is wrong with mine, but i thought to say what i think is wrong here.

Window size of 10 makes this code check last 10 values, all of them should be smaller than the always increasing new value. Median selects the middle one of those values (average of middle two with even number of values), after sorting them. So, that value which is compared to the new one, should be 5.5 smaller than the new value.

Then there is the radius of 5.0. I don’t actually know how that filter even worked as well as it seems in the graphs, maybe some values are read multiple times and that median of last 10 values usually gets closer to the new value and passes that filter. Those dips in the graph might happen when there are exactly one update per kilowatt increased and filter failure chooses the median of last 10 as the new output.

So, smaller window or larger radius should help.

Ps. I think that whole outlier filter is stupid, because it uses old median value when new value fails. It would be way better to discard the value and make the sensor not available or what was it called here. There is most likely a way to do it, but most users find these default functions and just believe that this is a normal way to fix the problem.