Misunderstanding Filters / Filters not working as expected

[jamespreedy]

Hi folks, thanks as always for your advice.

I use filters to try to weed out odd results from my Z-Wave (and other) devices before displaying them on my dashboards or sending them to Influx.

In general, I pass them through a range filter first (to weed out values that can’t be valid) - in the example below, the drier can’t have used less than 0kWh. I then run them through an outlier to cut out values outside a given range.

- platform: filter
  name: "Dryer Filtered [kWh]"
  entity_id: sensor.utility_room_dryer_electric_consumption_kwh
  unique_id: 'C6EC1477-C75B-4BDB-8FD2-20BC826C423E'
  filters:
    - filter: range
      lower_bound: 0 
    - filter: outlier
      window_size: 4
      radius: 4.0

I thought this would work but values like the below …

… are still getting through.

My thinking is that I’m using the wrong filter, or the wrong combination of filters.

Has anyone had any particular success in using filters to weed out unwanted values?

Thank you!

[petro]

I suggest you spend time reading the filters and understanding the math. It’s displayed for you in the documentation so you can decide if that filter is what you want.

1. You need to know how many datapoints you have. All filters work based on previous states. They ether use a time window or a window size. Time window is a duration and it chooses the points in that duration. window size is literally selecting n number of points.

2. Use your graph and values from the graph to run a rough calculation, the math is provided in the docs.

For outlier…

A window_size of 4, means it’s taking the median of the last 4 states and throwing out the values that are beyond 4.0 from the median. This is executed on EVERY state from your source sensor. It’s “Rolling”. If you didn’t set your window size properly, you will get peaks that squeak through.

In your graph, I only see flat lines. Meaning your last 4 states might not even be visible. And if the last 4 distance is greater than the median of the last 4, it outputs the median of the last 4.

So assuming that you actually have a ton of points in the first graph, you may just need to increase or decrease the window size.

[jamespreedy]

Thanks @petro, appreciate it.
I had a look through the maths and though the data from my states but I think I’m still missing something.
Sorry …

With these sensors, they are tracking the kWh power consumption of devices.
These are always increasing slowly, sometimes by not very much (when a device is in standby for example), or as little quicker (when the device is in use).

To try to show what I’m failing to understand (), I’ll use the same example as before.

In this case, we had a number of slowly increasing readings when the dryer was in use.
When it finished it’s cycle, it went into standby and used very little for a while.

For the sake of easy maths, lets say that readings were …

117, 118, 119, 120, 121, 122, 40

I’m using an outlier filter with a window_size of 4, and a radius of 4 too.
Documentation says …

distance = abs(state - median(previous_states))

if distance > radius:
    median(previous_states)
else:
    state

So when we get the incorrect reading, 40 …

distance = abs(40 - median ( 40, 122, 121, 120))
distance = abs(40 - 120.5) = abs( - 80.5) = 80.5

if distance > radius
if 80.5 > 4 [TRUE]

  median(previous_states)
  = 120.5

So in this case the incorrect reading, 40 would be thrown out, and replaced with 120.5 …
… but it wasn’t.

Sorry to be a pain - am I missing something with the logic, or the maths?