Outlier Filter Sensor not working as expected

I have defined a filter sensor to filter outliers from a sensor (a REST sensor retrieving air quality data):

sensor:
  - platform: filter
    name: 'PM'
    entity_id: sensor.pm_raw
    filters:
      - filter: outlier
        window_size: 4
        radius: 5

According to the documentation (https://www.home-assistant.io/components/sensor.filter/) this should replace any value which is higher or lower than the median of the four previous measurements ± 5 with the median of the four previous measurements.

Actual results are, however, quite different:

In this example, it correctly identifies the outlier, but seems to replace it with the previous value (not the median of the previous values). It goes back to normal after the next, non-outlier, measurement:

2019-02-18 15:48:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=39.4) -> 39.4
2019-02-18 15:53:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=38.0) -> 38.0
2019-02-18 15:56:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=39.0) -> 39.0
2019-02-18 15:58:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=38.5) -> 38.5
2019-02-18 16:03:16 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 1 in sensor.pm_raw: 59.2
2019-02-18 16:03:16 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=59.2) -> 38.5
2019-02-18 16:05:30 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=38.8) -> 38.8

In the following situation, however, the sensor gets completely stuck and continues to replace any following measurement with the same value

2019-02-18 19:23:20 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=36.1) -> 36.1
2019-02-18 19:26:11 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 4 in sensor.pm_raw: 44.0
2019-02-18 19:26:11 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=44.0) -> 36.1
2019-02-18 19:28:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 5 in sensor.pm_raw: 45.0
2019-02-18 19:28:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=45.0) -> 36.1
2019-02-18 19:31:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 6 in sensor.pm_raw: 44.1
2019-02-18 19:31:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=44.1) -> 36.1
2019-02-18 19:33:20 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=38.7) -> 38.7
2019-02-18 19:36:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 7 in sensor.pm_raw: 89.0
2019-02-18 19:36:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=89.0) -> 38.7
2019-02-18 19:38:20 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 8 in sensor.pm_raw: 84.9
2019-02-18 19:38:20 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=84.9) -> 38.7
2019-02-18 19:41:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=37.6) -> 37.6
2019-02-18 19:43:21 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 9 in sensor.pm_raw: 31.6
2019-02-18 19:43:21 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=31.6) -> 37.6
2019-02-18 19:46:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 10 in sensor.pm_raw: 30.6
2019-02-18 19:46:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=30.6) -> 37.6
2019-02-18 19:48:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] Outlier nr. 11 in sensor.pm_raw: 29.5
2019-02-18 19:48:10 DEBUG (MainThread) [homeassistant.components.sensor.filter] outlier(sensor.pm_raw=29.5) -> 37.6

Starting with outlier nr. 9, every following measurement is identified as an outlier and is being replaced with ‘37.6’.

The same thing happens with 6 different sensors and it goes on forever. Only restarting homeassistant resolves the issue temporarily. I am on version 0.87.1 (but it happened with previous versions as well).

So, my question is if I did anything wrong or have misunderstood the concept, or if this is a bug in the filter sensor?

I encountered similar issues with the outlier filter a few days ago (both the median vs previous issue and the filter getting “hung up” on step changes). I submitted a PR to correct these and it got merged yesterday, so the dev branch should now contain fixes for these problems.

On my own (hotfixed) server, I’m now getting the expected behaviour from an outlier filter in terms of filtering out actual “outliers” without also filtering step-changes in the sensor value that exceed the radius. Feel free to give the dev branch a shot (or patch your filter.py as described in the PR) and see if it works for you.

1 Like

@cf0 I am having an issue with a filter sensor but debug mode doesn´t work for some reason (no debug lines appear on the log for filter sensors), did you enable debug mode by adding this line to your config file or using other method?

logger:
default: info
logs:
homeassistant.components.sensor.filter: debug

Answering my own question just in case anyone arrives here with a similar issue, I just realised this is the correct line to enable debug for filter sensors:

logger:
  default: info
  logs:
    homeassistant.components.filter.sensor: debug