OUTLIER filter does not filter

[gegy (Georg)]

Hello.
i am reading the values from a temperature sensor. Sometimes the sensor sends very high values. (e.g. 6000 C°). This mess up my history graph card. So i implemented a filter sensor but it seems this does not filter correctly my data. I want to filter all values over 60 C°. This is my filter:

- platform: filter
  unique_id: "heizung_aussentemperatur_gefiltert" 
  name: "Außentemperatur (C°)"
  entity_id: sensor.heizung_aussentemperatur
  filters:
      - filter: outlier
        window_size: 4
        radius: 60.0
      - filter: lowpass
        time_constant: 5
        precision: 2

Can somebody tell me why my filter does not filter correctly?

[tom_l]

@gegy

      - filter: outlier
        window_size: 4
        radius: 60.0

That will allow any value from the 4 sample average of your sensor’s current state +/- 60.

So if your sensor reads 1000, 1000, 1000, 1000 then the next allowable value could be 940 to 1060.

[gegy (Georg)]

Oh, ok. I think this could be the solution. I will give it a try. Thank you very much!
One additional question: If i set the window_size to 40 for e.g, and it has read 40 samples, then it will drop the first sample of the 40 and adds the new sample at the end, right?

[0] = 600, [1] = 650, [1] = 850,… [40] = 345
now a new high value is added and the new range looks like this, right?
[0] = 650, [1] = 850, …[40] = 2000

Or would it be better not to use the window_size property?

[gegy (Georg)]

@gegy

outlier

hmmmm… i have read your answer now twice. So if set a radius value of 60 and a window_size of 1 and the sensor reads the value 6000, then 5940 is allowed?
But how can i set a maximum and a minimum? How can i set a filter which ignores all values over 60 °C and below -60 °C?

Seems the range filter is the better choice for this task.

But it is not he best solution, because it replaces the outbound values with the upper/lower bound value, what will cause strange spikes in a diagram. A “ignore” values filter would be the best but there is no such filter.

[phio (Philip Cheong)]

I have not got the outlier filter to work correctly yet either.

This is my filtered sensor:

- platform: filter
  name: "All appliances energy"
  entity_id: sensor.appliances_plus_computer_energy
  unique_id: sensor.filtered_appliances_plus_computer_energy
  filters:
    - filter: outlier
      window_size: 20
      radius: 1.0

The source sensor is a whole load of energy sensors (Shelly devices and Aqara outlets):

- sensor:
  - name: "All appliances energy"
    unique_id: sensor.appliances_plus_computer_energy
    state: >
      {% set energy = states('sensor.shelly_2pm_plus_kitchen_switch_1_energy') | float(0)
      + states('sensor.shelly_2pm_plus_kitchen_switch_0_energy') | float(0)
      + states('sensor.shelly_1pm_plus_dishwasher_switch_0_energy') | float(0)
      + states('sensor.shelly_1pm_plus_fridges_switch_0_energy') | float(0)
      + states('sensor.shelly_2pm_plus_laundry_switch_0_energy') | float(0)
      + states('sensor.shelly_2pm_plus_laundry_switch_1_energy') | float(0)
      + states('sensor.freezer_outlet_energy') | float(0)
      + states('sensor.electrical_cabinet_outlet_energy') | float(0)
      + states('sensor.internet_outlet_energy') | float(0)
      + states('sensor.tv_outlet_energy') | float(0)
      + states('sensor.office_outlet_energy') | float(0) %}
      {{ energy }}
    unit_of_measurement: kWh
    state_class: total_increasing
    device_class: energy

But I still occasionally get sensor total values creeping into the statistics which seem to be from restarts of HA. I’ve had multiple erroneous values today from morning and evening restarts.

Normal values are expected in the range of 0.01 - 0.1 kWh:

However, the abnormal values are in the 100 - 1000 kWh range. From the explanation that @tom_l gave, I would expect this outlier filter to eliminate those values…

[tom_l]

@phio

The source sensor is a whole load of energy sensors (Shelly devices and Aqara outlets):

Add an availability template to that instead of trying to filter it.

[phio (Philip Cheong)]

Yeah that is definitely a step in the right direction. Now my sensors seem to have less garbage values and I think I finally understand availability templates.

However with the following availability template, the sensor is still quite brittle because I’ve noticed that if a device upgrades (particularly the Shellys) then it’s data will typically be reset, meaning my availability template will return false until I go back to edit the sensor to correct for the new value.

Is there a better way to handle that situation than what I am doing below?

- sensor:
    - name: "All appliances energy"
      unique_id: sensor.appliances_plus_computer_energy
      state: >
        {% set energy = states('sensor.shelly_2pm_plus_kitchen_switch_1_energy') | float(0)
        + states('sensor.shelly_2pm_plus_kitchen_switch_0_energy') | float(0)
        + states('sensor.shelly_1pm_plus_dishwasher_switch_0_energy') | float(0)
        + states('sensor.shelly_1pm_plus_fridges_switch_0_energy') | float(0)
        + states('sensor.shelly_2pm_plus_laundry_switch_0_energy') | float(0)
        + states('sensor.shelly_2pm_plus_laundry_switch_1_energy') | float(0)
        + states('sensor.freezer_outlet_energy') | float(0)
        + states('sensor.electrical_cabinet_outlet_energy') | float(0)
        + states('sensor.internet_outlet_energy') | float(0)
        + states('sensor.tv_outlet_energy') | float(0)
        + states('sensor.office_outlet_energy') | float(0) %}
        {{ energy }}
      availability: >
        {{ states('sensor.shelly_2pm_plus_kitchen_switch_1_energy') | float(0) > 3.0
        and states('sensor.shelly_2pm_plus_kitchen_switch_0_energy') | float(0) > 2.5
        and states('sensor.shelly_1pm_plus_dishwasher_switch_0_energy') | float(0) > 1.2
        and states('sensor.shelly_1pm_plus_fridges_switch_0_energy') | float(0) > 3.5
        and states('sensor.shelly_2pm_plus_laundry_switch_0_energy') | float(0) > 10.4
        and states('sensor.shelly_2pm_plus_laundry_switch_1_energy') | float(0) > 9.3
        and states('sensor.freezer_outlet_energy') | float(0) > 135.0 
        and states('sensor.electrical_cabinet_outlet_energy') | float(0) > 187.0
        and states('sensor.internet_outlet_energy') | float(0) > 448.0
        and states('sensor.tv_outlet_energy') | float(0) > 328.0
        and states('sensor.office_outlet_energy') | float(0) > 53.0 }}
      unit_of_measurement: kWh
      state_class: total_increasing
      device_class: energy

[tom_l]

Try this:

      availability: >
        {{ has_value('sensor.shelly_2pm_plus_kitchen_switch_1_energy')
        and has_value('sensor.shelly_2pm_plus_kitchen_switch_0_energy')
        and has_value('sensor.shelly_1pm_plus_dishwasher_switch_0_energy')
        and has_value('sensor.shelly_1pm_plus_fridges_switch_0_energy')
        and has_value('sensor.shelly_2pm_plus_laundry_switch_0_energy')
        and has_value('sensor.shelly_2pm_plus_laundry_switch_1_energy')
        and has_value('sensor.freezer_outlet_energy')
        and has_value('sensor.electrical_cabinet_outlet_energy')
        and has_value('sensor.internet_outlet_energy')
        and has_value('sensor.tv_outlet_energy')
        and has_value('sensor.office_outlet_energy') }}

As for the values resetting after a firmware update, one way around that would be to use a whole lot of utility meters fed by your energy sensors.

[RonnieLast (Last of the Ronnies)]

Is the outlier filter just not working as expected? I’m seeing multiple posts where it’s indicated not to be functional, even though the yaml looks ok.

Three examples are as follows:

Problem with outlier filter

Hi, I’ve got a sensor for outdoor temperature that I am reading via the Modbus integration (its attached to my heat pump). The modbus connection is spotty, resulting in 0s being reported instead of the actual value. As you can see here: [Screenshot 2023-11-20 at 14.25.33] I have set up a filtered version of this sensor using the Outlier filter, with the goal of removing these 0 outliers from the sensor. The filtered sensor is set up like this: - platform: filter name: "Filtered Elga O…

Outlier Filter Sensor not catching outliers (sometimes)

Hi, I have an issue with the outlier filter not catching outliers, it seems to work as expected most of the time, but then every now and then it misses one. It hasn’t happened often enough to be sure, but it feels like it mostly happens with very extreme ones. The first graph is the filtered sensor, the second one is the unfiltered one. [image] This is my configuration for the filter sensor: - platform: filter name: "Living Temp" entity_id: sensor.living_temp_raw filters: - filter…

OUTLIER filter does not filter

I have not got the outlier filter to work correctly yet either. This is my filtered sensor: - platform: filter name: "All appliances energy" entity_id: sensor.appliances_plus_computer_energy unique_id: sensor.filtered_appliances_plus_computer_energy filters: - filter: outlier window_size: 20 radius: 1.0 The source sensor is a whole load of energy sensors (Shelly devices and Aqara outlets): - sensor: - name: "All appliances energy" unique_id: sensor.appliances_pl…

In my own case I have a filter setup on a sensor because it generates a randomly high temperature reading from time to time. It’s always a single incorrect value and tends to be between 1000 to 10,000 where the nominal range of the temperature sensor is between 20 and 50C. In this case you would think that an outlier filter would catch these excessive values on the basis that the median value will remain low and the next value will be well outside that range?

Here’s my sensor outlier showing the one incorrect value:

And here’s the filter for it from sensor.yaml:

# Filters for the hot-tank sensors which are showing some noise from time-to-time
  - platform: filter
    name: "hot_tank_air_temperature_5-filtered"
    entity_id: sensor.hot_tank_air_temperature_5
    filters:
      - filter: outlier
        window_size: 2
        radius: 4.0

Any clues what might be going on here?

[binarybucks (Alexander Rust)]

The outlier filter seems to indeed work as designed but not as expected.

I’m using it to filter glitches from my heatpump MQTT interface which glitches a bit if the device is disconnected from the network. It the nsends a single wrong value that is several million times greater than the old one which especially screws up any utility meter that e.g. tracks energy consumption throughout the year.

I created a simple test for the outlier filter with an input number and the following outlier config

- platform: filter
  name: "outlier_test"
  entity_id: input_number.outlier_test_raw
  filters:
    - filter: outlier
      precision: 1
      window_size: 1
      radius: 10

From the docs I would have expected that if the input_number changes ±10 in a single step, the value is accepted and if it is more than that it uses the median of the previous window, hence the last value in case window_size is 1.
Setting it from 1 to 9 works as expected.

Setting the input number from 9 to 100 the resulting value is 9 .
The debug log notes that an outlier is detected

2024-04-26 07:20:12.779 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=input_number.outlier_test_raw, old_state=<state input_number.outlier_test_raw=9.0; initial=None, editable=True, min=0.0, max=1e+17, step=1.0, mode=box, friendly_name=outlier_test_raw @ 2024-04-26T07:19:57.840015+02:00>, new_state=<state input_number.outlier_test_raw=100.0; initial=None, editable=True, min=0.0, max=1e+17, step=1.0, mode=box, friendly_name=outlier_test_raw @ 2024-04-26T07:20:12.777708+02:00>>
2024-04-26 07:20:12.780 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 1 in input_number.outlier_test_raw: 100.0
2024-04-26 07:20:12.780 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(input_number.outlier_test_raw=100.0) -> 9.0

It goes wrong if now a normal value is set again. E.g. setting the input number from 100 (the glitch value) to e.g. 10
The resulting value is not 9 as one would expect, but 100, the previously filtered value as that of course is now part of the window_size.

2024-04-26 07:29:48.964 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 2 in input_number.outlier_test_raw: 10.0
2024-04-26 07:29:48.964 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(input_number.outlier_test_raw=10.0) -> 100.0``` 

Using a larger window_size does not change this problem, it would just decrase the overall value which might not be enough depending on how different the outlier is.

It would help if the median was calculated based on the outlier filter result and not the input entity as that would exclude any previously ignored values.

Update: I’ve raised this as issue Outlier filter not dropping filtered value · Issue #116226 · home-assistant/core · GitHub. The filter code has an explicit flag to include the raw input value in the previous states. Changing this or making it configurable would make the filter work as expected.