Outlier Filter Does Not Feel Like It's Working Properly

[MrCaspan]

Sorry everyone,

I decided to make my own thread on this instead of jumping on that old thread here.

Here is the old post talking about this:

OUTLIER filter does not filter

Okay it has finally happened again and I wanted to document it so someone can tell me what I am doing wrong or that there is a bug somewhere the filter settings. I have a Z-Wave plug for my washing machine it is a NAS-WR01Z by Inteset. The device reported a spike in energy reading for some reason: This is from the Washing Machine device history: [image] This is the history for the filter that I setup for the washing machine: [image] You can see that both are showing the spike in energy c…

I just did a test and created a template sensor called “123 Test” that is set to {{ 0 }}. I then created a filter that used this as input and set it to the same settings window side: 1 radius 0.5, precision 4

I change the settings from 0.1 to 1.0 in 0.1 increments

2025-03-31 14:56:10.533 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:55:11.696613-04:00>, new_state=<state sensor.123_test_sensor=0.1; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:10.531203-04:00>>
2025-03-31 14:56:10.534 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.1) -> 0.1
2025-03-31 14:56:13.127 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.1; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:10.531203-04:00>, new_state=<state sensor.123_test_sensor=0.2; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:13.126162-04:00>>
2025-03-31 14:56:13.127 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.2) -> 0.2
2025-03-31 14:56:15.595 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.2; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:13.126162-04:00>, new_state=<state sensor.123_test_sensor=0.3; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:15.592532-04:00>>
2025-03-31 14:56:15.596 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.3) -> 0.3
2025-03-31 14:56:17.854 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.3; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:15.592532-04:00>, new_state=<state sensor.123_test_sensor=0.4; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:17.849458-04:00>>
2025-03-31 14:56:17.854 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.4) -> 0.4
2025-03-31 14:56:20.270 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.4; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:17.849458-04:00>, new_state=<state sensor.123_test_sensor=0.5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:20.267150-04:00>>
2025-03-31 14:56:20.271 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.5) -> 0.5
2025-03-31 14:56:22.546 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:20.267150-04:00>, new_state=<state sensor.123_test_sensor=0.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:22.545696-04:00>>
2025-03-31 14:56:22.546 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.6) -> 0.6
2025-03-31 14:56:25.271 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:22.545696-04:00>, new_state=<state sensor.123_test_sensor=0.7; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:25.261701-04:00>>
2025-03-31 14:56:25.272 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.7) -> 0.7
2025-03-31 14:56:27.649 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.7; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:25.261701-04:00>, new_state=<state sensor.123_test_sensor=0.8; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:27.646329-04:00>>
2025-03-31 14:56:27.650 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.8) -> 0.8
2025-03-31 14:56:30.583 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.8; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:27.646329-04:00>, new_state=<state sensor.123_test_sensor=0.9; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:30.580650-04:00>>
2025-03-31 14:56:30.584 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=0.9) -> 0.9
2025-03-31 14:56:36.982 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=0.9; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:30.580650-04:00>, new_state=<state sensor.123_test_sensor=1.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:36.980704-04:00>>
2025-03-31 14:56:36.983 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.0) -> 1.0
2025-03-31 14:57:29.321 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:36.980704-04:00>, new_state=<state sensor.123_test_sensor=5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:29.318384-04:00>>

The filter works as expected

Then I change from 1.0 to 5.0

2025-03-31 14:57:29.321 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:56:36.980704-04:00>, new_state=<state sensor.123_test_sensor=5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:29.318384-04:00>>
2025-03-31 14:57:29.322 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 1 in sensor.123_test_sensor: 5.0
2025-03-31 14:57:29.323 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=5) -> 1.0

Then I changed the value back to 1.0

2025-03-31 14:57:34.743 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:29.318384-04:00>, new_state=<state sensor.123_test_sensor=1.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:34.740110-04:00>>
2025-03-31 14:57:34.744 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 2 in sensor.123_test_sensor: 1.0
2025-03-31 14:57:34.745 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.0) -> 5.0

I then updated it from 1.0 → 1.6 by 0.1 increment

2025-03-31 14:57:37.599 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:34.740110-04:00>, new_state=<state sensor.123_test_sensor=1.1; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:37.596326-04:00>>
2025-03-31 14:57:37.599 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.1) -> 1.1
2025-03-31 14:57:40.681 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.1; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:37.596326-04:00>, new_state=<state sensor.123_test_sensor=1.2; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:40.678844-04:00>>
2025-03-31 14:57:40.682 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.2) -> 1.2
2025-03-31 14:57:43.450 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.2; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:40.678844-04:00>, new_state=<state sensor.123_test_sensor=1.3; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:43.445180-04:00>>
2025-03-31 14:57:43.457 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.3) -> 1.3
2025-03-31 14:57:46.610 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.3; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:43.445180-04:00>, new_state=<state sensor.123_test_sensor=1.4; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:46.608001-04:00>>
2025-03-31 14:57:46.611 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.4) -> 1.4
2025-03-31 14:57:49.098 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.4; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:46.608001-04:00>, new_state=<state sensor.123_test_sensor=1.5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:49.095900-04:00>>
2025-03-31 14:57:49.099 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.5) -> 1.5
2025-03-31 14:57:52.597 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:49.095900-04:00>, new_state=<state sensor.123_test_sensor=1.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:52.597159-04:00>>
2025-03-31 14:57:52.598 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.6) -> 1.6

Again this works as expected

Then again I change it from 1.6 to 5.0

2025-03-31 14:57:57.115 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:52.597159-04:00>, new_state=<state sensor.123_test_sensor=5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:57.112598-04:00>>
2025-03-31 14:57:57.116 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 3 in sensor.123_test_sensor: 5.0
2025-03-31 14:57:57.116 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=5) -> 1.6

Then updated from 5.0 to 1.6

2025-03-31 14:58:01.701 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=5; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:57:57.112598-04:00>, new_state=<state sensor.123_test_sensor=1.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:01.698912-04:00>>
2025-03-31 14:58:01.703 DEBUG (MainThread) [homeassistant.components.filter.sensor] Outlier nr. 4 in sensor.123_test_sensor: 1.6
2025-03-31 14:58:01.703 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.6) -> 5.0

Then updated from 1.6 to 2.0 by 0.1 increments

2025-03-31 14:58:04.161 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.6; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:01.698912-04:00>, new_state=<state sensor.123_test_sensor=1.7; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:04.158100-04:00>>
2025-03-31 14:58:04.162 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.7) -> 1.7
2025-03-31 14:58:06.874 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.7; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:04.158100-04:00>, new_state=<state sensor.123_test_sensor=1.8; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:06.871061-04:00>>
2025-03-31 14:58:06.874 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.8) -> 1.8
2025-03-31 14:58:09.425 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.8; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:06.871061-04:00>, new_state=<state sensor.123_test_sensor=1.9; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:09.423232-04:00>>
2025-03-31 14:58:09.426 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=1.9) -> 1.9
2025-03-31 14:58:13.895 DEBUG (MainThread) [homeassistant.components.filter.sensor] Update filter on event: <Event state_changed[L]: entity_id=sensor.123_test_sensor, old_state=<state sensor.123_test_sensor=1.9; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:09.423232-04:00>, new_state=<state sensor.123_test_sensor=2.0; state_class=total, unit_of_measurement=kWh, friendly_name=123 Test Sensor @ 2025-03-31T14:58:13.892191-04:00>>
2025-03-31 14:58:13.896 DEBUG (MainThread) [homeassistant.components.filter.sensor] outlier(sensor.123_test_sensor=2.0) -> 2.0

So if you can see the 5 values are still showing up

Why does it not filter out these results?

[MrCaspan]

I believe I know what is happening now…

Let’s take the following table of Original sensor values (OSV) with the expected filter value(EFV) calculations beside it. This table assumes the filter is taking it’s window values from the EFV not the OSV

OSV      EFV
0.0  ->  0.0
0.1  ->  0.1
0.2  ->  0.2  
0.3  ->  0.3
0.4  ->  0.4
....
0.9  ->  0.9
1.0  ->  1.0
5.0  ->  1.0
1.0  ->  1.0
1.1  ->  1.1  
.....
1.6  ->  1.6
5.0  ->  1.6
1.6  ->  1.6
1.7  ->  1.7

This is how I thought it would function but what I am not taking into account is where the filter is pulling its window values from. Is it pulling its window values from the filtered sensor or the original sensor values.

As I stated in the above table I assumed it pulled its window values form the filtered sensor. If I change it to show what the values would be if it pulled the window values form the original sensor values then the spike still happens

0.0  ->  0.0
0.1  ->  0.1
0.2  ->  0.2  
0.3  ->  0.3
0.4  ->  0.4
....
1.0  ->  1.0
5.0  ->  1.0  - This would caught and filtered properly
1.0  ->  5.0  - In this case the filter would look at the previous read from the OSV which is 5.0 and 1.0 is not in range so 1.0 would replace that value
1.1  ->  1.1  - then back to normal
.....
1.6  ->  1.6
5.0  ->  1.6
1.6  ->  5.0
1.7  ->  1.7

So I think this is the issue… The point of this filter is to smooth out any spikes NOT to eliminate them. Each reading that is off would brigh the new value closer and closer tothe real value till its close enough. I thought it could do a double job as a way to drop any outlier value. I believe this is the expected

Thought Ideas on how to actually filter out bad reads?