Filter sensor not working

rug · May 12, 2022, 11:22am

I collect data from my power meter via a serial interface (which is converted to MQTT). I have som noise which causes some values to be really wrong. Easiest way to fix this is to use the Filter component as data is sent every 5-10 seconds and I only get about 1-2 errors per hour.

This is my filter config:

 sensor:
 # Filtered sensors
   - platform: filter
     name: "filtered elimport"
     entity_id: sensor.import_el
     filters:
       - filter: outlier
         window_size: 1
         radius: 2

This filter uses imported kWh from my meter. My thought is to look at the previous value and if this differs more than 2kWh throw the value away and re-use the last value. This is not working. I get the same value in the filter sensor as the original sensor.
Original sensor:

Filtered sensor:

I tried turning on debug for the filter component but I see nothing in the log at all from this component:

logger:
  default: error
  logs:
    homeassistant.components.sensor.filter: debug

What am I doing wrong?

PeteRage · May 12, 2022, 11:33am

Try a larger window size. If you get two bad values in a row, with a window size of 1 the second will pass thru

rug · May 12, 2022, 11:55am

According to the documentation, the window size for the filter “outlier” is an integer for the number of states. It is not time. So just comparing the last recieved value should work as I have yet to see two wrong values received right after eachother. Also, averaging two values when the value is ever increasing does not feel right as the mean of those two values will always be lower than the last recieved correct value.

I have made a change to the config to a window size of 5 just to see if it makes a differens. Will report back as soon as I have an outlier.

rug · May 12, 2022, 1:44pm

@PeteRage: your suggestion acctualy worked! I do not know why using only one sample would not work though.

I tried to use a window size of 5 and increased the radius to 4 and it does work. It also does what I was affraid of. It averages the last 5 values and records a value slightly lower than the previous value. I do not think this will impact the energy metering though.

I will now try to decrease to a window size of 2 and see if it works.

I still do not get any debug information which would probably be useful to unstand why only using the last value does not work.

rug · May 12, 2022, 5:54pm

Nope, filtering with “outlier” does not do what I want or need it to do (remove new values that are very different from the previous value). I’ll have to find another solution…

PeteRage · May 12, 2022, 8:33pm

That’s unfortunate. I use these to filter out erroneous temperature values that I occasional receive. I stack the filter, first doing a range check then the outlier. Temperatures don’t fluctuate rapidly whereas power usage does - it may be hard to differentiate between a bad reading and the AC coming on.


sensor:
  - platform: filter
    name: "living_room_thermostat_temperature_filter"
    entity_id: sensor.living_room_thermostat_temperature
    filters:
      - filter: range
        lower_bound: 0
        upper_bound: 100
      - filter: outlier
        window_size: 4
        radius: 20

heisenberg · May 18, 2023, 2:34pm

@rug having a similar issue with a filter sensor and I cannot see any debug data either, did you find out why the debug information doesn´t work for filter sensors?

heisenberg · May 18, 2023, 2:54pm

Answering my own question just in case anyone arrives here with a similar issue, I just realised this is the correct line to enable debug for filter sensors:

logger:
  default: info
  logs:
    homeassistant.components.filter.sensor: debug

haraldh · November 15, 2024, 12:03pm

I have a similar task where I want to filter out calculated COP-values for a heat pump that don’t make sense.

# 2024-11-15 Harald
sensor:
  - platform: filter
    name: "Realistic COP Gree"
    entity_id: sensor.gree_cop
    filters:
    - filter: range
      lower_bound: -10
      upper_bound: 10

This doesn’t do what I think it should. History data might be unfiltered, but I now get “unavailable”. Is this filter correctly configured?

In the logs it looks like the filter is trying to parse comments i YAML?

2024-11-15 13:58:34.587 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template<template=({#       250, 270, 320, 390, 430, 470, 500  m³/h from Gree Pular Service Manual #}
    ( ( volume * airdensity * ( states('sensor.greeesp_temperature_out') | float - states('sensor.greeesp_temperature_in') | float ) ) * (1000/3600) ) /
2024-11-15 13:58:34.589 ERROR (MainThread) [homeassistant.components.template.template_entity] TemplateError('UndefinedError: dict object has no element None') while processing template 'Template<template=({#       250, 270, 320, 390, 430, 470, 500  m³/h from Gree Pular Service Manual #}
    ( ( volume * airdensity * ( states('sensor.greeesp_temperature_out') | float - states('sensor.greeesp_temperature_in') | float ) ) * (1000/3600) ) /
}}) renders=4>' for attribute '_attr_native_value' in entity 'sensor.gree_cop'
    ( ( volume * airdensity * ( states('sensor.greeesp_temperature_out') | float - states('sensor.greeesp_temperature_in') | float ) ) * (1000/3600) ) 
    ( ( volume * airdensity * ( states('sensor.greeesp_temperature_out') | float - states('sensor.greeesp_temperature_in') | float ) ) * (1000/3600) ) 
}}) renders=4>' for attribute '_attr_native_value' in entity 'sensor.gree_heatoutput'

edit: the template I’m trying to filter looks like this;

{#       250, 270, 320, 390, 430, 470, 500  m³/h from Gree Pular Service Manual #}
{# omitted lowest (silent?) and highest (turbo?) #}
{% set fanspeed = ({"low":"270", "medium low":"320", "medium":"390", "medium high":"430", "high":"470"}) %}
{% set airdensity = 1.17 %}
{% set cop_min = -10 %}
{% set cop_max = 10 %}
{% if state_attr('climate.c6a61428','preset_mode') == "away" %}
   {% set volume = fanspeed["low"] |float %}
{% else %}
  {% set volume = fanspeed[state_attr('climate.c6a61428','fan_mode')]|float %}
{% endif %}
{# Air volume is {{ volume }} m³/h #}
{{ 
    ( ( volume * airdensity * ( states('sensor.greeesp_temperature_out') | float - states('sensor.greeesp_temperature_in') | float ) ) * (1000/3600) ) /
    ( states('sensor.shellygaragetilp_power') | float )  
}}

edit²: It looks like it works, but it doesn’t?

edit³: Oh, it creates a new sensor sensor.realistic_cop_gree and I have to check that instead. OK