Ignore sensor values outside of a certain acceptable range

Filter sensor might be exactly what you need. Particulary outlier filter filters out any value outside a range.

4 Likes

its not exactly what you need, as you cannot define a range of permissable values in the outlier filter. Its close to the “range” filter, but range just floors and caps the values, not ignore them. :slight_smile:

This works! However… Isnt it problematic that it returns a nan value? Wouldnt it be better to just return the last “proper value”?

1 Like

Would a filter work?

@Benjamin_De_Smet
Brilliant! I love it. The only thing is that I’d much prefer if the “else” would just return the last valid reading. Any idea if that would be possible?

ok, but which filter with what attributes gives result as below:

  • if new sensor value is out of given range show last correct value
  • if not just show it
    ?

Hey,

you could use the “variable” component, to always save the last valid value from the sensor, as soon as it’s read. That way you have a ‘saved state’ sort of solution.
I’m sure there are more elegant solutions. Though I can think of none right now ^^.

If you’re new to the variable component, this helped me a lot:

When I start HA, Zwave devices get their initial values from cfg***.xml
My energy usage meter has cumulative values, but when HA starts, HA shows the value from file.
Every time I restart, it shows in dip in cumulative energy usage.

Wolud it be possible to create a filter that simply ignores values that are out of range. The Filter- component does not seem to ignore values but interpolates new values. It would be better to have an option just to drop the wrong values. Not recorded anywhere and not propagating value changes to objects depending on it?

I solved this by assigning template sensor its own value back.
Beware that if you do this you need to define entity_id, otherwise automatic detection will start infinite update loop.

1 Like

Hi @Korl, I’m in the process of try to do something smiler, whoud you be able to share your temolate sensor? I have a sensor where I want it to avoid any “0” or “100” values, as when the sensor get this value it’s usually a mistak and not a real value.

Here’s one of mine adapt it as you need : -

      s_heat_thermostat_temp:
        friendly_name: Heating Sensor
        value_template: >
          {% set val = states('sensor.qubino_qt1_temp') | float %}
          {% if val and val != 0 % and 0.1 < val < 62.6 }
            {{ '%.2f' | format(val) }}
          {% else %}
            {{ states('sensor.s_heat_thermostat_temp') }}
          {% endif %}
        icon_template: "{{ 'mdi:thermometer' }}"
        unit_of_measurement: '°C'
1 Like

having the same issue with a swimming pool thermometer (sonoff th16).
get the odd wild -2031c.
Really struggling with filtering out the rogue values.

temps are between 10 and 38c all year round. Daily variances are between 2-3c max so graphs are smooth but all out of shape with the odd values.

Another example would be appreciated.

Thanks

great thread, just implemented a template sensor using these examples to split - and + values from a grid power monitoring sensor

I have to bump that thread up again, sorry for that :slight_smile:

I mostlikely have the same issue, but I’m not able to limit the values to a specific range, because the sensor I’m using is serving data to my energy meters, and is a total_increasing one.

image

As you can see, this is the sensor grabbing data from my smart meter about the power from my photovoltaic system, injected back to the grid.

Due to the hiccups of the sensor, my energy meters for daily, weekly, monthy and so on, gets confused and this is messing up the historical data over the recording period.

Any chance I can filter the sensors value to get it sorted out, when it’s totaly different from the previous one that has been recorded?

You could use a derivative sensor to detect the anomalies and use this as an availability template in a template sensor.

Sounds good, even if I don’t have an idea how to implement it :joy:

I have tried to solve it with filter sensor as @jesjimher suggested. It works well in some cases:

sensor:
  - platform: filter
    name: "Living room temp. fixed"
    entity_id: sensor.livingroomtemp
    filters:
      - filter: outlier
        window_size: 4
        radius: 2.0

Although if the peaks are quite often (can happen more than once in the “window_size”) then it won’t work. Reason is that it doesn’t just reject the data and gives the previous state but it calculates and gives the median of values from the previous window. So if one of the values was extremally high/low then the result sensor value will be affected.

The only solution which works for me is via template sensor as people above suggested. This is taken from my config and it works perfectly:

sensor:
  - platform: template
    sensors:

      energy_usage:
        friendly_name: "Power"
        device_class: energy
        unit_of_measurement: W
        value_template: >
          {% if states('sensor.owl_energy_usage') | int > 20000 %}
           {{ states('sensor.energy_usage') }}
          {% else %}
           {{ states('sensor.owl_energy_usage') | int }}
          {% endif %}

If value exceeds max allowed number it just returns previous template state otherwise it returns original sensor value.

Edit: actually I have created little bit improved template sensor which validates the movement regardless of direction. So you can specify for example that the temperature shouldn’t jump by 10 degrees between readings regardless if it’s +10 or -10:

sensor:
  - platform: template
    sensors:
      living_room_temp_fixed:
        friendly_name: "Living room temperature"
        device_class: temperature
        unit_of_measurement: °C
        value_template: >
          {% if states('sensor.living_room_temp_fixed') | lower == "unavailable" %}
            {{ (states('sensor.livingroomtemp') | float) }}
          {%- else -%}
            {% if (states('sensor.livingroomtemp') | float - states('sensor.living_room_temp_fixed') | float) | abs > 10 %}
              {{ (states('sensor.living_room_temp_fixed') | float) }}
            {%- else -%}
              {{ (states('sensor.livingroomtemp') | float) }}
            {%- endif %}
          {%- endif %}

Of course in this case the source sensor is livingroomtemp and the one with fixed temp is living_room_temp_fixed.

This should work quite well but it is based on the assumption that the first sensor reading is correct :wink:

1 Like

Hi, could you help me for a little problem?
I use you yaml but when the temperature goes under negative value I set, yaml report 0, not last

- platform: template
    sensors:
      temperatura_esterna_fix:
        friendly_name: "Temperatura esterna fix"
        device_class: temperature
        unit_of_measurement: °C
        value_template: >
          {% if (states('sensor.temperature_49_fix') | float(0)) < -50 %}
            {{ (states('sensor.temperature_49') | float(0)) }}
          {%- else -%}
            {% if (states('sensor.temperature_49') | float - states('sensor.temperature_49_fix') | float(0)) | abs > 20 %}
              {{ (states('sensor.temperature_49_fix') | float(0)) }}
            {%- else -%}
              {{ (states('sensor.temperature_49') | float(0)) }}
            {%- endif %}
          {%- endif %}

Outdoor temperature sensor, randomly, give -100°C, so I used your code. But, if I use ==-100 or < -50 or other values to report last true value, I’ve 0
Thank you

Been digging for weeks looking for a solution, this thread points in the right direction but not exactly what i need…

I have many weight scales, using HX711, once calibrated to 0 and a known weight reference things work well… but on occasion the HX711 ( or the esp8266 does… not sure… ) reports a false number, way… WAY out of range…

I’d like to somehow impose a rule to ignore any value below 0 ( nothing weighs less than zero lol ) … and anything over a set High Limit… say 8 kg…

Also, i’d like to have a way to limit a sudden swing, having it go Up or Down by 50 grams per second at most… so say the platform is at 0g… put a 4 Kg weight on it and HA would ramp up the weight by 50g per second until the target is reached, then stabilize … this would make the history graphs nice and smooth.

Got a whole lot of these weight sensors ( 24 and growing… ) in my setup and it would be nice if i could implement this for all weight sensors in a single paragraph but can’t figure out how to do it. or where to put it…etc… … i’m trying my best but if anyone can point me in some direction or help, would be fantastic… thx.

Maybe this can help you
Method required to filter out extreme bad data · Issue #744 · esphome/issues (github.com)