What is the best way to eliminate 'spikes' in temperature/humidity sensors' readings in real time?

Hi All,

I use some DIGOO R8H temperature&humidity sensors in my setup and they work very well.
The readings come to HA via RFLink integration.
I noticed on the sensors’ history graph that once in a while, totally randomly, both temperature and humidity readings (they come in one message afaik) get far away from the previous values (namely - the temperature jumps some 10 degrees up and the humidity plunges some 20% down or vice versa). Apparently the next reading is back to normal so the graph looks absolutely fine apart from that momentary glimpse.

I know there are integrations like Filter or Threshold, but they all need some input readings before they start producing output. My physical sensors send their readings every 50 seconds and sometimes I miss a new reading and need to wait for the next one for another 50 seconds (maybe because of poor antenna(s), long distance, interference or a bit of all that - I tried to improve that, but there’s still not ideal). That’s why I just don’t want to use any of these integrations as they will make my slowly-updated sensors even slower and for some applications 2 minutes lag is a bit too much…

So far I use this template sensor to stop that spike propagating:

- platform: template
  sensors:
    rflink_filtered_outdoors_digoo_temperature:
      device_class: temperature
      unit_of_measurement: '°C'
      entity_id:
        - sensor.outdoors_digoo_temperature
      value_template: >
        {% set unknown = 'unknown' %}
        {% set value = states('sensor.outdoors_digoo_temperature') %}
        {% set prev_value = states('sensor.rflink_filtered_outdoors_digoo_temperature') %}
        {% if prev_value == unknown %}
          {{ value }}
        {% else %}
          {% set def_deviation = 5.0 %}
          {% set cur_deviation = states('input_number.thb_sensor_temperature_deviation') %}
          {% set deviation = def_deviation if cur_deviation == unknown else float(cur_deviation) %}
          {{ prev_value if value == unknown or (value | float - prev_value | float)|abs > deviation else value }}
        {% endif %}

It works, but only if the deviation is selected right - if it’s too small (imagine sharp rise/fall in temperature), the template is stuck until the readings get pretty close to the stored value (I witnessed that once).

I wonder if there is a better solution to this issue at all?

I created a statistics sensor based off my temperature sensor, and I use the average temperature for the last 10 minutes. It obviously lags a bit, but the graph is much smoother.

Forgot about Statistics. Do you use its state or median attribute? Assume it’s just max_age: 600 and nothing else.

I just use the state because I use it to trigger my thermostat automations. They’re the same though, it doesn’t really matter.

Well, mean and median are not the same, but it depends on the input.
Thanks for the pointer, I’ve created a similar sensor for each of my temp/hum sensors and will check on them from time to time to decide if it’s worth to use.

Would you please post an example of one of your Statistics sensors? Thanks!

It’s pretty straightforward if you read the docs:

sensor:
 - platform: statistics
    name: livingroom_temperature_statistics
    entity_id: sensor.livingroom_temperature
    sampling_size: 100
    max_age: '00:10:00'
    precision: 1
1 Like

I’m at work, and taking a break. Will definitely RTFM upon arrival home. Thank you!

There is also the possibility of wrapping the sensor in a template sensor that doesn’t let it increase or decrease more than a certain amount:

{%- set current = states('sensor.original') | float -%}
{%- set previous = states('sensor.this_sensor') | float -%}
{%- set delta = current - previous -%}
{%- set allowed = 0.2 %}

{%- if delta > allowed -%}
  {{ current + allowed}}
{%- elif delta < (allowed * -1) -%}
  {{ current - allowed }}
{%- else -%}
  {{ current }}
{%- endif -%}

I haven’t tried this one myself, but it should let the templated sensor catch up to the actual sensor in allowed increments.

Actually, that’s a good alternative idea comparing to my filtering sensor as it allows to change value towards the current.
Thanks for sharing!
I’ll rework it a little bit as in python subtraction plays nasty tricks with us so it’s better to avoid it in comparison…

UPDATE: I think in the second part of the code it should be {{ previous + allowed }} and {{ previous - allowed }} as we use the last readings as basis, not new ones. And don’t forget that initial state of the template sensor is unknown:wink:

Anyway, here’s my version:

- platform: template
  sensors:
    this_sensor:
      device_class: temperature
      unit_of_measurement: '°C'
      entity_id:
        - sensor.original
      value_template: >
        {% set current = states('sensor.original') %}
        {% set result = 'unknown' %}
        {% if current != result %}
          {% set current = current | float | round(1) %}
          {% set previous = states('sensor.this_sensor') | float %}
          {% set max_delta = 0.2 %}

          {% set result = current %}
          {% if previous %}
            {% if current > previous %}
              {% set upper_bound = previous + max_delta %}
              {% set result = [current, upper_bound] | min  | round(1) %}
            {% elif current < previous %}
              {% set lower_bound = previous - max_delta %}
              {% set result = [current, lower_bound] | max  | round(1) %}
            {% endif %}
          {% endif %}
        {% endif %}
        {{ result }}

and here is a shortened and compressed (without Jinja’s if) main part:

{% if previous %}
  {% set sign = current > previous %}
  {% set boundary = previous + max_delta if sign else previous - max_delta %}
  {% set pair = [current, boundary] %}
  {% set result = (pair | min if sign else pair | max) | round(1) %}
{% endif %}

It might be a bit shorter using python (not Jinja!), but that’s another story.

5 Likes

This will never work as you convert previous to float in the first time.