Filter sensor not being updated when source sensor is updated

[baz123 (Brian)]

I do not understand why the filter sensor is not updated when the source sensor is. I have defined a time_date sensor. I would expect the filter to update every time the source sensor is updated. Is that an incorrect expectation? This is a moving window but I have also tried it with lowpass with same behaviour. The system has been running for a while since last restart and I can see delays of over 30 mins sometimes.

Sensor and filter

- platform: time_date
  display_options:
    - 'time'
    - 'date'
    - 'date_time'
    - 'date_time_utc'
    - 'date_time_iso'
    - 'time_date'

## Study
- platform: mqtt
  name: "studyroomtemp"
  state_topic: "emon/sensor/mi_t_4c65a8d8e0d8/state"
  unit_of_measurement: '°C'
  device_class: temperature

- platform: filter
  name: studyroomtempfil
  entity_id: sensor.studyroomtemp
  filters:
    - filter: time_simple_moving_average
      window_size: 00:05
      precision: 1

Looking at the states

{{ states('sensor.studyroomtemp' )}}
{{ states.sensor.studyroomtemp.last_updated}}
{{ states('sensor.studyroomtempfil' )}}
{{ states.sensor.studyroomtempfil.last_updated}}

20.21
2020-03-30 09:50:12.171099+00:00
20.5
2020-03-30 09:48:40.278234+00:00  

[tom_l]

Try:

window_size: '00:05'

[baz123 (Brian)]

No difference. same for lowpass as well.

[AhmadK (akasma74)]

@baz123

I would expect the filter to update every time the source sensor is updated. Is that an incorrect expectation?

possibly. I find the filter integration a bit tricky to configure properly and had to look inside many times
what do you want to achieve anyway?

[baz123 (Brian)]

@AhmadK

what do you want to achieve anyway?

Just trying to smooth some temperatures. I could do it in Node-Red but if the filter but if the filter sensor has a problem that should be raised as an issue.

[AhmadK (akasma74)]

I don’t think it’s a problem… have you tried to set the filter’s log level to debug and see what it’s doing?
The reason your filtered state is not updated is because it hasn’t changed.
It hasn’t changed because of the filter’s configuration - it looks at previous value(s) and calculates the current one and pushes it as a new one but if it’s the same as the state machine has, you won’t see any change because that’s the way HA works (unless you force_update).

So you need to find a way to configure your filter(s) so they work as you want.

I had an issue with ‘spikes’ in my room temperature’s readings and tried filter with no satisfactory outcome.
Ended up creating a custom jitter_filter and my hair temperature graph is much smoother now

[petro]

1. How many samples occur in a 5 minute window?
2. Does this sensor have a recorded history?

EDIT: And my last piece of advice, if the temperature hasn’t changed on the main device, a record is not created in the database. If the record doesn’t get created, you won’t see a change on the filtered device. That coupled with a low number of samples in the 5 minute window, you may end up with results that only update periodically. Especially if the temperature doesn’t change much.

[baz123 (Brian)]

@AhmadK

It hasn’t changed because of the filter’s configuration - it looks at previous value(s) and calculates the current one and pushes it as a new one but if it’s the same as the state machine has, you won’t see any change because that’s the way HA works (unless you force_update ).

Ah that makes sense.

@AhmadK

Ended up creating a custom jitter_filter

Care to share?

@petro, thanks yes I see that. I’ll revert to my Node-Red filtering as that seems to produce a better result,

[AhmadK (akasma74)]

@baz123

Care to share?

I think I’ll post it in Share your project one day but need time to at least a) upload it to Github and b) write readme.md as it has some more features compared to ordinary sensors - for example, I can dynamically change its attributes (like window_size etc) without restarting HA so a clever automation can adjust the filtering process depends on some factors.
The main thing is it’s designed for sensors that send their states regularly like my temperature sensors (every 50s).

[baz123 (Brian)]

Sounds great.

[2.m (Martynas Mickevičius)]

I am in a similar situation. I have a sensor that reports either 0 or 100 values. But when it is flipping from one value to another, it is flipping between the two for 5 minutes or so until it gets stable.

I wanted to write a filter sensor that would smooth out the period when it is flipping back and forth. The filter sensor works correctly until the original sensor flips the last time. After that the filter sensor is not updated anymore and the filter value is stuck.

The jitter_filter would be useful here. Would you mind sharing it?

[HeyImAlex (Alex)]

Just ran into the same issue. The filter component is completely broken. As it only updates when the source data actually changes, it can’t deal with stable source data and will get stuck on its last (incorrect) value as soon as the source stabilizes. The only situation where it doesn’t generate incorrect values is when the source is guaranteed to constantly change (basically like noise).

I’m trying to lowpass filter wind speed data. As soon as there is no wind anymore (source value is 0), the filtered value gets stuck on its last value, even if it’s hours / days old. Is there any way to trigger an update of the filter value manually or through an automation, say every n minutes ? Calling update_entity does not work with the filter component, it’s simply ignored.

[DanBendavid]

I have the same issue. Any alternative ?

[HeyImAlex (Alex)]

I ended up ditching the filter sensor. I rewrote the filter myself and integrated it directly into the firmware of the MCU that provides the data. So it now provides the prefiltered data directly to HA.

Another alternative is to use an automation with a time trigger, firing maybe every 5 minutes, that will do the filtering and write the result into an input_number. I did this for a trend filter for atmospheric pressure and it works well. I used pyscript to do it, as it provides persistent storage for intermediate values.

[DanBendavid]

Thank you very much for the fast reply. Could you share the PyScript ?

[HeyImAlex (Alex)]

It wouldn’t be too helpful, as it has tons of dependencies and won’t work on its own. But at the core it’s just a simple time trigger. It will process the value at regular intervals even if it didn’t change, so the filtered results will be correct. Rough example with a very simple exponential lowpass:

@time_trigger("period(now+10s, 5m)")
def process_filter():
    a = sensor.raw_value
    b = input_number.filtered_value
    w = 0.1
    input_number.filtered_value = a * w + b * (1 - w)

To make it robust, you need to add code handling situations where your sensor is unavailable, etc.

[bjiirn (Björn)]

Is there still no general solution? It would be great if there was an option to use the last value in the window instead of the first.
In my case it is power consumtion of a heater (orange) that apears to use more power than it actually does.
(White is general power consumtion of all devices.)

 - platform: filter
   unique_id: "modbus_energy_meter_2_l1_power_5s"
   entity_id: sensor.modbus_energy_meter_2_l1_power
   filters:
     - filter: time_throttle
       window_size: "00:00:05"
     - filter: outlier
       window_size: 4
       radius: 4.0
     - filter: range
       upper_bound: 4000
       lower_bound: 0

[petro]

You can write an automation to update the sensor when the source isn’t changing using the homeassistant.update_entity service. Slap a timepattern trigger on that and you’ll have a consistent update frequency on any entity you supply to the homeassistant.update_entity service.

[mariojas (Mariusz)]

It does not work brother.

[HeyImAlex (Alex)]

Unless they fixed that since, last time I tried the filter sensor will simply ignore calls to update_entity. That’s why I eventually turned to using the pyscript (see snippet above) to do my own filtering.