Filter is unknown when source is 0

Hi,
after restart filter state is “unknown” if source sensor is 0. Until source gets some value greater than 0. And it gets frozen at last value grater than 0 when source goes to 0.

homeassistant:
  customize:
      default_sensor_settings: &default_sensor
         slave: 1
         scan_interval: 1
         input_type: holding
         data_type: float32

modbus:
 - name: iem3150
   type: serial
   method: rtu
   port: /dev/com1
   baudrate: 38400
   stopbits: 1
   bytesize: 8
   parity: N
   timeout: 0.1
   message_wait_milliseconds: 3

   sensors:
    - name: "IEM3150 voltage"
      address: 3019
      <<: *default_sensor
      virtual_count: 7
      unit_of_measurement: "V"

    - name: "IEM3150 current"
      address: 2999
      <<: *default_sensor
      virtual_count: 3
      unit_of_measurement: "A"

    - name: "IEM3150 power"
      address: 3053
      <<: *default_sensor
      virtual_count: 3
      unit_of_measurement: "kW"

sensor:
 - platform: filter
   name: "L1 voltage"
   entity_id: sensor.IEM3150_voltage_4
   filters:
    # - filter: time_simple_moving_average
    #   window_size: "0:03"
    - filter: lowpass
      time_constant: 30
      precision: 1

 - platform: filter
   name: "L2 voltage"
   entity_id: sensor.IEM3150_voltage_5
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 1

 - platform: filter
   name: "L3 voltage"
   entity_id: sensor.IEM3150_voltage_6
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 1

 - platform: filter
   name: "L1 current"
   entity_id: sensor.IEM3150_current
   filters:
    # - filter: time_simple_moving_average
    #   window_size: "0:03"
    - filter: lowpass
      time_constant: 15
      precision: 2

 - platform: filter
   name: "L2 current"
   entity_id: sensor.IEM3150_current_1
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 2

 - platform: filter
   name: "L3 current"
   entity_id: sensor.IEM3150_current_2
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 2

 - platform: filter
   name: "L1 power"
   entity_id: sensor.IEM3150_power
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 3

 - platform: filter
   name: "L2 power"
   entity_id: sensor.IEM3150_power_1
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 3

 - platform: filter
   name: "L3 power"
   entity_id: sensor.IEM3150_power_2
   filters:
    - filter: time_simple_moving_average
      window_size: "0:03"
      precision: 3



Looks like 0 value in not an event because I don’t see in log any “Update filter on event” of those sensors.

You need at least one numeric state change - i.e. one number to another (not unknown or unavailable) - for the filter to update. It requires at least two numbers to calculate an average.

But filter working this way is useless, stable value is valid value, so it should be taken by filter too.

If it does not fit your needs you need to look elsewhere or develop your own filter integration.

I’m afraid it doesn’t fit anyone’s needs. Who needs to filter 0’s to “unknown” or previous nonzero value. A moving average should be an average across a window of values, regardless they are changing or not. In my case, all the values are valid 0’s.

Zero. Not plural. There is only one state.

That is your problem

No, there are many 0’s. HA it simplifies to “no change → no new state”.
It should work as any hw sensor (e.g. esthome) ADC or so, any new sample is input to filter. And here it’s every new value red from modbus.

0 to 0 is not a change.

Of course, but output of the filter should be 0!

I’ve already explained why that is not the case and what your options are.

Filters don’t work this way. Please, study some filtering backgroud. Moving average - Wikipedia

That’s how home assistants state machine works. It only provides (and stores) state changes, and 0 to 0 does not cause a state change. Meaning there’s only 1 data point in the database for filter to use. You’re getting hung up on the word filter without learning how HA works in the background.

The filter integration can only work with what it’s given. And if you configure it to have a time_constant of 30 seconds but your upstream sensor does not provide 30 seconds of data, that’s something you need to work around.

OK, I understand that HA works with state changes, but the filter does not behave correctly.
So up to your explanation of state changes are only taken in, then 60s window filter with 0 at 0s and 1 at 59s will result in 0.5?

on the second state change, the one at 59 seconds, the filter will look behind 60 seconds, take the 2 values for whatever filter you’re using. If the previous value does not fall within the 60 second window, it will only have 1 value to work with which will result in unknown if your filter requires at least 2 points.

The previous value falls within window. There are 2 values in window. What is the result?

What filter?

Moving average.

it would be:

moving_sum = (now_time - previous_time) * previous_value

then

moving_sum / configured_total_seconds

All documented here:

based on this whitepaper here:

http://www.eckner.com/papers/Algorithms%20for%20Unevenly%20Spaced%20Time%20Series.pdf

It’s not a traditional moving average with a known step size between points. It accounts for uneven step sizes.

code is here if you want to put in a PR to make it whatever you think it should be

So in regards to your question… the answer would be:

((1 * 0) + (60 * 1)) / 60 = 1

So my final solution is to run esphome on the same host as hass:

esphome:
   name: modbus-iem3150
   friendly_name: ""

host:
   mac_address: "02:de:ad:01:be:af"

api:
   port: 60531
   encryption:
      key: !secret API_key

button:
   platform: restart
   name: "restart"
   id: restart_id

logger:
   level: INFO

uart:
   port: /dev/com1
   baud_rate: 38400
   #debug:

modbus:
   send_wait_time: 77ms

modbus_controller:
   address: 1
   max_cmd_retries: 1
   update_interval: 1s

.iem3150: &common_iem3150
   platform: modbus_controller
   register_type: holding
   value_type: FP32
   state_class: measurement

.current: &common_current
   device_class: current
   unit_of_measurement: "A"
   accuracy_decimals: 2
   filters:
    - sliding_window_moving_average:
         window_size: 32
         send_every: 4

.voltage: &common_voltage
   device_class: voltage
   unit_of_measurement: "V"
   accuracy_decimals: 1
   filters:
    - sliding_window_moving_average:
         window_size: 256
         send_every: 4

.power: &common_power
   device_class: power
   unit_of_measurement: "kW"
   accuracy_decimals: 3
   filters:
    - sliding_window_moving_average:
         window_size: 32
         send_every: 4

sensor:
 - platform: uptime
   name: "uptime"
   id: uptime_id

 - name: "L1 current"
   address: 2999
   <<: *common_iem3150
   <<: *common_current

 - name: "L2 current"
   address: 3001
   <<: *common_iem3150
   <<: *common_current

 - name: "L3 current"
   address: 3003
   register_count: 16
   <<: *common_iem3150
   <<: *common_current

 - name: "L1-L2 voltage"
   address: 3019
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L2-L3 voltage"
   address: 3021
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L3-L1 voltage"
   address: 3023
   register_count: 4
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L1 voltage"
   address: 3027
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L2 voltage"
   address: 3029
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L3 voltage"
   address: 3031
   register_count: 22
   <<: *common_iem3150
   <<: *common_voltage

 - name: "L1 power"
   address: 3053
   <<: *common_iem3150
   <<: *common_power

 - name: "L2 power"
   address: 3055
   <<: *common_iem3150
   <<: *common_power

 - name: "L3 power"
   address: 3057
   <<: *common_iem3150
   <<: *common_power

 - name: "Total power"
   address: 3059
   register_count: 24
   <<: *common_iem3150
   <<: *common_power

 - name: "Power Factor"
   address: 3083
   register_count: 26
   <<: *common_iem3150
   device_class: power_factor
   unit_of_measurement: ""
   accuracy_decimals: 3
   filters:
    - sliding_window_moving_average:
         window_size: 64
         send_every: 4

 - name: "Frequency"
   address: 3109
   <<: *common_iem3150
   device_class: frequency
   unit_of_measurement: "Hz"
   accuracy_decimals: 3
   filters:
    - sliding_window_moving_average:
         window_size: 64
         send_every: 4

 - name: "Total energy"
   address: 3203
   <<: *common_iem3150
   value_type: S_QWORD
   device_class: energy
   unit_of_measurement: "kWh"
   state_class: TOTAL_INCREASING
   accuracy_decimals: 2
   filters:
    - multiply: 0.001