Outlier filter issues

Configuration
1

I’ve been playing around with the outlier filter for a few weeks now to filter out huge power spikes from my Aeotec Energy Meter like the one shown in the daily kWh chart attached. A 150 kWh instant jump is obviously an outlier, although sometimes they are more like 1-3 kWh sudden jumps that I need to filter out.

My config is below but as you can see there is very little difference between my filtered and unfiltered data (the lines are overlying each other in the charts). In fact most of the time my filtered data is higher!

What am I doing wrong?

  - platform: filter
    name: "live power filtered"
    entity_id: sensor.energymeter_electricity_w_meter
    filters:
      - filter: outlier
        window_size: 3
        radius: 1500.0
  - platform: filter
    name: "live kwh filtered"
    entity_id: sensor.energymeter_electricity_kwh_meter
    filters:
      - filter: outlier
        window_size: 3
        radius: 2.0

2 1

2

that radius seems huge. Those 6k spikes could be 8 readings of 1000 separation, making it seem like it’s doing nothing. Filter the live with a lower value. Seems like you barely break 500 w at any given moment so what’s the harm in using a value that low.

3

I went with a large radius as some electrical appliances do cause a sudden jump such as electric oven and kettle when used at once. But maybe I’ve misunderstood the settings and I’ll try as you say and report back

4

You really just need to find the stepsize that won’t be filtered by the filter. So I think you should analyze your jumps and analyze your devices turning on. Then choose a number that will properly filter one but not the other.

5

Also those spikes in your power sensor could very well be real. Turning on reactive loads like fridge compressor motors can draw huge current spikes.

6

It’s really the second of the two charts I was asking about, the one showing kWh which uses a different output from the meter not the live. The spikeyness of the live doesn’t bother me that much (I probably shouldn’t have posted it).

Yesterday the kWh output jumped by almost 150 kWh but my filter made no difference to the filtered data. It’s this consumption feed that I want to fix otherwise the stats held in the HA meter are completely wrong and far too high for monthly and annual consumption.

7

Are you using the integration sensor to calculate energy?

If so Petro wrote a great post on why the method chosen is important,

If not, whoever designed your energy meter may have chosen a poor integration method. You could integrate your power sensor for a comparison.

8

thanks I will take a look at that over weekend

9

Thanks for the link, I took a look but the recommendation at the bottom of that thread is to use the utility meter integration which is what I’m using.

However the erroneous kWh data is coming directly from my sensor which is why I need to get the outlier filter working, before I pass it to the utility meter integration.

It seems to me that the outlier filter isn’t working as intended unless someone can help me out? I tried reducing the radius as suggested but that made no difference.

10

Which is why I suggested the alternative of integrating your power sensor.

11

A working outlier filter would be much much simpler though

And the inbuilt utility meter does all the integration needed, that’s why it exists

12

Except it’s not working is it?

Open an issue and use the workaround until it’s fixed, which could be some time.

Also there’s another issue with the outlier filter. If you restart during one of the spikes the state will be trapped at this higher level. I had this issue when trying to filter glitchy temperature sensors. The only solution was to restart, and restart, and restart hoping all the sensors would report sensible values - until I got sick of it and replaced the sensors. Another option you should consider.

That is not correct. The utility meter keeps a running maximum until the reset time. Integration with respect to time will transform your power sensor to an energy sensor. That is not what the utility meter does. It needs to be fed an energy sensor to begin with.

13

OK will do thanks