I have a couple of z-wave relays that provide energy consumption data. But a bug in the firmware causes the devices to send a single incredible high consumption value every few days. This completely screws up the energy statistics. So I want to create filtered copies of the consumption entities, where e.g. values higher than 10% of the previous value is filtered away. I don’t want to set a fixed range, since sooner or later the consumption value will hit that upper limit, or else I have to set the limit so high that an error value might be received that’s below the limit (I’ve seen error values of both a few hundred kWh’s and a few million kWh’s).
You could make a template sensor that calculates the delta between the old and the new state of the buggy sensor and then decide what the template sensor’s value should be.
If the anomalous values are as you implied, incredibly higher than the normal value range you should be able to filter them with the outlier filter while leaving the rest of your data intact.
This is exactly the purpose of this sensor. You don’t need to reinvent the wheel.
Why not give it a shot and see if it works before dismissing it? It will only take you a few minutes to configure.
Then set up a graph with the original sensor, and the filter sensor and compare the data for a week or two. If it gives the results you want then you’re golden. If not, tinker with it till it does.
It looks to me like OP’s sensor is cumulative and not a time series, so I’m not sure how that would work then. Unless they have a power sensor on which to filter and then integrate it again to get a new cumulative value.
Aw shucks. I’m on mobile right now and I thought you were OP. I also thought we were talking about a power use sensor. I didn’t go back and read the original message again.
All of my zwave devices have current power use (W) sensors and I use those with integration sensors for my energy data, guess my head is stuck there.
If OP’s offending devices provide current power sensors he could probably do the same in order to utilize the filter sensor as a solution to his issue.
Anyway, it’s late and I’m tired so I’m checking out before I embarrass myself further
Yes, basically. There will be trade-offs whether you do this, set it to zero or something else. If the interval between updates are fairly short, then the impact should be small. Either way, this should cause less havoc than the main issue.
I’m actually quite curious about the issue. Do you have a link to it if it’s logged somewhere on a z-wave site? It could be that they’re getting some internal integer overflow that’s not handled correctly. If so, there could be better ways to deal with it.
As an example, I implemented this fix for my SNMP bandwidth monitor which wrap arounds quite frequently.
{# safe delta catering for wrap-around of a 32-bit unsigned int (snmp counter is 32-bit unsigned int) #}
{# basically taking 2's complement #}
{% set from = trigger.from_state.state | int %}
{% set to = trigger.to_state.state | int %}
{% set traffic_delta = (to - from) if (to >= from) else (4294967295 - from + to + 1) %}
{% set time_delta = as_timestamp(trigger.to_state.last_updated) - as_timestamp(trigger.from_state.last_updated) %}
{{ (traffic_delta * 8) / time_delta }}
Thank you for your reply. I’ve been trying for a few evenings now to make this work, but I can’t seem to get the hang of templates yet.
Is the template you’re showing part of an automation? I guess it is as you’re using a trigger object, and I can’t find references to a trigger object in value templates in custom entities added in yaml. If so, do you just trigger on all state changes on the entity with transient errors? And furthermore, how do you set the corrected value to a filtered entity?
zwave_38_electric_consumption_kwh_filtered:
friendly_name: 'Water heater: filtrered consumption [kWh]'
value_template: >-
{% set value = states('sensor.zwave_38_electric_consumption_kwh') %}
{{ value }}
device_class: energy
unit_of_measurement: 'kWh'
Bah, this wasn’t as easy as I’d hoped. I’ve been trying this on and off for some time now. I’m a bit pressed for time, so haven’t had much time to look at it.
Anywho, I was looking into using an input_number helper and an automation as you suggested. But I saw eventually that the Energy module can’t use input_number type entities for measuring energy consumption. It has to be a sensor type entity.
Is there a way to create a template entity which is either set by an automation, or have it check and discard error values as part of its value_template?
