Add options to utility_meter to discard unrealistic values?

Feature Requests
1

Copying this from gh as I posted it there as a “bug” unfortunately :slight_smile:

It would be great if the utility meter had some sort of functionality/options to ignore unrealistic values.

When this is used with my main electric energy meter, my main fuse and all components in the house makes it impossible that I have a consumption of lets say 8000 kWh between state updates.
Assuming most entities reporting energy will do this “live”, but even if values were throttled to once every hour, or even once per day I believe the majority of households would be below 100 kWh/daily consumption even for a house.

With a (example) limit of 100kWh/daily set, a logic could allow around 4kWh delta per hour or 0,07 kWh/minute. Any value above this would be ignored.
This could/should maybe use now() vs last_changed and “new state” vs “previous state” to validate before adding to the utility_meter count.

It would be great if this functionality could be in the utility_meter options to avoid needing to feed the source sensor through various other intermediate sensors to “verify” the data:
“source_sensor → statistic_sensor → filter_sensor → template_sensor → utility_meter”

2

Wouldn’t it be better to fix the sensor used as input? Most likely it is a situation where unavailable is treated as a 0 or something like that. If you fix it in the utility meter, the original sensor is still putting out garbage from time to time.

Oh, and maybe you should vote for your own request :wink:

3

Well I agree. But the issue is that these things occur if HA restart, or zwaveJS restarts.
Occationally a zwave plug might get unplugged when “someone” need that outlet for something else etc.

As long as HA is running and my zwave entity is online theres no issues…or well, I have one Heinmann plug that occationally will output a spike I’ve noticed.

Hah, thanks for the tip :wink:

4

There really is no easy solution ro that problem.

There are two options.

  1. Let the device calculate the values
  2. Let HA calculate the values

If you go with option one then the device will measure the consumption at regular intervals and calculate a value for consumption per time slice.
The issue here is to get HA ti read these values correct.
It needs to read each value before it is changed again by the device.
If it too early it will read the same value twice, which can be countered with a timestamp, butbifbit is too late then a value will be missed and a timestamp will only tell HA that it missed a value and not what it was. HA can then choose. Value of 0, a guesstimate value, a mean value or repeat the last value. None of these are probably correct.
HA, in other words, needs to be in sync with the device to read it correctly, but the lower the interval of the calculations get on the device the more precise this sync needs to be.
If HA is restarted or just hangs, freeze or otherwise gets delayed then you will have your dreaded spikes.

If you choose option 2 then the calculations occur in HA.
The device just have a counter for consumption.
When HA reads the counter it might be 128, and when it reads it 7 seconds later it might be 136.
HA knows both timestamps for the reads and can then calculate the difference which is the consumption.
If HA get a freeze or delay, then the read will just occur at maybe 13 seconds later, but the calculations are the same, so no sync is needed.
If HA is restarted the. It might write the value down to it’s disc and read it again after boot up and everything just works.
But there are issues with this solution too.
One issue is if overflow is not handled correctly.
Let’s say your counter is a 8-bit counter, which means it can count to a max of 255. Once it hits 255 and adds a number more it will restart to count from 0, so HA will see a change from like 253:to 7 in a few seconds, which will give a huge negative consumption if not handled correctly.
Likewise if you unplud the device and the counter was at 136, then when it comes back up it might start at 0 and it will again calculate a huge negative consumption.
If HA is restarted it might save the value, which can counter this to some extend, unless the time period is so long that the counter could overflow twice. If that is the case, then value is completely unreliable.

Option 2 is the most used one, but for integration/add-on developers there is the issue of not always knowing what kind of counter you are dealing with.
Is it 8, 16 or 32-bit and is it signed or unsigned.
For example a signed 8-bit counter will go from -128 to +127, while an unsigned will go from 0 to 255.
And these numbers are just the hardware limitations.
The device manufacturers can choose a smaller range if the want, so instead of 0 to 255 it will maybe be 0 to 100.

If the developer choose a too small number type in HA then higher values will cause an error.
And if it is bigger then the developer needs to find the right limit for overflow and handle it.
An 8-bit counter for like watthours are somewhat easy to test for, since after 255 or less watthours it will reset, but a 16-bit value might first reset after 65536 watthours and the number for 32-bit is insane.

I hope this give some insight in the issues.

5

Thank you for the thorough explanation of the scenarios there.
In my (non developer mind) it is a bit simpler :slight_smile:

Last value in HA utility_meter: 23556 (kWh) - a maximum increase of 5 kWh is allowed, either from the last stored value in HAs utility_meter or last received vs new value from device.
New value from source sensor: 23558 (kWh) → OK PASS → 23558 stored as new total in HA utility meter
New value from source sensor: 0 → Value is lower than last stored → Ignore
New value from source sensor: 5000 → Value is lower than last stored and delta from last state received is too high → Ignore
New value from source sensor: 5002 → Value is lower than last stored, but within acceptable delta from last value → Add 2 kWh to utility meter → HA Stores 23560 as new value

This scenario happened to me as I replaced/upgraded the zwave plug my boiler was connected to, the old plug had been running and accumulated like 2000 kWh while it was in use, while the new AeotecHeavy Duty relay was new and its counter started at 0.

If the utility_meter was set to “net_metering=true” this would have reset the yearly utility_meter for the boiler to 0 and started from there and I’d need to do a manual calibration, which is OK when these things happen in a controlled manner.

If it was set as “net_metering=false” I assume the scenario described above would happen - it would ignore the first decreasing value, but continue counting up from the next value (not sure about this)
This setting however will cause values coming in as 0 for whatever reason and then coming back with the “correct” value of…lets say 600 cause the decreasing value to be ignored, but it will then add 600 when the correct counter is read from the device.

When it comes to the device counting or HA counting that is not really relevant unless you are calculating energy from power+time with the Riemann sum integral component. But if the device has its own energy metering I don’t see any point in doing that.

Since devices usually will count indefinitely and not reset by any interval this is the point of the HA Utility meter to show the difference of the first and last value of the source sensor in the timeframe selected (daily/weekly/monthly)

6

It is up to you to sanitise your data. There are far too many ways the data coming in to the utility meter could go wrong to catch them all, and it is not a filter, it is a utility meter.

Here’s how I do it:

7

Thanks, I will have to have a look at your tips and figure out how I can get around this. It could be that I haven’t fully understood how I could set up the filter or in what order they should be stacked. Maybe I’ll need a automation running in the background updating a input_number and/or a average sensor running as a baseline to off which a maximum change is allowed.
From how I see the situation the data is just fine, with the exception of that one device i have some glitches on. The z-wave energy meter-entities have been working very well and consistent, It just came off to me as if its rather how HA handles a restart or reloading entities and that such a “validating” feature would be a valid addition to a utility meter.

It hasn’t really bothered me a lot, but since I’m now moving my setup to a new installation I’m also taking some time to clean up my configuration and reduce the number of template sensors etc that I have accumulated through the years.
It would also be fun if the graphs do look nice.

8

You will still have to figure out what to do with the ignored value.
That is where your spike is.

9

The ignored values are just kept to compare with the following value.

The HA utility meter is already ignoring all values that would cause a decreasing total if net_consumption=false.

What I want is to also make it ignore any value that would cause a increase or decrease of more than a set limit set for “max_delta”
Consecutive identical values will of course not do anything to the HA utility meter as its indicating no consumption (delta=0)

So the source meter from the device might restart to 0, for whatever reason, a glitch while restarting HA, device is unplugged temporarily or the device is replaced with a new, or with a different plug.
HAs utility meter had counted to 2678 kWh.
Consecutive 0 values would be ignored until it started showing increments/decrements that are “valid”, lets say the source meter moved from 0 to 50 kWh which would be ignored. Next value could be 50.2, which would indicate a correct consumption. thereafter the HA utility meter would continue adding the following delta values to its utility_meter.

10

I get that you can get back quickly, but you still have not answered what value should be set for the interval with the discarded value.
It’s that value that is the entire problem which cause the spikes.

11

I’m not sure I’m following.
If there is a interval that is too large it will be skipped… so it should not do anything.
If the HA utility meter had counted to 2678, it will not change until the source sensor provides two updates with an acceptable delta.

12

If you have a ok value for Monday, Tuesday, but Wednesday the value fails, then you can get an ok value for Thursday again, but when you want to draw you graph of the week, then the line will be broken.

13

Well no, the utility_meter will be sitting at the value it had last accumulated from the source sensor.

The main meter (source sensor) could be at 30000, but the HA utility_meter will start at 0 according to the “cycle” option set for it. We agree on that?

So on Monday the source sensor is at 30000, while HA utility_meter is reset to 0.
As the source sensor continues from 30000 and increases (unless you are feeding back to the grid)

If the weekly (or whatever period) utility_meter had counted up to 50 kWh consumption by Wednesday (source sensor at 30050), in what we could expect to be at least daily updates if its a really lazy source sensor (if it has consumption), for simplicity we can assume its the main meter, so it will have consumption every day and probably every hour of the day.

If it fails on Thursday, by trying to output a increase of 2000 (source sensor jump from 30050 to 32050) in one interval it would(should) fail.
But if it receives lets say 2010 on Friday that would mean a 10 kWh increase (from 2000), it would then increase the weekly utility_meter (which was stuck since Wednesday at 50) with an addition of 10 to show a value of 60 (50+10)

I do assume a few things in this reasoning though, that the source_sensor connected to Home Assistant is actually connected and “live” and that its not removed from the house and taken “somewhere” its not communicating with HA for a long period (accumulating) and then brought back to the house and within range.
A different factor is that most zwave/smartplugs are rated 16A/3600W @ 230V which means its not possible to accumulate more than about 88 kWh within a day, 2650 in a month or 31800 kWh within a year.

Also, I’m suggesting a option.

I tested a workaround which seems to be doing the trick

alias: Update Electronics energy meter
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.total_electronics_energy_throttled_5m
condition:
  - condition: template
    value_template: >-
      {{ trigger.to_state.state|float(0) > trigger.from_state.state|float(0) }}"
  - condition: template
    value_template: >-
      {{ (trigger.to_state.state|float(0)-trigger.from_state.state|float(0))<10  }}
action:
  - service: input_number.set_value
    data_template:
      entity_id: input_number.total_electronics_energy
      value: >-
        {{ (states('input_number.total_electronics_energy')|float(0) +
        (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(4) }}
mode: single
14

Can the filter sensor listed under home-assistant utility integrations, be used on the source sensor, and then use this new value as input to the utility meter?

15

I haven’t been able to set a filter that works really.
I was thinking the outlier function, but that will eventually fail when the the window_size is exceeded.
The range filter will not work as it uses a set min/max value and replaces any value outside this range with the max/min instead of discarding it. Since the source sensor will be accumulating this will have to be updated regularly, but I don’t want the sensor to update to the max or minimum if a invalid value is received.

I will see how this automation will work going forward, it should handle all my aggregated energy meters, which then is used as a souce sensor for the utility_meters for daily/weekly/monthly/yearly periods.

alias: Update per category energy meters
description: ""
trigger:
  - platform: state
    entity_id:
      - sensor.emizb132_energy
    id: Main Meter Energy
  - platform: state
    entity_id:
      - sensor.total_electronics_energy_throttled_5m
    id: Electronics Energy
  - platform: state
    entity_id:
      - sensor.total_heating_energy_throttled_5m
    id: Heating Energy
  - platform: state
    entity_id:
      - sensor.boiler_energy
    id: Boiler Energy
  - platform: state
    entity_id:
      - sensor.total_basics_energy_throttled_5m
    id: Basics Energy
  - platform: state
    entity_id:
      - sensor.total_utilities_energy_throttled_5m
    id: Utilities Energy
condition:
  - condition: template
    value_template: "{{ trigger.to_state.state|float(0) > trigger.from_state.state|float(0) }}"
  - condition: template
    value_template: >-
      {{ (trigger.to_state.state|float(0)-trigger.from_state.state|float(0))<10
      }}
action:
  - choose:
      - conditions:
          - condition: trigger
            id: Main Meter Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.main_meter_energy
              value: >-
                {{ (states('input_number.main_meter_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
      - conditions:
          - condition: trigger
            id: Electronics Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.total_electronics_energy
              value: >-
                {{ (states('input_number.total_electronics_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
      - conditions:
          - condition: trigger
            id: Boiler Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.boiler_energy
              value: >-
                {{ (states('input_number.boiler_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
      - conditions:
          - condition: trigger
            id: Heating Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.total_heating_energy
              value: >-
                {{ (states('input_number.total_heating_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
      - conditions:
          - condition: trigger
            id: Basics Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.total_basics_energy
              value: >-
                {{ (states('input_number.total_basics_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
      - conditions:
          - condition: trigger
            id: Utilities Energy
        sequence:
          - service: input_number.set_value
            data_template:
              entity_id: input_number.total_utilities_energy
              value: >-
                {{ (states('input_number.total_utilities_energy')|float(0) +
                (trigger.to_state.state|float(0)-trigger.from_state.state|float(0)))|round(2)
                }}
    default: []
mode: parallel
max: 6
16

An alternativ can be PowerCalc: Home Assistant Virtual Power Sensors
Custom component to calculate estimated power consumption of lights and other appliances.

I am looking into using this component to create virtual power sensors for balancing my power load per hour (I want e.g maximum 5kwh) pr hour.

My base load is about < 650W.
I want to automate the rest of my flexible loads e.g temperture dependint thermostats/hydrostats and sceduled loads like washing machine/dish washer, battery charging.

to be distribuated based on the cheapest hour, to the most expensive hour during a 24 hour window.

The price groups comes from Nordpol :slight_smile: