Measure unmeasured energy consumption

[fversteegen (Frank)]

With the current energy prices in The Netherlands I wanted to get better grip on where my electricity was actually going to. In order to understand the consumption of various appliances I use 14 power plugs (3 times a Gosund SP-1 and 9 times a Gosund SP111 flashed with Esphome). I have a smart meter which I read via a Homewizard P1 plug.

As I also have quite a lot of Philips Hue bulbs I decided to also measure those with the help of the great plugin Powercalc which gives me an additional 10 measurements of consumption.

As I have solar panels I first have to create a sensor that shows the actual consumption of the house. The logic behind it is home consumption =(imported energy + solar production - exported energy). I do this via a template sensor:

  - platform: template
    sensors:
      calculated_energy_consumption_house:
        friendly_name: "Total calculated energy Consumption house"
        unit_of_measurement: 'kWh'
        value_template: "{{ states('sensor.p1_meter_3c39e72bf1a4_total_power_import_t1')|float + states('sensor.p1_meter_3c39e72bf1a4_total_power_import_t2')|float + (states('sensor.solaredge_lifetime_energy')|float /1000) - states('sensor.p1_meter_3c39e72bf1a4_total_power_export_t1')|float - states('sensor.p1_meter_3c39e72bf1a4_total_power_export_t2')|float }}"

The above value gets fed into a utility meter sensor which calculates the daily/mothly/yearly value:

utility_meter:
  daily_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: daily
  monthly_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: monthly
  yearly_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: yearly

Now onto the powerplugs: each powerplug produces the power value (ie: how much Watts is being consumed). This value gets fed into a Riemann Sum sensor (to calculate the energy consumption) which then gets fed into a utility meter sensor. Find below the example for 1 power plug.

sensor:
  - platform: integration
    unique_id: 5961ce29-02bd-4bfa-b038-b0285546cbf1
    source: sensor.eettafel_stopcontact_power
    round: 2
    method: left
utility_meter:
  utility_meter_eettafel_stopcontact_daily:
    source: sensor.sensor_eettafel_stopcontact_power_integral
    cycle: daily
  utility_meter_eettafel_stopcontact_monthly:
    source: sensor.sensor_eettafel_stopcontact_power_integral
    cycle: monthly
  utility_meter_eettafel_stopcontact_yearly:
    source: sensor.sensor_eettafel_stopcontact_power_integral
    cycle: yearly

Now, as a bonus, it is time to add the Riemann Sum or Integration sensors to Energy Dahboard (NOT the utility meters!). If that’s done correctly it will produce the below:

As a next step I wanted to know what part of my total consumption I was actually measuring. In order to do so I created a template sensor which sums up the power consumption (ie: Watt) values of all the individual sensors, feed that into a Reimann Sum integration sensor and feed that into a utility sensor. In order to keep things tidy I will only show 2 sensors.

sensor:
  - platform: template
    sensors:
      measured_power:
        friendly_name: "Measured power consumption"
        unit_of_measurement: 'W'
        value_template: "{{ states('sensor.3d_printer_power')|float + states('sensor.cvketel_power')|float }}"
  - platform: integration
    source: sensor.measured_power
    name: Measured power energy
    unit_prefix: k
    round: 2
    method: trapezoidal
utility_meter:
  daily_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: daily
  monthly_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: monthly
  yearly_calculated_energy_consumption_house:
    source: sensor.calculated_energy_consumption_house
    cycle: yearly

Now I can create cool graphs like the below:

As I have so many power plugs it should however show me almost all power consumption, but by judging from the gap between the above lines I measured only about 75% of my power consumption. In order to track this a bit better I created a template sensor which shows my unmeasured power consumption:

sensor:
  - platform: template
    sensors:
      unmeasured_daily_energy_consumption_house:
        friendly_name: "Total unmeasured daily energy Consumption house"
        unit_of_measurement: 'kWh'
        value_template: "{{ (states('sensor.daily_calculated_energy_consumption_house')|float - states('sensor.utility_meter_measured_power_daily')|float) | round(2) }}"

Now I can track both the power consumption of the entire house AND the unmeasured energy consumption in real time:

Now the morale of the whole story: I think I need to calibrate my power plug better. The line between “Daily calculated” and “Total unmeasured daily energy Consumption house” looks so very similar that I suspect that the delta is caused by my power plugs being a bit off. Will post an update as soon as I have managed to go through the tedious process of calibrating all of them. How? That has been clearly descibed by @frenck at: Calibrating an ESPHome flashed power plug » Frenck.dev

Hope this write up helps others in hunting down their energy consumption.

[md4711]

Hello Frank,
Many thanks for the very good and detailed description.
With this I was able to take a big step forward in my area.
In the last step you use the sensor “sensor.utility_meter_measured_power_daily”.
Could you post the definition of it ?

[fversteegen (Frank)]

That sensor is simply the utility sensor of the measured daily energy. All yaml is in the description. For clarity:

Template sensor “Measured power consumption” → Integration sensor “Measured power energy” → Utility sensor “daily_measured_energy_consumption_house”

[md4711]

Hello Frank,
in the meantime I have adapted the configuration but I get a negative value:

my config:
…

• sensor:
  • name: “unmeasured_daily_energy_consumption_house”
    unit_of_measurement: ‘kWh’
    state: >
    {{ (states(‘sensor.daily_calculated_energy_consumption_house’)|float(default=0.0)
    - states(‘sensor.utility_meter_measured_power_daily’)|float(default=0.0)) | round(2)
    }}

daily_calculated_energy_consumption_house:
source: sensor.calculated_energy_consumption_house
cycle: daily

utility_meter_measured_power_daily:
unique_id: uid_utility_meter_measured_power_daily
source: sensor.measured_power
cycle: daily
…

[fversteegen (Frank)]

I discovered more or less the same issue. I am therefore now using two daily utility sensors to calculate the not-measured power. Am now testing it since 2 days and appears to be working as expected. Am still wondering how there is unmeasured consumption in my house…

[fversteegen (Frank)]

I keep on getting strange jumps in the unmeasured power consumption. As an example yesterday in which I worked from home and had nothing special running with the exception of running the washing machine 1 time (at around 14:15). Now look at the energy consumption graph of the washing machine (tend to say it appears accurate), but compare that to the graph of the unmeasured energy: you see the same steepness of the curve. This leads me to the conclusion that either the power plug of the washing machine reports incorrectly (which I doubt as I calibrated it) OR that something else is wrong.

Any tips on where I should look?

[wattmatters]

@fversteegen

Any tips on where I should look?

Do you have any dedicated circuits, e.g. air-conditioning, water heater, pumps?

[fversteegen (Frank)]

Nope: plain old central heating (which is off) and no mechanical ventilation. Aircondition is unplugged.

Will run a cross test today on the washing machine and post results.

[fversteegen (Frank)]

Argh! Ran a cross test with the same results. Now I need to check all calculations (won’t be much fun…)

[WillS]

Are you calibrating with resistive or inductive loads for plugs to be used with motorized appliances?

Might some of the CTs or circuits be noisy, like found in this comparison?

[fversteegen (Frank)]

Calibrated the sensors with resistive loads (a incandescent lightbulb, a soldering iron, an halogen high power light) I still have the feeling that my readings might be due to inconsistency despite the readings being quite consistent across the days. Find below a graph of about a week (forget about the graph on December 24th as I had some problems that day)

[fversteegen (Frank)]

Resurrecting my old thread: I think I finally managed to get a coherent overview of all energy consumption thanks to adding a lot of virtual sensor through Powercalc for lights and a few loads which are constant and “known” (like 12 watt for a wifi AP).

When not home I know come to about 89% of energy being measured, but when I am home it is close to 95% which is sufficient for me. See below a screenshot of the percentage of consumption which is measured. Now what remains is to continue to measure and drive the consumption down

[rkonings (Ronald)]

I found this thread as I have some ghost consumption in my house.
I need to look into your code as this looks to solve my problem.

I was wondering was this unmeasured consumption is sometimes completely off.
Just to check with this forum:
My Enphase solar panels(the main inverter unit, the Envoy), are not updating frequently, sometimes once a minuut en then 5 minutes there are no updates.

same for all my power plugs, they also seem to have this sort of behaviour.
Same with the P1 meter, this also lose connection sometimes

Looks like this might cause so weird and unreliable behaviour.
For daily and monthly it looks fine though.

any thoughts?

[fversteegen (Frank)]

The timing doesn’t need to be exact. If you look at my graph of yesterday you can see that my solar panels sometimes skew the graph a little bit due to difference in timing. The same is true at midnight when the sensor resets itself. In the end you should look at the approximate value and not the hard number.