Custom Component: IoTaWatt Energy Monitor integration

jyavenard · August 20, 2021, 6:34am

I’m thinking that we could probably use just the iotawatt and the “integration” platform

So rather than using the blah_wh sensor generated by the IoTaWatt integration, you use the sensor directly and create a new sensor based on the integration platform.
This will estimate the Wh, with probably sufficient accuracy for drawing nice graph.

I’m going to try that, and if I find that I have too much discrepancy I’ll write some code that calculates the energy value from the collected measurement (performed every 6s).

So using @Eoin template above, use the Convert W to kWh part bit.

OzGav · August 20, 2021, 6:36am

Looking forward to hearing how it goes

jyavenard · August 20, 2021, 7:00am

So I created two outputs in the IotaWatt:
Import : Watt = (Phase1 + Phase2 + Phase3 + Tesla1 + Tesla2 + Tesla3 + Pool) + Solar1 + Solar2 + AC1 + AC2 + AC3 + HotWater max 0
and
Export: Watts = Phase1 + Phase2 + Phase3 + Tesla1 + Tesla2 + Tesla3 + Pool + Solar1 + Solar2 + AC1 + AC2 + AC3 + HotWater min 0

This gives negative value for imports, same for solar production sensors; and I want to keep it that way
So in HA I have template sensors that make them always positive:

- platform: template
  sensors:
    energy_solar1_wh:
      friendly_name: "Solar1 Energy"
      unit_of_measurement: Wh
      device_class: energy
      value_template: "{{ states('sensor.iotawatt_input_solar1_wh')|float|abs }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_input_solar1_wh', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_input_solar1_wh', 'state_class') }}"
    energy_solar2_wh:
      friendly_name: "Solar2 Energy"
      unit_of_measurement: Wh
      device_class: energy
      value_template: "{{ states('sensor.iotawatt_input_solar2_wh')|float|abs }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_input_solar2_wh', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_input_solar2_wh', 'state_class') }}"
    energy_solar_total_wh:
      friendly_name: "Solar Total Energy Output"
      unit_of_measurement: Wh
      device_class: energy
      value_template: "{{ states('sensor.iotawatt_output_total_solar_wh')|float|abs }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_total_solar_wh', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_total_solar_wh', 'state_class') }}"
    power_grid_import:
      friendly_name: "Grid Power Import"
      unit_of_measurement: W
      device_class: power
      value_template: "{{ states('sensor.iotawatt_output_import')|float|abs }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_import', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_import', 'state_class') }}"
    power_grid_export:
      friendly_name: "Grid Power Export"
      unit_of_measurement: W
      device_class: power
      value_template: "{{ states('sensor.iotawatt_output_export')|float|abs }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_export', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_export', 'state_class') }}"

and two integrations sensors:

- platform: integration
  source: sensor.power_grid_export
  name: energy_export_wh
  method: right

- platform: integration
  source: sensor.power_grid_import
  name: energy_import_wh
  method: right

And I would configure HA energy with the relevant sensor:
sensor.energy_import_wh for import
sensor.energy_export_wh for export
energy_solar1_wh for my first solar array
energy_solar2_wh for the second.
(haven’t decided if I like have separate solar data rather than the total, will probably change as I’m not a fan of the current graph HA is drawing)

Not sure why @Eoin would want to use the left method over the default trapezoidal.

Edit: you need to use the right method for the integration component as we are adding values that have already been averaged.

OzGav · August 20, 2021, 7:37am

Looking at your top two equations they are exactly the same except for the max and min at the end. Is that right? If so how?

Edit: I have thought about it some more and I get it. I was thrown by the brackets you had in the top equation and thought that something different should be going on.

ogiewon · August 20, 2021, 1:01pm

Is this an issue because you’re using solar? I have a simple system without solar, and I find that Home Assistant matches exactly what my IoTaWatt reports for daily energy usage.

jyavenard · August 20, 2021, 1:26pm

Yes.
Because how you calculate Export vs Import has to be done whenever a read is done by the IoTaWatt.

When you query it via the REST API it will do:
SUM(energy used)-SUM(energy_produced)

consider the following:
you have 2 inputs defined: main and solar.
It’s now August, you’ve consumed 3MWh since the beginning of the year, and your solar panels have generated 1MWh of energy
if you were to query the iota how much energy you’ve imported it will do:
3MHz - 1MHh = 2MWh
and you’ve exported 0,because your consumption is greater than your production under this scenario.

But that is not how export are calculated. You need to measure for each sample (e.g. about 30 times per second for the iota) if your usage is greater than solar and if so you’re exporting.

So say you’re appliances are running flat at 2kW, after a day you’ve used 48kWh.
But during that time, your solar produce energy between 8AM and 5PM, and during that time the power they produced was 3kW.
Between 8AM, you’ve exported (3kW-2kW)*1h = 1kWh
so you’ve exported that day 1kWh * (17:00 - 8:00) = 9kWh

In the example earlier you had produced 1MWh on a single hour, your import would have been 3MWh and your export about 1MWh!

As the author of iota watt explained Wrong values returned by REST API? - explained - #7 by overeasy - Support - IoTaWatt User Community
to calculate it properly, the iotawatt for each queries by HA would need to run thousands of calculations for every samples recorded for that year.
It just can’t, it doesn’t have the storage nor the processing power to do so.

If you don’t have solar, and never ever export, then it doesn’t matter how you calculate it, the sum of the averages is identical to the average of the sums

I hope this post explains properly on what is going on here.

ogiewon · August 20, 2021, 2:06pm

Thank you for the detailed explanation. Makes sense to me now.

OzGav · August 20, 2021, 8:30pm

So if I understand it correctly the integration is retrieving the kWh values. Why doesn’t/can’t it just retrieve the current power value, say every second, and then HA could do the running calculation of energy used/exported?

This might be what your template is doing just want to check. If it is what you are doing why isn’t this accurate (enough)?

jyavenard · August 21, 2021, 2:20am

This is what the above with the “integration” integration is doing.

It’s not particularly accurate from what I’ve seen though, about 10% off. I have an idea on how we could retrieve the data from the iota that would better calculate export/import.

This won’t be using this iota watt integration however, will have to do a REST one manually

OzGav · August 21, 2021, 3:15am

Ok that is quite an error. Do you have a feel for where the error creeps in?
A) Poll rate of Iotawatt? (Is it once a second?)
B) Accuracy of Iotawatt measurement since the last poll (a second ago?)?
C) Calculation of the Integration integration?

I read the accuracy of the Iotawatt is about 1%?

jyavenard · August 21, 2021, 3:36am

every 30s it polls the average of the last 30s

It should be more than enough for accuracy.
When I plot a graph

and over my mouse on the graph, the value you read there is spot-on. This graph was made at 11h so it uses minutes interval.

So the problem is obviously not a polling interval being too long.
I’m told that PVOutput uses 5 minutes interval and get very accurate result (I’ve always used one minute interval myself over the past 10 years and it’s been very accurate, not much different to what I’m billing for.

with a properly calibrated voltage sensor, the accuracy of the iota watt is extremely good, way better than 1% in my experience.

I used the default, I’ve just changed to “left” method will see if that’s any better.

Because right now it’s not very good, and the integration method being wrong is the only thing that makes sense here. All other values are correct.

OzGav · August 21, 2021, 3:52am

Thanks that all makes sense. If we get a reading every minute then why can’t we just take that reading, lets say it is 600W, and divide that by 60 and then by 1000 to get that minutes worth of kWhs (0.01kWh in my example) and then add that to a running total? Wouldn’t that work instead of doing integration?

jyavenard · August 21, 2021, 4:37am

Reading again the Riesmann integration, with the left method this is exactly what it will do
We are adding averages already, so there’s no point to use the default integration method. It would over read.

OzGav · August 21, 2021, 4:56am

good to know. Look forward to hearing your results. I have a few of these I have been playing with to keep track of total power of some lights. I will go and change these to the left method…

jyavenard · August 21, 2021, 12:45pm

On second thought, and trying to remember math from a long time ago, I believe when we are summing the past averages, it’s the right method that should be used.

Look at the test code for the Riesman integration

github.com

dgomes/home-assistant/blob/982807fbc15703f0ccf68305581f2931625abe3b/tests/components/sensor/test_integration.py#L131


      
              }
          }
          
          assert await async_setup_component(hass, 'sensor', config)
          
          entity_id = config['sensor']['source']
          hass.states.async_set(entity_id, 0, {})
          await hass.async_block_till_done()
          
          # Testing a power sensor with non-monotonic intervals and values
          for time, value in [(20, 10), (30, 30), (40, 5), (50, 0)]:
              now = dt_util.utcnow() + timedelta(minutes=time)
              with patch('homeassistant.util.dt.utcnow',
                         return_value=now):
                  hass.states.async_set(entity_id, value, {}, force_update=True)
                  await hass.async_block_till_done()
          
          state = hass.states.get('sensor.integration')
          assert state is not None
          
          assert round(float(state.state), config['sensor']['round']) == 9.17

The values plotted were
(20, 10), (30, 30), (40, 5), (50, 0)]

So 10kW during the first 20 minutes, 30kW the following 10, 5kW the next 10

The calculation as such is 10/(6020)+30/(6010)+5/(60*10)=9.17kWh
Which is exactly what we want when the data we get from the iotawatt is the average power since it was last queried 30s prior

jyavenard · August 22, 2021, 9:31am

Another problem I’ve experienced is that when HA restarts, the iota sensors won’t exist yet or will return a value of 0.

This was problematic with the utility_meter integration to calculate daily values. Because if you restart the device during the day, for a small time the iota sensor will not be available. So if a template attempts to read the value, it will set it at 0. The next time it reads the Wh value, you will now have two the value:

to go around this you use the availability_template attribute, like so;

    power_solar_total:
      friendly_name: "Solar Total Power Output"
      unit_of_measurement: W
      device_class: power
      value_template: "{{ states('sensor.iotawatt_output_total_solar')|float|abs }}"
      availability_template: "{{ states('sensor.iotawatt_output_total_solar') not in ['unavailable', 'unknown', 'none' ] }}"
      attribute_templates:
        last_reset: "{{ state_attr('sensor.iotawatt_output_total_solar', 'last_reset') }}"
        state_class: "{{ state_attr('sensor.iotawatt_output_total_solar', 'state_class') }}"

glazza · August 22, 2021, 9:47am

Thanks - this is all very helpful.

In your earlier example, for solar you are using the wh sensor, but for import/export you are using W… any reason for this?

Cheers

OzGav · August 22, 2021, 12:09pm

Yes that is how I would have calculated it. I will change to right and see how that goes.

jyavenard · August 22, 2021, 11:58pm

solar is accumulative, it doesn’t rely on summing discrete data points.As such, the figure read from the IotaWatt will be correct.

When you create an output that is Export = (Main + Solar) min 0 (where solar can be negative).
And you ask the iotawatt for the Wh of export since January 1st (like the iotawatt integration is doing), how it’s calculated is be doing:
(SUM(Main) + Sum(Solar)) min 0. And the result of that calculations for most people will always be 0. I provided more explanations in the post above Custom Component: IoTaWatt Energy Monitor integration - #94 by jyavenard

I did find another problem with the integration that cause the right integration to cause a wrong result.
I upgraded the HA just after midnight. So the export value at that time was always going to be 0.
After a reboot, the integration doesn’t update the sensor until it reads a value greater than 0 (or different than the previous read).

So here my last read on the meter at 0 was at 12:03AM , and the next update on the sensor was when export just started at 9:00:39 at 188.1W
So using the right method, we have an initial kWh value of 0.1181W * 9h = 1.06kWh when it should have been 0

That would have been compensated with the left method, but then after that it would read too low.
I think it’s a bug in the integration, it should read and update the sensor constantly after a reboot, not when the data change again.

Edit: I lodged Sensor isn't updated until a new different value is read · Issue #15 · gtdiehl/iotawatt_ha · GitHub

jyavenard · August 23, 2021, 2:57am

This makes no sense whatsoever, I have created sensors using all integration methods. The results are wildly different.

The iotawatt Graph+ interface gives me for the daily export: 3.9kWh and 19.3kWh for import.

I think the calculation using the right method got wildly wrong because I rebooted the HA twice today (was adding new sensors) and due to the bug above, it count things multiple times.