On second thought, and trying to remember math from a long time ago, I believe when we are summing the past averages, it’s the right method that should be used.
Look at the test code for the Riesman integration
The values plotted were
(20, 10), (30, 30), (40, 5), (50, 0)]
So 10kW during the first 20 minutes, 30kW the following 10, 5kW the next 10
The calculation as such is 10/(6020)+30/(6010)+5/(60*10)=9.17kWh
Which is exactly what we want when the data we get from the iotawatt is the average power since it was last queried 30s prior
Another problem I’ve experienced is that when HA restarts, the iota sensors won’t exist yet or will return a value of 0.
This was problematic with the utility_meter integration to calculate daily values. Because if you restart the device during the day, for a small time the iota sensor will not be available. So if a template attempts to read the value, it will set it at 0. The next time it reads the Wh value, you will now have two the value:
to go around this you use the availability_template attribute, like so;
power_solar_total:
friendly_name: "Solar Total Power Output"
unit_of_measurement: W
device_class: power
value_template: "{{ states('sensor.iotawatt_output_total_solar')|float|abs }}"
availability_template: "{{ states('sensor.iotawatt_output_total_solar') not in ['unavailable', 'unknown', 'none' ] }}"
attribute_templates:
last_reset: "{{ state_attr('sensor.iotawatt_output_total_solar', 'last_reset') }}"
state_class: "{{ state_attr('sensor.iotawatt_output_total_solar', 'state_class') }}"
solar is accumulative, it doesn’t rely on summing discrete data points.As such, the figure read from the IotaWatt will be correct.
When you create an output that is Export = (Main + Solar) min 0 (where solar can be negative).
And you ask the iotawatt for the Wh of export since January 1st (like the iotawatt integration is doing), how it’s calculated is be doing:
(SUM(Main) + Sum(Solar)) min 0. And the result of that calculations for most people will always be 0. I provided more explanations in the post above Custom Component: IoTaWatt Energy Monitor integration - #94 by jyavenard
I did find another problem with the integration that cause the right integration to cause a wrong result.
I upgraded the HA just after midnight. So the export value at that time was always going to be 0.
After a reboot, the integration doesn’t update the sensor until it reads a value greater than 0 (or different than the previous read).
So here my last read on the meter at 0 was at 12:03AM , and the next update on the sensor was when export just started at 9:00:39 at 188.1W
So using the right method, we have an initial kWh value of 0.1181W * 9h = 1.06kWh when it should have been 0
That would have been compensated with the left method, but then after that it would read too low.
I think it’s a bug in the integration, it should read and update the sensor constantly after a reboot, not when the data change again.
The iotawatt Graph+ interface gives me for the daily export: 3.9kWh and 19.3kWh for import.
I think the calculation using the right method got wildly wrong because I rebooted the HA twice today (was adding new sensors) and due to the bug above, it count things multiple times.
That may be true for some cases, but it’s definitely not what I’m seeing for the template I’m using in combination sur the utility_meter and the integration integration.
It reads a value, followed by 0, then another value
I took a screenshot earlier which I unfortunately can’t find now where you can clearly see the original IotaWatt output sensor going steadily up, with a few gaps in the middle following a restart of HA.
And the template sensor that takes the absolute value of it, see its value go back to 0 wherever there was a gap in the original value.
That totally screws up the utility_meter integration.
For export, it seems so.
Yesterday, I restarted the HA during the day, and on restart the “integration” integration doing W → Wh added another extra 1kWh ; I ended up with the right method being 3kWh more than reality.
This is now, 3 hours after my solar panels started producing energy:
the real value is 1.64kWh
The right method isn’t just inaccurate, it’s totally nonsensical
In fact, I have no explanation on what it is doing, other than there’s a bug somewhere. How could it be so wrong?
The first recorded value is totally wrong to start with:
the reason for that is that the last value the original templates recorded was 0 at midnight, and then 1.67kWh at 9AM ; nothing in between. It took the first reading and assumed that was the average of the last 9 hours.
And it did so again 15 minutes later doubling the value instantly.
This is based on this sensor:
sensor.ower_grid_export which is defined as:
power_grid_export:
friendly_name: "Grid Power Export"
unit_of_measurement: W
device_class: power
value_template: "{{ states('sensor.iotawatt_output_export')|float|abs }}"
availability_template: "{{ states('sensor.iotawatt_output_export') not in ['unavailable', 'unknown', 'none' ] }}"
attribute_templates:
last_reset: "{{ state_attr('sensor.iotawatt_output_export', 'last_reset') }}"
state_class: "{{ state_attr('sensor.iotawatt_output_export', 'state_class') }}"
try:
# integration as the Riemann integral of previous measures.
area = 0
elapsed_time = (
new_state.last_updated - old_state.last_updated
).total_seconds()
if self._method == TRAPEZOIDAL_METHOD:
area = (
(Decimal(new_state.state) + Decimal(old_state.state))
* Decimal(elapsed_time)
/ 2
)
elif self._method == LEFT_METHOD:
area = Decimal(old_state.state) * Decimal(elapsed_time)
elif self._method == RIGHT_METHOD:
area = Decimal(new_state.state) * Decimal(elapsed_time)
integral = area / (self._unit_prefix * self._unit_time)
assert isinstance(integral, Decimal)
except ValueError as err:
_LOGGER.warning("While calculating integration: %s", err)
except DecimalException as err:
_LOGGER.warning(
"Invalid state (%s > %s): %s", old_state.state, new_state.state, err
)
except AssertionError as err:
_LOGGER.error("Could not calculate integral: %s", err)
else:
self._state += integral
self.async_write_ha_state()
So to me it looks like the right method just takes the new state and yes assumes that has been the value since the last update. Left appears to take the state immediately prior to the update and assume that has been the value since the last update. Trapezoidal takes the average of the two states and assumes that has been the value for the whole time.
So it is explainable why the huge jump for you when a reading comes in after nine hours (And I would have expected Trapezoidal to do a similar thing (halved)?). Note sure what happened after that because it should have settled down. Regardless if the IotaWatt integration can keep updating the state regularly then this should be fixed.
On a side note and exciting news. The IoTaWatt integration will be added to Home Assistant in the next release or two. I’m not sure if it’ll make it in 2021.9 or 2021.10.
The PR has had some comments and suggestions from the devs. I have made some good progress in resolving them and trying to fix some smaller issues that are in the current integration.
If any fixes are made they will appear in both the custom integration and the official one until the integration is released in Home Assistant. At that time I’ll deprecate the custom integration.
This creates a 3rd entity for power entities that will construct a high resolution readout of the output and don’t face the issues mentioned in the thread when reading all from January 1st of the year.
I’ve handled cases where if the iota watt will become unresponsive or unreachable for a while, it will retrieve the missing data since the last update.
I’m considering if it’s required to handle the case where the HA instance is down for a while (like what could happen during an update), which is on restart, the HA will rebuild the missing data.
I’m waiting for a few days to see how that work locally. I’m pretty pleased already in that I get much better data for the import/export calculations.
@jyavenard Great I will take a look this week. Need to catch up on some school work, been a bit behind spending too much time with the review comments. I don’t mind though it’s fun and I’m learning!
An update on the release of the integration with Home Assistant. The code was reviewed and has been approved and merged into the latest Home Assistant beta build, 2021.9.0b6.
With the 2021.9 release (if you want to use the bundled integration) you will need to remove the custom integration, reboot HA, and run the setup from the Integrations area.
Just a heads up to anyone checking this post before migrating to the official one, you need to make some custom additions to your config.yaml file for each output you have (not just the mains). Just wanted to say thanks again for all of the work on this, it’s been great so far and i’m happy that it’s gone official!
EDIT: One other thing to note is that you’ll lose all of your historical data. There doesn’t seem to be a way to tell HA that the name of a sensor has change but that it’s the same data source.
