Odd behaviour of riemann integral over templates, different values for direct/indirect sensors

[Ben-Bitdiddle-DE (Ben Bitdiddle)]

Is it possible to attach files here?

Direct: https://dpaste.org/zfTL1
Wrapped: https://dpaste.org/046jf

Both until this afternoon somewhere, already after the diff taken place.

The sensor is read all 10 seconds, so the derived wrapper is. So there are no long(er) times between measurements, there all about being equally distributed.

I have been curious about Edwins remark, as a saw that the CSV began at Day 23 22h, must have to do with UTC mapping (-2 hours local time).

(i am adding here as i came over my daily posting number limit :-))

history_integraldirect.csv: https://dpaste.org/046jf
history_integralwrapper.csv: https://dpaste.org/nQvJE

Both the integral value files are not 100% identical, i assume because i did not manage to create them completly parallel (i made two windows to create the helpers, but there is some time in between).

So from the graphics i took 8am as relevant time (which is 6am in the logs).

I can see, that the integrals are updated all 10 seconds (might correlate to the 10 seconds the sensors are read out). The timing values for both integral are synchronized for all entries.

I jumped to somewhere around 6am. Until there both integral values are somehow nearby with each other.

Integral data:

After 4:35a there are no new values until 6:03am? (in both integrals)
For hour 6, there are only 2 value updates.
After 7:03am it seems that everything goes on within that 10 seconds interval.

At 6:03 there is this spike, the wrapped integral jumps up, the direct integral does not. Both sensors provide the same value into their respective integral.

Regarding both integral before that time (also after), one can see that with each sensor data update both integrals sum up about the same value (which makes sense, as both sensor data come in at the same timestamp with the same value)

Sensor data:

Looking at the sensor data, these times correlate somehow. The sensors (both) are not read after 4:35am, only two times in hour 6 and again after 7:03am.

Idle time

I cannot say why there is this gap between 4:35am and 7:03am. I check with my working/primary installation and can report this gap also as existing there. Dunno why. Maybe the fronius inverter is doing something more important during this time.

Configuration:

sensor.solarnet_entladeleistung: state_class: measurement
sensor.wrapper_hausspeicher_entladeleistung_aktuell: state_class: measurement

[petro]

Maybe? I’m not 100% sure and I probably should know. Try it. If it doesn’t work, you can use dpaste.org.

[petro]

Do you have the data when the diff occurred? Also, can you output the integral data as well. Lastly, just to check things out, both source sensors are state_class measurement right?

[mekaneck (Rick)]

I’m willing to bet the issue is this:

The Riemann sum integral helper will calculate a new value upon every state_reported event.

The template sensor helper will only report a new value on a state_changed event.

When the source sensor is the same value for a long period of time, and keeps reporting that same value (in your case, zero), the integral helper will update its value on each report.

When the source sensor of the integral is a template sensor, it doesn’t re-report the same value, so doesn’t fire a state_reported event nor does it fire a state_changed event, so nothing happens and the integral doesn’t update.

When you use trapezoidal, the integral will calculate the area from the last update. If your source sensor is reporting zeros every minute, the area of the trapezoid is only since one minute ago. If your source sensor is a template sensor, the area of the trapezoid will be from the first zero, not the one from 1 minute ago.

TL;DR: the state_reported event from the source sensor is critical when using trapezoidal or right integration methods. The template sensor “wrapper” doesn’t pass these events through.

[petro]

That’s possible, template entities do not support last_reported yet…

[Ben-Bitdiddle-DE (Ben Bitdiddle)]

@petro

Do you have the data when the diff occurred? Also, can you output the integral data as well. Lastly, just to check things out, both source sensors are state_class measurement right?

I added my checks and data above (Odd behaviour of riemann integral over templates, different values for direct/indirect sensors - #21 by Ben-Bitdiddle-DE).

Everything which had been said about trapezodial, states reported/changed (or not) makes sense to me. Only from the data exported I do not get it together, why both integrals sum up differently. There might be other things going on under the hood, which i don’t see or know. I’ll leave it that way here.

As it can be seen, both sensors concurrently und regularly push (non zero) data (at the same time with the same value) and both integrals get updated concurrent (timestamps match) by that. Only that the wrapped integral has these peak spots and has a resulting higher value. Anyway…

I let it run with sum.left from yesterday, as expected the rieman left did the trick (there are new jumps today for the trapez version)

Red/Cyan: Trapezoid integral (continued from yesterday)
Orange/Blue: Left sum integral (blue hidden by orange as they are equal).

[Ben-Bitdiddle-DE (Ben Bitdiddle)]

Just asking:
For the trapez integral: Would setting max-sub-interval (10 seconds maybe as this is the normal sensor update interval) also do the trick?

I will want to change my integral sensors to left now, is it safe just to change them in the configuration (or do i have to delete and create them new). By changing to left, is it so that the existing integral values stay as current (as history) and only from now on just the calculation for new values will be changed/adopted?

[mekaneck (Rick)]

@Ben-Bitdiddle-DE

Only from the data exported I do not get it together, why both integrals sum up differently.

The data from the source sensors will be the same. There’s not enough information there to re-create how the integral is calculated by HA.

If the source sensor reports a value that is identical to the previous value, that value will not be saved in the history database and will therefore not appear in the exported data. But that data point does impact the calculation of the integral.

Looking at the historical data of your source sensor data it is not possible to determine how many updates occurred during the period when the sensor was zero. Even if a zero was reported every second, when you export the data you’ll see a single zero at the start of the period and then nothing until the first non-zero number was reported. However, those zeros that aren’t in the data will affect the right and trapezoidal integral calculations.

[mekaneck (Rick)]

@Ben-Bitdiddle-DE

For the trapez integral: Would setting max-sub-interval (10 seconds maybe as this is the normal sensor update interval) also do the trick?

Yes this will make the integral based off the template sensor match much more closely to the “original” integral.

@Ben-Bitdiddle-DE

By changing to left, is it so that the existing integral values stay as current (as history) and only from now on just the calculation for new values will be changed/adopted?

Yes, changing the configuration of an entity will never impact historical data.

[Ben-Bitdiddle-DE (Ben Bitdiddle)]

@mekaneck

@Ben-Bitdiddle-DE

For the trapez integral: Would setting max-sub-interval (10 seconds maybe as this is the normal sensor update interval) also do the trick?

Yes this will make the integral based off the template sensor match much more closely to the “original” integral.

Is then there a rule of thumb, when to use left and when trapezoid(with max-sub-interval)?

[brooksben11 (Ben)]

With the way HA stores data, I can’t really come up with a scenario where anything but left makes the most sense.

The reason trapezoidal is the default, is that in most any other application, it’s actually preferred.

[mekaneck (Rick)]

If we didn’t have the repeated values issue with HA, it would be:

• Left: Source sensor sends an update when the value changes. Most binary sensors fall in this category, but we typically don’t use them as a source for integrals. Some numeric sensors do fall in this category and push new values when a large-enough change is detected.
• Trapezoid: Source sensor sends an update irrespective of the value changing. For example, the update is sent at a fixed time interval. Most numeric sensors fall in this category.
• Right: Source sensor sends the previous value when the value changes. This rarely ever applies, so it’s safe to just never use this method.

Now, with the repeated values problem in HA, it adds a little more complexity.

The left method is immune to the issue, so it is the approach most often recommended.

If your source sensor falls into the category where trapezoid applies, then you have to consider if it could report an unchanging value for a long period of time. If that can happen, then you next have to determine if the sensor entity’s last_reported property will be updated at a frequent interval during the time where the value is constant.

• Sensor will never have long periods of a constant value: trapezoidal with no max_sub_interval
• Sensor can have long periods of a constant value, but the last_reported property will get updated frequently during that time: trapezoidal with no max_sub_interval
• Sensor can have long periods of a constant value, and the last_reported property will not get updated during that time: trapezoidal with a max_sub_interval set to slightly longer than the typical reporting period of the sensor

Sorry, not really any good simple rules of thumb here. I guess the rule of thumb is: if you don’t know, use left