Negative Values in Energy Dashboard since last update

[HerrRausB]

Thanks a lot - that saved my day (so far)

[e-raser]

To summarize two questions:

1. This was introduced with HA Core 2023.5 and only affected Riemann Sum integration. Right?

2. Which HA Core release fixed this issue? I went through all posts but could not find a PR for the fix.

---

I think I might have run into the same issue once updating to HA Core 2023.4.6 back in November. Since then I see this regularly and only discovered today…

Energy dashboard screwed due to decreasing input sensor - how to a) fix and b) prevent in future?

I took a look in the history, using statistics card set to 365 days with hour data. And there are more incidents of that type, as already shown with: https://community-assets.home-assistant.io/optimized/4X/6/1/2/61248e82709c11a64f6201fe3bbcc82375807d9d_2_1380x286.png But today is not the first incident/time, it had an actual impact on the LTS and energy dashboard data. Here filtered for 180 days to have a better scale: [grafik] The first incident was on 2023-11-13. There were much more since…

Edit:
I made some more research in my InfluxDB and I can confirm updating HA Core to 2023.4.6 definitely introduced this issue (I updated on 2023-11-04). My utility meters used are unavailable from time to time, causing these issues to sometimes show up in the energy dashboard.

Question 2 please! Which HA Core release fixed this issue?

Thank you.

[Edwin_D]

Please note that this is often caused by sensors that return 0 when the sensor should have reported unavailable, for instance during a reboot. This can be due to a flaw by the integration builder, but often it is caused by template sensors added by us users that are missing an availability template. My problems were fixed when I added the availability template to my misbehaving water sensor.

See also this discussion:

Energy dashboard screwed due to decreasing input sensor - how to a) fix and b) prevent in future?

The fact that you didn’t have this problem without availability template is luck. Also the behavior change could be from the integration that provides the three subsensors. I must admit I too had more problems like these from my water sensor around the same time. They also lacked an availability template, so indeed something changed in HA for the issue to become more apparent. I suspect the statistics start working earlier during the boot process, when not all integrations are done loading. B…

[e-raser]

Likely a good thing to work with availability_template, which is what I did yesterday.

---

raw input sensor → utility meter sensor → template sensor

---

BUT as the input sensor for my affected template sensors are from a utility meter…

• that integration should know about how to handle unavailable its own input sensors the right way
• still something has changed in HA!

Before (in my case) HA Core 2023.4.x there was no need to suppress unavailable states, this was handled bullet-broof by HA internally. The fact so many (at least those who care about their data or accidentally stumble over broken energy dashboards) suffer from this proves this is not an individual issue.

Therefore while monitoring if there are more incidents after I added the availability_templates I urgently want to know if and which HA Core version fixed this definitely with an update introduced behavior.

[Edwin_D]

@e-raser

Before (in my case) HA Core 2023.4.x there was no need to suppress unavailable states, this was handled bullet-broof by HA internally.

You’re still not getting it. You yourself are putting | float(0) in your template, saying to use 0 if the sensor is unavailable - instead of making the summation sensor itself unavailable.

In doing you created a sensor that goes up and down instead of just up. And because your summation sensor is never unavailable, even if you miss a value, is exactly why HA cannot do a single thing about it and will just do what the state_class says it should do when the value decreases. There’s no bullet proofing for that.

The only thing that happened since a few months is that your sensors are more often seen unavailable, most likely during a reboot. During that time it is quite normal that sensors are not available. So it is not a bug, it is expected behaviour happening more often than it used to. Even if this is reverted, you will still encounter the problem when sensors become unavailable and you do not have the availability template.

Putting utility meter inbetween does not solve anything, it may even make it worse.

[e-raser]

@Edwin_D

You yourself are putting | float(0) in your template, saying to use 0 if the sensor is unavailable - instead of making the summation sensor itself unavailable.

In doing you created a sensor that goes up and down instead of just up. And because your summation sensor is never unavailable, even if you miss a value, is exactly why HA cannot do a single thing about it and will just do what the state_class says it should do when the value decreases. There’s no bullet proofing for that.

Right. float is needed and it needs to have a default like int since idk, HA 2021.something.

You’re right: not using an availability_template is (was, hopefully once confirmed working) one part of the problem.

I’m right: the other part - which I proved undeniably with long term sensor data from my InfluxDB and the HA Core update event - is that this only became a problem with a specific HA release.

…And I want to know if and which release might have fixed this. I read all the release notes twice, especially the breaking changes sections. There was NOTHING there stating “starting with this release, you must use availability templates, otherwise you’ll get into serious trouble”. Did you?

@Edwin_D

The only thing that happened since a few months is that your sensors are more often seen unavailable, most likely during a reboot.

Wrong assumption. Incidents do not match restarts of HA. It’s more connection/stability issues as one part of the sensors in my case are WiFi based (Shelly devices) or ZigBee based.

You can’t deny some change in Core made not using availability templates expose all those problems at all - even it always seems to have been a good idea of using them. …which is why I upvoted your FR by the way.

[e-raser]

So: using availability templates obviously “fixed” this, no more 0 values anymore. Example:

I believe I can answer my question #2 …

@e-raser

Which HA Core release fixed this issue? I went through all posts but could not find a PR for the fix.

…on my own based on the reactions: likely none. And never. It’s not a bug, it’s more the simple fact a behaviour has changed bringing not using availability templates up, this was somehow suppressed before (until 2023.3, started for me with 2023.4).

Nothing left to say… besides “use availability templates!”

By the way:

@Edwin_D

You yourself are putting | float(0) in your template

That’s sometimes inevitable - within a value template (int and float etc. need a default for many years, so usually people set it to 0). And not a problem as long as that value template is combined with an availability template.

[Edwin_D]

@e-raser

That’s sometimes inevitable

It is indeed inevitable. I meant to say that was where the 0 came from. I know not providing a default yields an error if no cast is possible, but that ends up in the log. It would be nice if you could say float(unavailable) to say: if you cannot cast, make the whole result unavailable. But I’m pretty sure you can’t.