Feature Request: Fix (or disable) "Untracked Consumption" in Energy Dashboard when only daily grid readings are available

[HappilyOffensive]

Firstly about my HA instance:

Core 2024.9.0
Supervisor 2024.08.0
Operating System 13.1
Frontend 20240904.0

With the last HA update the “untracked consumption” feature, handled in this pull-request, got released.

What I found so far, which I see as an “issue”:

1. I didn’t see an option to disable it… (although I can manually disable the “Untracked consumption” entity in the bar diagram)
2. When you receive your grid energy consumption just on a daily base (e.g. your utilities provider only offers a daily reading), the graph gets more or less un-readable → see
   
   
   It would be a blessing if this would be handled differently. I could imagine that the daily reading is divided by 24 and equally spread to each hour (in the hourly view).
3. When there are days, where the utility provider does not provide consumption for a whole day, the consumed energy measured by entities gets displayed as negative “untracked consumption” → see the first day in
   
   
   The same thing applies also to the hourly reading, for slots where no consumption readings are available (the negative “untracked consumption” bars), see
   

I guess this feature was implemented for crowds in mind, that have an immediate grid consumption readings and are not struggling that their utilities provider only delivers consumption readings infrequently. (yeah, it’s 2024 and this still happens) And I think it’s also worth to mention: The grid readings are from the previous day…

As long as point 2. and 3. are not addressed (for users with daily readings) it would be, imho, good for affected users to have the possibility to disable this functionality (point 1.), because it get’s annoying to always have to disable the “Untracked consumption” entity to get the same diagram as before the release of this functionality…

Although I would prefer if the daily readings are spread across the hourly bars and when no readings are available the bar for the untracked consumption isn’t displayed as negative untracked consumption.

What are your thoughts on this?

[Kastenbrust (Thomas)]

It’s clearly extremely buggy, it’s completely ruining the Energy dashboard and needs to be depreciated until the devs can get around to fixing it or showing the data in a more graceful manner

It shows completely whacky results for me like a random 3kWh of energy use at 4am today when literally nothing in the house was running and no meter including the grid one shows that energy use… It also shows a random amount of negative use which doesn’t exist for every hour of every day, which simply isn’t possible.

[BBE-FR (BBE)]

@HappilyOffensive , @Kastenbrust , you can already disable it !

You just have to clic on the unwanted device consumption in the caption (like any other device) to remove it from the display:
default view :

After clicking on “consommation non suivie” (french for “untracked consumption”)

And apparently this setting is stored (at least for your browser), at least for a while…

But I agree a way to disable it more permanently should be provided, thus you have my vote…

[sargonas]

There is definitely something buggy with this feature. I have a Span panel, which means every circuit on my breaker panel is individually monitored and tracked. It’s literally impossible for any device in my house to consume any amount of power and not have it run through one of the breakers that are monitored. Yet somehow, I am having all sorts of “untracked consumption” variations since this was enabled, with variations of +/- almost every single hour. On average, it’s around .5 kWh per hour either positive or minus, in increments everywhere from 0.01 to 0.80 with an average being close to around .4 or .5 kWh. If I average out every day, I seem to be coming in around being a net difference of around 1kwh give or take .25 a day. Sometimes positive, and sometimes negative.

However, my Span panel reports exactly how much power each circuit uses, all of which I monitor and track. It also reports exactly how much power I consume from the grid, and how much I send back TO the grid from excess solar, as well as how much my solar itself generates in total, so I’m really confused on how earth there can be a margin of error this big every hour, unless there is some kind of flaw in the feature? (or unless there is a secret outlet somewhere in my house tapped directly into my mains before the panel, that someone other than me is using that I don’t know about, LOL)

[karwosts]

There’s not much special about the calculation that could have an error, it just takes your calculated whole house consumption, and subtracts the sum of individual devices, and what’s left is the remainder is displayed.

If you think there’s a mathematical error, please raise and issue and we’ll take a look at it. But it should be fairly simple for you yourself to verify the readings.

All the information needed to calculate untracked should be in the energy.csv that you can download, so it should be pretty simple to cross-check the result. Feel free to attach a csv file and we’ll take a look.

[deurk (Alexandre Nizoux)]

Here you go. This is what I see this morning:

[gdesfeux (Guillaume Desfeux)]

Same problem here and I know why I can see some negative offset on my charts. Some of my devices aren’t reporting less than 1kWh consumption (not accurate enough let’s say). When they consume, HA is reporting negative consumption because in reality, they did not consume what they reported…

[deurk (Alexandre Nizoux)]

The data that I download does not show any negative untracked data:

Here is the CSV data:

entity_id,type,unit,2024-09-11T22:00:00.000Z,2024-09-11T23:00:00.000Z,2024-09-12T00:00:00.000Z,2024-09-12T01:00:00.000Z,2024-09-12T02:00:00.000Z,2024-09-12T03:00:00.000Z,2024-09-12T04:00:00.000Z,2024-09-12T05:00:00.000Z
sensor.main_power_corrected,grid_consumption,kWh,0.5099999999999909,1.1700000000000728,0.42000000000007276,0.4799999999997908,0.42000000000007276,0.42000000000007276,1.1699999999998454,0.6700000000000728
sensor.main_power_corrected_cost,grid_consumption_cost,EUR,0.10291800000000428,0.23610600000001014,0.08475600000002181,0.09686399999995032,0.08475600000002181,0.0847560000000076,0.23610599999996928,0.13520600000001792
sensor.water_l,water_consumption,L,4,0,0,0,0,0,0,20
sensor.water_l_cost,water_consumption_cost,EUR,0.012492400000000181,0,0,0,0,0,0,0.06246199999999824
sensor.sp_01_energy,device_consumption,kWh,0.020000000000010232,0.030000000000001137,0.01999999999998181,0.020000000000010232,0.020000000000010232,0.01999999999998181,0.020000000000010232,0.030000000000001137
sensor.sp_02_energy,device_consumption,kWh,0.03999999999996362,0.03999999999996362,0.049999999999954525,0.03999999999996362,0.0500000000001819,0.03999999999996362,0.03999999999996362,0.07999999999992724
sensor.sp_03_energy,device_consumption,kWh,0.010000000000019327,0.009999999999990905,0,0.009999999999990905,0.009999999999990905,0,0.010000000000019327,0.15000000000000568
sensor.sp_04_energy,device_consumption,kWh,0.11000000000001364,0.11000000000001364,0.11000000000001364,0.15999999999996817,0.11000000000001364,0.12000000000000455,0.10000000000002274,0.11000000000001364
sensor.sp_05_energy,device_consumption,kWh,0.009999999999990905,0.009999999999990905,0.010000000000104592,0,0.009999999999990905,0.009999999999990905,0,0.01999999999998181
sensor.sp_06_energy,device_consumption,kWh,0.029999999999972715,0.029999999999972715,0.030000000000086402,0.01999999999998181,0.029999999999972715,0.029999999999972715,0.029999999999972715,0.030000000000086402
sensor.sp_07_energy,device_consumption,kWh,0,0,0,0,0,0,0,0
sensor.sp_08_energy,device_consumption,kWh,0.08000000000004093,0.05999999999994543,0.06000000000005912,0.049999999999954525,0.05000000000006821,0.049999999999954525,0.049999999999954525,0.05000000000006821
sensor.sp_09_energy,device_consumption,kWh,0,0.6999999999998181,0,0,0.010000000000218279,0,0.7100000000000364,0
sensor.sp_10_energy,device_consumption,kWh,0.009999999999990905,0.01999999999998181,0.009999999999990905,0.020000000000095497,0.009999999999990905,0.009999999999990905,0.01999999999998181,0.009999999999990905
sensor.sp_11_energy,device_consumption,kWh,0,0,0,0,0,0,0,0
sensor.sp_12_energy,device_consumption,kWh,0.03999999999999204,0.05000000000001137,0.03999999999999204,0.03999999999999204,0.03999999999999204,0.040000000000020464,0.04999999999998295,0.020000000000010232
sensor.sp_13_energy,device_consumption,kWh,0,0,0,0,0,0,0,0
sensor.sp_14_energy,device_consumption,kWh,0,0.010000000000005116,0.00999999999999801,0,0.00999999999999801,0,0.00999999999999801,0.010000000000005116
sensor.sp_15_energy,device_consumption,kWh,0,0,0,0,0,0,0,0
sensor.sp_16_energy,device_consumption,kWh,0.09999999999999432,0.09000000000000341,0.09000000000000341,0.09000000000000341,0.08999999999997499,0.10000000000002274,0.09000000000000341,0.08999999999997499

[deurk (Alexandre Nizoux)]

The oddities in my case happen very consistently at 4am and I’ve seen that time mentioned in the comments above. Maybe a hint?

[muellthos (Hans-Thomas M.)]

My utility meter that is used in the energy dashboard configuration counts energy fed back from PV to the grid as negative. This also leads to negative values in untracked consumption.

[karwosts]

In your 4-5AM hour I see your total grid consumption is 0.42, and the sum of all individual devices is 0.44. 0.42 - 0.44 gives -0.02 for the untracked.

[deurk (Alexandre Nizoux)]

You’re absolutely right, don’t know what I have done to not see this. Leaves me clueless as to why it happens

[Stormageddon6]

I have solar panels with net metering and my untracked consumption values are just flat out wrong. It doesn’t seem this feature considers my setup in its assumptions at all.

I very much regret updating.

[HappilyOffensive]

So, having thought about it a little more, I would want the following configuration options for this feature:

• disabled (don’t show it, don’t calculate it, just don’t spend any resources on this - although: I don’t know the implementation and if that would actually help in that case (I will not engage in a discussion of “come on, this is just one little tiny calculation…”))
• default (as it is now, for the blessed people that have in-time and accurate readings)
• from-helper (then the maintainer of that instance could provide a helper entity and probably put his own algorithm in place on how to handle inconsistent daily readings from a utility company)

[msrbhome (Matt)]

There is obviously an error in the calculation, or the approach used for the calculation. In my data I see negative untracked consumption, which is strange as I have only consumption from the grid.

Could the error be a timing error? For example, I already calculate ‘Other energy’ as the difference between the mains energy and the sum of the individual circuits’ energy. However, depending on when each sensor updates the result is a very noisy signal in my calculations. I see large swings in the calculated result as large appliances turn on and off. To overcome this noise, I average over a period of time. But this is not an accurate way, in my experience, to gain a reading on untracked consumption.

[CptHindsight (Radu)]

I just set up my HA, so I might be making some mistakes.

I have a Shelly 3EM to read all the incoming energy on 3 phases.
Furthermore, I have a Valiant heat pump with a solar panel for hot water. (no PV)

I have integrated it via the Valiant HACS.

But I still have negative values when the hot water is consuming:

Am I doing anything wrong ?

[remizik (Remi)]

I also have this issue, always with negative values it seems (I don’t have solar panels or any other power generation source at home) :

[thomaskilian]

Where do tese values come from? From thin air? So it appears to me. I get a mix of positive and negative sources. I found some source by good guesses (my solar controller proviedes a total-lifetime and a resettable amount, one of which I discarded). But I have no clue whee the remainin ones come from.

[petro]

karwosts in this thread already described the functionality above.

@karwosts

it just takes your calculated whole house consumption, and subtracts the sum of individual devices, and what’s left is the remainder is displayed.

And karwosts would know because they added the functionality.

[thomaskilian]

That still leaves me in the dark with the question as to what that remainder is made up. I’d appreciate some pop-up or whatever to indicate the parts so I can eventually add them to the list of indiidual devices.