WTH is wrong with Energy Usage when Importing from Grid

[wernerhp (Werner)]

Discord

I have a weird issue with my Energy Dashboard’s Energy Usage.
At 05:00-06:00 and again at 19:00-20:00 my battery was charged using the Grid.

The morning seems ok, but the evening doesn’t show any Grid Import for 19:00-20:00. (It does show for 18:00-19:00). What gives?

As can be seen on the History for the sensors, there was Grid Power Imported at both these times, but it doesn’t reflect on the Energy Dashboard.

The energy sensors also show an increase at these times

Sensor attributes for reference

Energy Dashboad Config

Energy Distribution seems to align with that from Victron.

Any ideas?

I suspect it’s a bug in the Energy Dashboard.

Energy Dashboard shows wrong Data (from Grid) in Hour Overview

Energy Dashboard doesn't display 'consumed solar' in some scenarios

[davidrapan (David Rapaň)]

Hi, I’m also charging battery ~2 times a day and Energy dashboard is working fine for me.

In your case it looks like you are both charging and discharging to/from the battery at the same time. Could you also show Victron Battery Energy Discharged chart from that period?

[wernerhp (Werner)]

See the Discord discussion linked at the top and the last issue linked, which is related. Many people in that last issue agree that there is a bug.

[Mircea (Mircea Chiriciuc)]

Hi,
I have the same issue, at random times, the discharge is negative. It is impossible and the utility meter is counting up.
I have downloaded de csv data from the energy dashboard and in the downloaded data there is no negative value. Also the values on the csv is correct, but on the graph it shows bogus data -1.44kWh. Also on the graph there is no grid consumption.

[Mircea (Mircea Chiriciuc)]

And in the csv you cannot find any negative numbers. Also battery out is 0.12, not -1.44.

Checking the history for the entity shows that 0.12 is the correct number

I think we can conclude that there is a bug somewhere

[davidrapan (David Rapaň)]

Hi,

Your case:

0.8314 - 2.39 + 0.12 = -1.4386

My case (2024-11-27T03:00:00.000Z) and shows correct chart:

3.2 - 2.5 + 0 = 0.7

Note: Chart is displayed as UTC+1.

[Mircea (Mircea Chiriciuc)]

Hi,

As far as I know, that is how “Total Consumed” is calculated.
I was talking about “Consumed Battery” which should be 0.12 since that is the energy that came out of the battery. And the fact that the blue bar for “Grid Energy Consumed” is missing, although there are 0.8kWh consumed from grid as per csv.

Looking again at the numbers, the 2.39kWh “Battery Charge” is absolutely wrong, since my only source of energy was the grid (winter and night). So yes, 1.436kWh are comming out of the blue. And it is confusing for me too, not only to the graph

[davidrapan (David Rapaň)]

You said:

And in the csv you cannot find any negative numbers.

So I showed you where that negative -1.44 kWh is.

So yes, 1.436kWh are comming out of the blue.

But it’s visible in the csv. Your graph matches the data from the csv.

So either Grid Energy Import (should be a lot higher) or Battery Charge (a lot lower) are incorrect.

Can you show me long term stats of those two entities from that time period?

[Mircea (Mircea Chiriciuc)]

IN the CSV there is no negative value. You computed that.
What does “Consumed battery” mean?
What does “Grid energy consumed” mean?

[davidrapan (David Rapaň)]

And what do you think the data in the csv are for? Energy dashboard does exactly the same calculation and then displays it as a graph.

Consumed battery is energy from battery which was used for Home/Workplace Consumption:

• If there is in the same hour span also any energy going into the battery this will be subtracted from solar if there is any. If not (or not enough) then it’s subtracted from Grid Import.
• Then what’s left in Grid Import is added to Battery Discharge and is Total consumed. If result of this operation is negative (as in your case) = no column above 0 kWh line.

So steps for your highlighted column:

1. Grid Import - Battery Charge: 0.8314 - 2.39 = -1.5586
2. Result of previous calculation + Battery Discharge: -1.5586 + 0.12 = -1.4386
3. And the final result which is negative and has energy deficit produces skewed Energy usage.

And if the result of step number 1 above was positive value it would be used for Grid energy consumed.

[Mircea (Mircea Chiriciuc)]

Got it. Not the way I understand “cosumption” but fair enough.
Thank you for your time and patience to make that clear to me.

I found the source for the way to high value on the Battery Charge, it was the intergal helper set to trapezoidal, not to left integration method. It was an overshoot as a response to step input.

Thanks and my best wishes!

[wernerhp (Werner)]

Here’s the expected vs actual for the last 4 days…

[davidrapan (David Rapaň)]

Hi,

Day 1 - 19:00:
Grid energy consumed: 1.56 - 2.07 = -0.51 kWh
Total consumed: -0.51 + (0.23 - 0.11) = -0.39 kWh

Day 3 - 19:00:
Grid energy consumed: 2.64 - 2.14 = 0.5 kWh
Total consumed: 0.5 kWh

Days 1, 2 and 4 are similar in terms of there is not enough energy recorded to be to satisfy charging of the battery thus it looks like nothing was consuming energy.

Only on the day 3 there is left enough (> 0) to display any consumption at all.

Locations where the problem is likely to occur:

1. Energy meter [device or sensor]
2. Utility meter [sensor]
3. Any other sensors involved in calculating of your Grid Import/Export
4. Energy dashboards processing of long term statistics. But not in a way how it was originally presented because data from csv in fact do match what’s displayed in the graphs.

In short, in a given hour there is not enough energy coming from the grid recorded to satisfy charging of the battery and any additional home/workplace consumption.

[wernerhp (Werner)]

Grid Power

Grid Energy

Battery Power

Battery Energy

There’s a werid jump at around 19:15 on the energy sensor. Why would that be?

Energy Sensors

- platform: integration
  source: sensor.victron_battery_power_charged
  name: Victron Battery Energy Charged
  round: 3
  unit_prefix: k
  unit_time: h
  unique_id: victron_xxxxxxxxxx_battery_energy_charged

- platform: integration
  source: sensor.victron_battery_power_discharged
  name: Victron Battery Energy Discharged
  round: 3
  unit_prefix: k
  unit_time: h
  unique_id: victron_xxxxxxxxxx_battery_energy_discharged

Power Sensors

  - state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
    name: "Battery Power"
    object_id: "victron_battery_power"
    unique_id: "victron_xxxxxxxxxx_battery_power"
    device_class: power
    state_class: measurement
    value_template: "{{ value_json.value|float(default=0.0)|round(3, default=0) }}"
    unit_of_measurement: "W"
    icon: mdi:alpha-p

  - state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
    name: "Battery Power Charged"
    object_id: "victron_battery_power_charged"
    unique_id: "victron_xxxxxxxxxx_battery_power_charged"
    device_class: power
    state_class: measurement
    value_template: >
      {% set p = value_json.value|float(default=0.0)|round(3, default=0) %}
      {% if p > 0 %}
        {{ p }}
      {% else %}
        0.00
      {% endif %}
    unit_of_measurement: "W"
    icon: mdi:alpha-p

  - state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
    name: "Battery Power Discharged"
    object_id: "victron_battery_power_discharged"
    unique_id: "victron_xxxxxxxxxx_battery_power_discharged"
    device_class: power
    state_class: measurement
    value_template: >
      {% set p = value_json.value|float(default=0.0)|round(3, default=0) %}
      {% if p < 0 %}
        {{ p|abs }}
      {% else %}
        0.00
      {% endif %}
    unit_of_measurement: "W"
    icon: mdi:alpha-p

[davidrapan (David Rapaň)]

Okay, so now it’s really clear to me. This missing energy is recorded in the previous hour.

Because all this battery charge energy was recorded at a single point in time, i.e. it takes one hour and then two hours to import the energy from the grid, the calculation is “out of sync” and produces distorted graphs.

So communication is most likely to blame. Or somewhere on the border between HA, integration and battery.

Is it strange that this seems to happen very regularly. But now that you know, you can explore further.

Edit: Check what the comm between HA and battery is doing during charging from grid hours.

[wernerhp (Werner)]

Your observation is valid, but it doesn’t mean that the graph is correct. The fact that everything doesn’t nett out is a different problem all together.

There is no math required to display the usage of different energy sources on a graph. If the sensor values don’t nett out it just means is that they don’t integrate the power values correctly. The graph should still be correctly showing what I recorded as per the spreadsheet, but instead, some unnecessary calculation is performed which leads to some energy sensor values cancelling out. It may be required for other calculations (e.g. Self-sufficiency) but not for the Energy Usage graph.

Not sure about the “out of sync”. It looks like the battery only starts charging at around 19:15. Prior to that the grid is consumed, so ti actually made sense that there was a flat line, but then from the previous point to the next point there passes some time and then the graph doesn’t smooth out the line from the last to the next point, but continues on the same value before shooting up to the latest point. The sudden jump in the battery energy charged makes sense, because the value increased by a significant amount since the previous data point. That could be an issue with how the energy sensor integrates, loss of connection or whatever, but doesn’t impact the way the Energy usage is displayed.

The spreadsheet uses the CSV data exported from the energy dashboard. Why does the dashboard not produce the same chart with the same data? Because it unnecessarily manipulates the data before rendering the chart.

[davidrapan (David Rapaň)]

Your observation is valid, but it doesn’t mean that the graph is correct.

Not true.

The fact that everything doesn’t nett out is a different problem all together.

And that’s exactly what the graph shows. That your data does not net out.

There is no math required to display the usage of different energy sources on a graph.

We are in a realm of “simulation”, it is simulation of real data, yes. But still, graph is only simulating hourly usage from data provided/recorded.

Energy dashboard works per hour. Not per day, week, month, year or lifetime.

If the sensor values don’t net out it just means is that they don’t integrate the power values correctly.

But this is exactly what you need to correctly display the graphs.

The graph should still be correctly showing what I recorded

And the graph in fact does exactly that.

Shows very precisely exactly that what’s recorded.

No guessing. No assumptions.

but instead, some unnecessary calculation is performed which leads to some energy sensor values cancelling out

If you just want to see columnar graphs which just shows values of your sensor that’s not what’s the Energy usage graph is for. (It’s in the name that it shows usage)

It may be required for other calculations (e.g. Self-sufficiency) but not for the Energy Usage graph.

And that usage involves math operations too.

The sudden jump in the battery energy charged makes sense, because the value increased by a significant amount since the previous data point.

Yes but the data clearly shows that some of that sudden jump energy belongs to previous hour span because you did not recorded enough Grid Import from 19:00 to 20:00.

As I said Energy dashboard works on per hour basis thus won’t look for missing energy outside of that hour segment.

This would be already mentioned guessing and making assumptions.

That could be an issue with how the energy sensor integrates, loss of connection or whatever, but doesn’t impact the way the Energy usage is displayed.

Energy dashboard needs accurate data.

The spreadsheet uses the CSV data exported from the energy dashboard. Why does the dashboard not produce the same chart with the same data? Because it unnecessarily manipulates the data before rendering the chart.

Your graph is either just merge of all sensors (showing data from all sensors within single graph which means is not showing Energy usage) or does not provide view on per hour basis which is core of the Energy dashboard.

---

• Find the issue with your sensors, fix it and you will fix the graph too.
• I myself would change many things about how it works but the fact that the Energy dashboard doesn’t work the way you want it does not make it wrong. It works exactly as described. If you don’t agree with that you can propose a change or make a PR yourself.

[wernerhp (Werner)]

I made some changes to my energy sensors to ensure they update every second even when the power sensor has not changed. No improvement.

Here’s the sensor data for power and energy for yesterday. I’ve also added the energy dashboard data to the sheet

Where would you suggest I start looking?