IN the CSV there is no negative value. You computed that.
What does “Consumed battery” mean?
What does “Grid energy consumed” mean?
And what do you think the data in the csv are for? Energy dashboard does exactly the same calculation and then displays it as a graph.
Consumed battery is energy from battery which was used for Home/Workplace Consumption:
- If there is in the same hour span also any energy going into the battery this will be subtracted from solar if there is any. If not (or not enough) then it’s subtracted from Grid Import.
- Then what’s left in Grid Import is added to Battery Discharge and is Total consumed. If result of this operation is negative (as in your case) = no column above 0 kWh line.
So steps for your highlighted column:
- Grid Import - Battery Charge:
0.8314 - 2.39 = -1.5586 - Result of previous calculation + Battery Discharge:
-1.5586 + 0.12 = -1.4386 - And the final result which is negative and has energy deficit produces skewed Energy usage.
And if the result of step number 1 above was positive value it would be used for Grid energy consumed.
Got it. Not the way I understand “cosumption” but fair enough.
Thank you for your time and patience to make that clear to me.
I found the source for the way to high value on the Battery Charge, it was the intergal helper set to trapezoidal, not to left integration method. It was an overshoot as a response to step input.
Thanks and my best wishes!
1 Like
Hi,
Day 1 - 19:00:
Grid energy consumed: 1.56 - 2.07 = -0.51 kWh
Total consumed: -0.51 + (0.23 - 0.11) = -0.39 kWh
Day 3 - 19:00:
Grid energy consumed: 2.64 - 2.14 = 0.5 kWh
Total consumed: 0.5 kWh
Days 1, 2 and 4 are similar in terms of there is not enough energy recorded to be to satisfy charging of the battery thus it looks like nothing was consuming energy.
Only on the day 3 there is left enough (> 0) to display any consumption at all.
Locations where the problem is likely to occur:
- Energy meter [device or sensor]
- Utility meter [sensor]
- Any other sensors involved in calculating of your Grid Import/Export
- Energy dashboards processing of long term statistics. But not in a way how it was originally presented because data from csv in fact do match what’s displayed in the graphs.
In short, in a given hour there is not enough energy coming from the grid recorded to satisfy charging of the battery and any additional home/workplace consumption.
Grid Power
Grid Energy
Battery Power
Battery Energy
There’s a werid jump at around 19:15 on the energy sensor. Why would that be?
Energy Sensors
- platform: integration
source: sensor.victron_battery_power_charged
name: Victron Battery Energy Charged
round: 3
unit_prefix: k
unit_time: h
unique_id: victron_xxxxxxxxxx_battery_energy_charged
- platform: integration
source: sensor.victron_battery_power_discharged
name: Victron Battery Energy Discharged
round: 3
unit_prefix: k
unit_time: h
unique_id: victron_xxxxxxxxxx_battery_energy_discharged
Power Sensors
- state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
name: "Battery Power"
object_id: "victron_battery_power"
unique_id: "victron_xxxxxxxxxx_battery_power"
device_class: power
state_class: measurement
value_template: "{{ value_json.value|float(default=0.0)|round(3, default=0) }}"
unit_of_measurement: "W"
icon: mdi:alpha-p
- state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
name: "Battery Power Charged"
object_id: "victron_battery_power_charged"
unique_id: "victron_xxxxxxxxxx_battery_power_charged"
device_class: power
state_class: measurement
value_template: >
{% set p = value_json.value|float(default=0.0)|round(3, default=0) %}
{% if p > 0 %}
{{ p }}
{% else %}
0.00
{% endif %}
unit_of_measurement: "W"
icon: mdi:alpha-p
- state_topic: "victron/N/xxxxxxxxxx/system/0/Dc/Battery/Power"
name: "Battery Power Discharged"
object_id: "victron_battery_power_discharged"
unique_id: "victron_xxxxxxxxxx_battery_power_discharged"
device_class: power
state_class: measurement
value_template: >
{% set p = value_json.value|float(default=0.0)|round(3, default=0) %}
{% if p < 0 %}
{{ p|abs }}
{% else %}
0.00
{% endif %}
unit_of_measurement: "W"
icon: mdi:alpha-p
Okay, so now it’s really clear to me. This missing energy is recorded in the previous hour.
Because all this battery charge energy was recorded at a single point in time, i.e. it takes one hour and then two hours to import the energy from the grid, the calculation is “out of sync” and produces distorted graphs.
So communication is most likely to blame. Or somewhere on the border between HA, integration and battery.
Is it strange that this seems to happen very regularly. But now that you know, you can explore further. 
Edit: Check what the comm between HA and battery is doing during charging from grid hours.
Your observation is valid, but it doesn’t mean that the graph is correct. The fact that everything doesn’t nett out is a different problem all together.
There is no math required to display the usage of different energy sources on a graph. If the sensor values don’t nett out it just means is that they don’t integrate the power values correctly. The graph should still be correctly showing what I recorded as per the spreadsheet, but instead, some unnecessary calculation is performed which leads to some energy sensor values cancelling out. It may be required for other calculations (e.g. Self-sufficiency) but not for the Energy Usage graph.
Not sure about the “out of sync”. It looks like the battery only starts charging at around 19:15. Prior to that the grid is consumed, so ti actually made sense that there was a flat line, but then from the previous point to the next point there passes some time and then the graph doesn’t smooth out the line from the last to the next point, but continues on the same value before shooting up to the latest point. The sudden jump in the battery energy charged makes sense, because the value increased by a significant amount since the previous data point. That could be an issue with how the energy sensor integrates, loss of connection or whatever, but doesn’t impact the way the Energy usage is displayed.
The spreadsheet uses the CSV data exported from the energy dashboard. Why does the dashboard not produce the same chart with the same data? Because it unnecessarily manipulates the data before rendering the chart.
Your observation is valid, but it doesn’t mean that the graph is correct.
Not true.
The fact that everything doesn’t nett out is a different problem all together.
And that’s exactly what the graph shows. That your data does not net out.
There is no math required to display the usage of different energy sources on a graph.
We are in a realm of “simulation”, it is simulation of real data, yes. But still, graph is only simulating hourly usage from data provided/recorded.
Energy dashboard works per hour. Not per day, week, month, year or lifetime.
If the sensor values don’t net out it just means is that they don’t integrate the power values correctly.
But this is exactly what you need to correctly display the graphs.
The graph should still be correctly showing what I recorded
And the graph in fact does exactly that.
Shows very precisely exactly that what’s recorded.
No guessing. No assumptions.
but instead, some unnecessary calculation is performed which leads to some energy sensor values cancelling out
If you just want to see columnar graphs which just shows values of your sensor that’s not what’s the Energy usage graph is for. (It’s in the name that it shows usage)
It may be required for other calculations (e.g. Self-sufficiency) but not for the Energy Usage graph.
And that usage involves math operations too.
The sudden jump in the battery energy charged makes sense, because the value increased by a significant amount since the previous data point.
Yes but the data clearly shows that some of that sudden jump energy belongs to previous hour span because you did not recorded enough Grid Import from 19:00 to 20:00.
As I said Energy dashboard works on per hour basis thus won’t look for missing energy outside of that hour segment.
This would be already mentioned guessing and making assumptions.
That could be an issue with how the energy sensor integrates, loss of connection or whatever, but doesn’t impact the way the Energy usage is displayed.
Energy dashboard needs accurate data.
The spreadsheet uses the CSV data exported from the energy dashboard. Why does the dashboard not produce the same chart with the same data? Because it unnecessarily manipulates the data before rendering the chart.
Your graph is either just merge of all sensors (showing data from all sensors within single graph which means is not showing Energy usage) or does not provide view on per hour basis which is core of the Energy dashboard.
- Find the issue with your sensors, fix it and you will fix the graph too.
- I myself would change many things about how it works but the fact that the Energy dashboard doesn’t work the way you want it does not make it wrong. It works exactly as described. If you don’t agree with that you can propose a change or make a PR yourself.
I made some changes to my energy sensors to ensure they update every second even when the power sensor has not changed. No improvement.
Here’s the sensor data for power and energy for yesterday. I’ve also added the energy dashboard data to the sheet
Where would you suggest I start looking?
Can you give me info about your setup? (What are the devices which provide this sensors and how is it inter-connected)
I have for example Deye inverter and all sensors used in energy dashboard comes from single device so I’m guessing your case is a bit different and you are gathering data from multiple devices right?
Where would you suggest I start looking?
Can you somehow track (w/ sufficient resolution) connection state to a device which provide Victron Battery Energy Charged? (and last_updated and last_changed attributes of this sensor too)
I have a Victron setup. All power data comes from the Cerbo GX via MQTT which is bridged with my Mosquitto broker on HA. I have power sensors configured with the MQTT sensor integration and use integration sensors to create an energy sensors from the power sensors.
There are examples of the sensor config in WTH is wrong with Energy Usage when Importing from Grid - #14 by wernerhp
The Cerbo GX provides a bi-direction sensor for Grid and Battery, so I use a value template in the MQTT power sensor to split the power sensors for grid and battery into imported, exported, charged and discharged respectively. Solar and AC Loads are mono-directional so there’s no need to split them.
HA Energy Dashboard is configured to use the Grid Imported, Grid Exported, Solar Yield, Battery Charged and Battery Discarded sensors. AC Loads are not used by HA Energy Dashboard.
The data updates constantly (like every second) over MQTT, but only when it changes. The Battery MQTT message would therefore update from - discharging to + charging, and due to the value template, the discharging sensor stops updating when the battery starts charging and the charging sensor starts updating.
Let me know if you need anything else.
Can you show me that raw sensor (w/o any postprocessing using templates, etc.) which is the source for Victron Battery Energy Charged? (Or if it’s already somewhere between the data you provided which one is it?)
https://drive.google.com/drive/folders/14Wz2WgxHtOhOi_T_VQBS-K9lH_0p4AA2
Here you go The Battery Power sensor is the data I get over MQTT.
value_template: "{{ value_json.value|float(default=0.0)|round(3, default=0) }}"
unit_of_measurement: "W"
It says this folder is empty…?
Whoop! Apologies, it was awaiting WiFi to upload the files. Should be available now.
The Victron log sheet contains data downloaded from Victron’s Dashboard. This should be what I get over MQTT
Show me configuration of your Riemann sum integrals for Grid Power and Battery Power.
- platform: integration
source: sensor.victron_grid_power
name: Victron Grid Energy
round: 3
unit_prefix: k
unit_time: h
max_sub_interval:
seconds: 1
unique_id: victron_grid_energy
- platform: integration
source: sensor.victron_grid_power_imported
name: Victron Grid Energy Imported
round: 3
unit_prefix: k
unit_time: h
max_sub_interval:
seconds: 1
unique_id: victron_grid_energy_imported
- platform: integration
source: sensor.victron_grid_power_exported
name: Victron Grid Energy Exported
round: 3
unit_prefix: k
unit_time: h
max_sub_interval:
seconds: 1
unique_id: victron_grid_energy_exported
- platform: integration
source: sensor.victron_battery_power_charged
name: Victron Battery Energy Charged
round: 3
unit_prefix: k
unit_time: h
max_sub_interval:
seconds: 1
unique_id: victron_battery_energy_charged
- platform: integration
source: sensor.victron_battery_power_discharged
name: Victron Battery Energy Discharged
round: 3
unit_prefix: k
unit_time: h
max_sub_interval:
seconds: 1
unique_id: victron_battery_energy_discharged
Oh, I did not notice that when you posted your configuration before but you are missing:
method: left
w/o it defaults to trapezoidal.
Edit: Yeah I think left should be the default but…
1 Like
Ok, will give it a shot.