Riemann integration for discharge and loading behave different

[mxwi]

Hi,

I have a problem that is bringing me headache for a long time now… I can not figure out what I am doing wrong…

I have a Varta Element Battery with 5.5 kwh energy that is getting charged via my solarpanels.
I get the values via TCP-Modbus. If the values is negative the battery is discharging and vice versa.

To bring up my values to the energy-dashboard I split them up via templating

    varta_entladung_only: #discharge only. sensor.Varta_Aktuelle_Leistung = current power from battery
      friendly_name: "Varta aktuelle Entladung"
      device_class: power
      unit_of_measurement: kW
      attribute_templates:
        hour_last_updated: "{{ now().hour }}"
      value_template: >-
        {% if states('sensor.Varta_Aktuelle_Leistung') | float < 0 %}
          {{ ((states("sensor.Varta_Aktuelle_Leistung") | float /1000) | round(2)) * -1 }}
          {% else %}
          {{ 0 }}
        {% endif %}

    varta_beladung_only: #charging only. sensor.Varta_Aktuelle_Leistung = current power from battery
      friendly_name: "Varta aktuelle Beladung"
      device_class: power
      unit_of_measurement: kW
      attribute_templates:
        hour_last_updated: "{{ now().hour }}"
      value_template: >-
        {% if states('sensor.Varta_Aktuelle_Leistung') | float > 0 %}
          {{ ((states("sensor.Varta_Aktuelle_Leistung") | float /1000) | round(2)) }}
          {% else %}
          {{ 0 }}
        {% endif %}

With these two power values I calculate the amount of dischage/charge energy via the rieman integration.

- platform: integration
  source: sensor.varta_entladung_only
  name: varta_entladung_kwh3
  round: 2
  method: trapezoidal

- platform: integration
  source: sensor.varta_beladung_only
  name: varta_beladung_kwh2
  round: 2
  method: trapezoidal

On the first sight, everything is looking good and works. But something isn’t working as I expect it…
The two values don’t grow the same… The amount of energy going in the battery is magicly more, than what is coming out…

I even don’t know what happed about 1/14… the values were getting more closed together… I don’t know why… I didn’t touch the config since ~ christmas…

my ideas so far:

• the AC gets inverted to DC when charging and vice-versa. Varta Support told me the values are cleaned up and should match when compared
• change rieman method to left: nothing changed
• added hour_last_updated: “{{ now().hour }}” to template so the value gets a touch every hour. Nothing changed…
• Fetched the values via xml File to have another data source. Nothing changed.

I hope you have some ideas for this!

[tom_l]

@mxwi

The amount of energy coming out of the battery is magicly more, than what is going in…

Nice!

But seriously, your {% else %} statements are indented too far (move them in line with the {% if.... I’m not sure if that will affect it the template though.

The energy dashboard appears to be displaying more in than out.

Perhaps it is an issue with the aggregation/selection setting you used in Grafana?

Can you show a screenshot of the graph settings?

[mxwi]

But seriously, your {% else %} statements are indented too far (move them in line with the {% if... . I’m not sure if that will affect it the template though.

I don’t really understand what you mean, sorry. You mean I should write the whole statement in one line? Can you show me what you mean, please?

The energy dashboard appears to be displaying more in than out.

I confound in and out in my first post. Of course morge energy is going in the battery, than out. I fixed it in my first post.

Perhaps it is an issue with the aggregation/selection setting you used in Grafana?

The values are differencing in Grafana and on the energy-dashboard as well.

[tom_l]

@mxwi

Can you show me what you mean, please?

Change this:

      value_template: >-
        {% if states('sensor.Varta_Aktuelle_Leistung') | float < 0 %}
          {{ ((states("sensor.Varta_Aktuelle_Leistung") | float /1000) | round(2)) * -1 }}
          {% else %}
          {{ 0 }}
        {% endif %}

To this:

      value_template: >-
        {% if states('sensor.Varta_Aktuelle_Leistung') | float < 0 %}
          {{ ((states("sensor.Varta_Aktuelle_Leistung") | float /1000) | round(2)) * -1 }}
        {% else %}
          {{ 0 }}
        {% endif %}

[tom_l]

@mxwi

The values are differencing in Grafana and on the energy-dashboard as well.

No they aren’t.

This shows more going in than coming out of the battery (as would be expected due to efficiency losses):

Show your Grafana graph settings.

[mxwi]

Hi,

“Entladung” and “Beladung” are the same values in Grafana and Energydashboard. But they are differencing in value. Maybe I am using the wrong word. I would expect that they are nearly the same.

I understand that it would be maybe possible (even if the Varta-Support said something different) that due to efficiency the value are not 100% the same. But here we speak about 32% (degree of efficiency should be least 85%).

Grafana Settings: Pastebin

[tom_l]

80-90% Charge / discharge efficiency. there may be other losses.

Either way, as you are not grouping your values, try select: last instead of mean in grafana.

[KNXBroker]

Hi, I also have a VARTA Batterie. I’m a little bit frustrated that VARTA does have a charging counter but no discharging counter in the modbus protocol.

My I ask you which update interval you are using for using the Riemann integration?

[mxwi]

Thanks for your help so far!

@tom_l

This is my graph with “last”.
I can not see any significant difference. I can not believe, that the efficieny is at just 68%.
My guess is, that the problem is associated with the riemann integration.

@KNXBroker

Interval is 3 seconds for charge/discharge. 10 seconds for other values.

Nice hint for the energy counter in the modbus register! Missed that.But as you said, it’s sad to see no discharge counter.

I added a utily meter for a week for the charging value and will compare it to my calculted one. Exciting!

How did you configure your energy dashboard with the Varta Battery?

[dr-chef]

Hi,
I’ll also chime in here
I just got a brand new Varta Element and of course I’m facing the same issue. I’ve also contacted the Varta support, but aside from the latest version of the modbus register to no avail regarding the “total discharge”.
Another thing to do would be to use the “total capacity”, the “total charge” and the “SOC” to calculate the total discharge. But this has some drawbacks :

• Of course this will not include any efficiency loss
• As the percentage is rounded it will also “change” when it’s charging while it’s not discharging at all
• If the “total capacity” value does not change to reflect the actual capacity this will get worse when the battery gets older of course…

I’m currently trying this but it’s a bit bugging that it changes while it doesn’t really discharge. I need to test the riemann integration as well.

I’m happy to share / hear any other tipps about this device

[mxwi]

someone just dropped this on Github: https://github.com/home-assistant/core/pull/63736#issuecomment-1047713415

I did not know the link and it is not in any documentation I know. I have not test it, but I think it will be pretty easy to use…

In this post we have three guys who have a Varta Battery. I would be happy, if you would share your values with me. We could compare the degree of efficiency…

mine:

EGrid_AC_DC = 6135021;
EGrid_DC_AC = 7717033;
EWr_AC_DC = 2558221;
EWr_DC_AC = 1878959;
Chrg_LoadCycles = [386];

[KNXBroker]

Element 4, 6kwh, 5 years old: 75%. Which battery type do you have and how old?

[mxwi]

Varta Element (don’t know version, but it was installed 2020): ~ 73% calculating with the new values from the json file.

[dr-chef]

@mxwi That someone was actually me. @KNXBroker and I went on a little debugging excursion in the webinterface of the battery and discovered this “Endpoint”. At least this way we have the possibility to get the values from the battery and do not have to use the Riemann integration.

But to answer your question, my battery is brand new (Element S5) and I got around ~82% for now.

I hope hat this get’s added to the integration, it would be awesome if this would just work out of the box. But in case you’re interested I wrote a litte integration configuration to make it work in HA Energy (however I’m pretty sure my code could be improved upon, I’m really no expert in converting JS to JSON / readable format with jinja2)

[KNXBroker]

If you are using node red, here some code to get the energy.js data:

[{"id":"4cbf71c03cf0e4c9","type":"function","z":"42a5d63b.36e0c","name":"VARTA Netzbezug","func":"/*\n\nPattern codes:\n--------------\nEGrid_AC_DC == Grid -> Home (Wh)\nEGrid_DC_AC == Home -> Grid (Wh)\nEWr_AC_DC == Inverter AC -> DC == Total Charged (Wh)\nEWr_DC_AC == Inverter DC -> AC == Total Discharged (Wh)\nChrg_LoadCycles == Charge Cycle Counter\n\nExample query:\n--------------\nEnergy Data: curl -X GET http://Varta-IP/cgi/energy.js\n\nExample answer of battery:\n---------------------------\nEGrid_AC_DC = 6135021;\nEGrid_DC_AC = 7717033;\nEWr_AC_DC = 2558221;\nEWr_DC_AC = 1878959;\nChrg_LoadCycles = [386];\n\n*/\n\nvar code = \"EGrid_AC_DC\";\nconst pattern = new RegExp(code + \" = (.*?);\");\nvar found = msg.payload.match(pattern);\nreturn { payload: parseInt(found[1].replace('[','').replace(']','')) };","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":1540,"wires":[["9fa61834a2d6604c"]]},{"id":"62d6bf0c4e40611f","type":"function","z":"42a5d63b.36e0c","name":"VARTA Netzlieferung","func":"/*\n\nPattern codes:\n--------------\nEGrid_AC_DC == Grid -> Home (Wh)\nEGrid_DC_AC == Home -> Grid (Wh)\nEWr_AC_DC == Inverter AC -> DC == Total Charged (Wh)\nEWr_DC_AC == Inverter DC -> AC == Total Discharged (Wh)\nChrg_LoadCycles == Charge Cycle Counter\n\nExample query:\n--------------\nEnergy Data: curl -X GET http://Varta-IP/cgi/energy.js\n\nExample answer of battery:\n---------------------------\nEGrid_AC_DC = 6135021;\nEGrid_DC_AC = 7717033;\nEWr_AC_DC = 2558221;\nEWr_DC_AC = 1878959;\nChrg_LoadCycles = [386];\n\n*/\n\nvar code = \"EGrid_DC_AC\";\nconst pattern = new RegExp(code + \" = (.*?);\");\nvar found = msg.payload.match(pattern);\nreturn { payload: parseInt(found[1].replace('[','').replace(']','')) };","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":560,"y":1600,"wires":[["dbad85148a751892"]]},{"id":"5c9afd39e7dbdfdf","type":"function","z":"42a5d63b.36e0c","name":"VARTA Ladezähler","func":"/*\n\nPattern codes:\n--------------\nEGrid_AC_DC == Grid -> Home (Wh)\nEGrid_DC_AC == Home -> Grid (Wh)\nEWr_AC_DC == Inverter AC -> DC == Total Charged (Wh)\nEWr_DC_AC == Inverter DC -> AC == Total Discharged (Wh)\nChrg_LoadCycles == Charge Cycle Counter\n\nExample query:\n--------------\nEnergy Data: curl -X GET http://Varta-IP/cgi/energy.js\n\nExample answer of battery:\n---------------------------\nEGrid_AC_DC = 6135021;\nEGrid_DC_AC = 7717033;\nEWr_AC_DC = 2558221;\nEWr_DC_AC = 1878959;\nChrg_LoadCycles = [386];\n\n*/\n\nvar code = \"EWr_AC_DC\";\nconst pattern = new RegExp(code + \" = (.*?);\");\nvar found = msg.payload.match(pattern);\nreturn { payload: parseInt(found[1].replace('[','').replace(']','')) };","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":1660,"wires":[["1c0e43a464e67da9"]]},{"id":"31576e8ae1c19e14","type":"function","z":"42a5d63b.36e0c","name":"VARTA Entladezähler","func":"/*\n\nPattern codes:\n--------------\nEGrid_AC_DC == Grid -> Home (Wh)\nEGrid_DC_AC == Home -> Grid (Wh)\nEWr_AC_DC == Inverter AC -> DC == Total Charged (Wh)\nEWr_DC_AC == Inverter DC -> AC == Total Discharged (Wh)\nChrg_LoadCycles == Charge Cycle Counter\n\nExample query:\n--------------\nEnergy Data: curl -X GET http://Varta-IP/cgi/energy.js\n\nExample answer of battery:\n---------------------------\nEGrid_AC_DC = 6135021;\nEGrid_DC_AC = 7717033;\nEWr_AC_DC = 2558221;\nEWr_DC_AC = 1878959;\nChrg_LoadCycles = [386];\n\n*/\n\nvar code = \"EWr_DC_AC\";\nconst pattern = new RegExp(code + \" = (.*?);\");\nvar found = msg.payload.match(pattern);\nreturn { payload: parseInt(found[1].replace('[','').replace(']','')) };","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":560,"y":1720,"wires":[["0c19a49d3c81d049"]]},{"id":"b177fee147ec29e4","type":"function","z":"42a5d63b.36e0c","name":"VARTA Ladezyklen","func":"/*\n\nPattern codes:\n--------------\nEGrid_AC_DC == Grid -> Home (Wh)\nEGrid_DC_AC == Home -> Grid (Wh)\nEWr_AC_DC == Inverter AC -> DC == Total Charged (Wh)\nEWr_DC_AC == Inverter DC -> AC == Total Discharged (Wh)\nChrg_LoadCycles == Charge Cycle Counter\n\nExample query:\n--------------\nEnergy Data: curl -X GET http://Varta-IP/cgi/energy.js\n\nExample answer of battery:\n---------------------------\nEGrid_AC_DC = 6135021;\nEGrid_DC_AC = 7717033;\nEWr_AC_DC = 2558221;\nEWr_DC_AC = 1878959;\nChrg_LoadCycles = [386];\n\n*/\n\nvar code = \"Chrg_LoadCycles\";\nconst pattern = new RegExp(code + \" = (.*?);\");\nvar found = msg.payload.match(pattern);\nreturn { payload: parseInt(found[1].replace('[','').replace(']','')) };","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":550,"y":1780,"wires":[["0b6e6539e43e04a5"]]},{"id":"ddb0038ae62e2b81","type":"inject","z":"42a5d63b.36e0c","name":"Abfrage VARTA 30s","props":[],"repeat":"30","crontab":"","once":false,"onceDelay":0.1,"topic":"","x":240,"y":1520,"wires":[["3e0232260f56ed4c"]]},{"id":"0b6e6539e43e04a5","type":"rbe","z":"42a5d63b.36e0c","name":"Größer Gleich","func":"deadbandEq","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":760,"y":1780,"wires":[["9e5f9a6ce0963d81"]]},{"id":"1c0e43a464e67da9","type":"rbe","z":"42a5d63b.36e0c","name":"Größer Gleich","func":"deadbandEq","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":760,"y":1660,"wires":[["2c994af343981e5f"]]},{"id":"0c19a49d3c81d049","type":"rbe","z":"42a5d63b.36e0c","name":"Größer Gleich","func":"deadbandEq","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":760,"y":1720,"wires":[["32c290a1bd5d52b3"]]},{"id":"dbad85148a751892","type":"rbe","z":"42a5d63b.36e0c","name":"Größer Gleich","func":"deadbandEq","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":760,"y":1600,"wires":[["16f7d3772f7d1f8f"]]},{"id":"9fa61834a2d6604c","type":"rbe","z":"42a5d63b.36e0c","name":"Größer Gleich","func":"deadbandEq","gap":"","start":"","inout":"out","septopics":true,"property":"payload","topi":"topic","x":760,"y":1540,"wires":[["6f49af089abc484c"]]},{"id":"32c290a1bd5d52b3","type":"ha-entity","z":"42a5d63b.36e0c","name":"VARTA Entladezähler","server":"fbf6781b.5ad848","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"VARTA Entladezähler"},{"property":"device_class","value":"energy"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":"Wh"},{"property":"state_class","value":"total_increasing"},{"property":"last_reset","value":""}],"state":"payload","stateType":"msg","attributes":[],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":1060,"y":1720,"wires":[[]]},{"id":"2c994af343981e5f","type":"ha-entity","z":"42a5d63b.36e0c","name":"VARTA Ladezähler","server":"fbf6781b.5ad848","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"VARTA Ladezähler"},{"property":"device_class","value":"energy"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":"Wh"},{"property":"state_class","value":"total_increasing"},{"property":"last_reset","value":""}],"state":"payload","stateType":"msg","attributes":[],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":1050,"y":1660,"wires":[[]]},{"id":"16f7d3772f7d1f8f","type":"ha-entity","z":"42a5d63b.36e0c","name":"VARTA Zähler Netzlieferung","server":"fbf6781b.5ad848","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"VARTA Zähler Netzlieferung"},{"property":"device_class","value":"energy"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":"Wh"},{"property":"state_class","value":"total_increasing"},{"property":"last_reset","value":""}],"state":"payload","stateType":"msg","attributes":[],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":1080,"y":1600,"wires":[[]]},{"id":"9e5f9a6ce0963d81","type":"ha-entity","z":"42a5d63b.36e0c","name":"VARTA Ladezyklen","server":"fbf6781b.5ad848","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"VARTA Ladezyklen"},{"property":"device_class","value":""},{"property":"icon","value":""},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":"measurement"},{"property":"last_reset","value":""}],"state":"payload","stateType":"msg","attributes":[],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":1050,"y":1780,"wires":[[]]},{"id":"6f49af089abc484c","type":"ha-entity","z":"42a5d63b.36e0c","name":"VARTA Zähler Netzbezug","server":"fbf6781b.5ad848","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"VARTA Zähler Netzbezug"},{"property":"device_class","value":"energy"},{"property":"icon","value":""},{"property":"unit_of_measurement","value":"Wh"},{"property":"state_class","value":"total_increasing"},{"property":"last_reset","value":""}],"state":"payload","stateType":"msg","attributes":[],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"allow","outputOnStateChange":false,"outputPayload":"$entity().state ? \"on\": \"off\"","outputPayloadType":"jsonata","x":1070,"y":1540,"wires":[[]]},{"id":"3e0232260f56ed4c","type":"http request","z":"42a5d63b.36e0c","name":"HTTP Abfrage","method":"GET","ret":"txt","paytoqs":"ignore","url":"http://VARTA-IP/cgi/energy.js","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"credentials":{},"x":280,"y":1580,"wires":[["4cbf71c03cf0e4c9","62d6bf0c4e40611f","5c9afd39e7dbdfdf","31576e8ae1c19e14","b177fee147ec29e4","fc0e24343bf93f19"]]},{"id":"fbf6781b.5ad848","type":"server","name":"Home Assistant","version":2,"addon":true,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":true,"cacheJson":true,"heartbeat":false,"heartbeatInterval":30}]

[mxwi]

has someone a snippet to pull the data in sensors from the json?

[dr-chef]

I’m happy to share.

But as it’s not real json but more plain javascript syntax you can’t use the home assistant REST JSON parser (or at least I didn’t find out how), so I’ve built some jinja code and used templating to get the values. Works pretty well, but I’m sure the code could be improved
Feedback is always welcome. Here we go:

Put everything in the configuration.yaml (you may need to merge with your own custom config)

First setup the “REST” call to pull the raw data and store it as string (don’t forget to replace VARTAIP with your actual IP)

sensor:
  - platform: rest
    name: varta_energy_totals_string
    scan_interval: 30
    resource: http://VARTAIP/cgi/energy.js
    value_template: "{{ value.replace(' ', '').replace('\n', '') }}"

Use templating to split this raw value up in four actual sensors:

template:
  - sensor:
      - name: "Varta Grid to Home Total"
        unit_of_measurement: "kWh"
        device_class: energy
        state_class: total
        state: >-
          {% set value = states('sensor.varta_energy_totals_string') | string %}
          {% for item in value.split(';') %}
            {% if item.startswith('EGrid_AC_DC') %}
              {{ item.split('=')[1] | int / 1000 }}
            {% endif %}
          {% endfor %}
      - name: "Varta Home to Grid Total"
        unit_of_measurement: "kWh"
        device_class: energy
        state_class: total
        state: >-
            {% set value = states('sensor.varta_energy_totals_string') | string %}
            {% for item in value.split(';') %}
            {% if item.startswith('EGrid_DC_AC') %}
                {{ item.split('=')[1] | int / 1000 }}
            {% endif %}
            {% endfor %}
      - name: "Varta Charged Total"
        unit_of_measurement: "kWh"
        device_class: energy
        state_class: total
        state: >-
          {% set value = states('sensor.varta_energy_totals_string') | string %}
          {% for item in value.split(';') %}
            {% if item.startswith('EWr_AC_DC') %}
              {{ item.split('=')[1] | int / 1000 }}
            {% endif %}
          {% endfor %}
      - name: "Varta Discharged Total"
        unit_of_measurement: "kWh"
        device_class: energy
        state_class: total
        state: >-
          {% set value = states('sensor.varta_energy_totals_string') | string %}
          {% for item in value.split(';') %}
            {% if item.startswith('EWr_DC_AC') %}
              {{ item.split('=')[1] | int / 1000 }}
            {% endif %}
          {% endfor %}

Sometimes HA complains about the “last_reset” not being set, in that case add:

        attributes:
          last_reset: '1970-01-01T00:00:00+00:00'

to each template sensor. I’m not quite sure when you actually need this.

[mxwi]

Thank you very much!

[mxwi]

aw man, my whole energy dashboard history is gone. what a mess!
Is it possible to just replace my old calculated battery value with the given one from the json?

[dr-chef]

If you change the name of the sensors all the history will be gone… That is true.

What you can try:
Name the new sensors exactly as you named the old ones. Then the history will probably stay intact.
However, as the values will probably differ you will have a completely messed-up day, because the values changed in “one hour” drasticly and HA thinks that you generated / lost a lot of energy in one day

I had a similar thing when I played around with the values of my heat pump.

I’ll probably try to “reset” my Energy Dashboard once I have everything up and running the way I like it.

Note: Be sure to make a backup before, just in case