Solar Analytics API integration into HA Energy Management

PeterH24x7 · December 21, 2021, 10:01pm

*NOTE: THE COMPLETE FUNCTIONING DETAILS FOR INTEGRATING SOLAR ANALYTICS WITH HA ENERGY MANAGEMENT ARE INCLUDED WITHIN THIS THREAD. REFER TO GITHUB FOR ANY UPDATES AND ENHANCEMENTS.

https://github.com/PeterH24x7/-Solar-Analytics-integration-for-Home-Assistant-Energy-monitoring

Hi everyone, First time poster here. I have an ultimate goal of integrating the data from my Solar Analytics energy monitoring device (swtich-board installed 6 channel clamp based energy monitor). It has a public API (https://api-docs.solaranalytics.com/OpenAPI-Specs/sapublic-openapi.yaml) which I’ve been able to readily use via python and curl based queries. I’m now trying to do the same in HA and am firstly doing some basic data integration before delving into Energy Management. I can do the “gets” that provide system status and total reads without trouble. However I’m stuck when I wish to read the latest hourly data which requires an array index. I can work out the index (e.g. 7 for 7am), but then can’t work out how to use this index in the json_attributes_path.

If I use a constant fixed index - this is what I get and it works as expected. Anything other than a fixed index leaves me with no attributes. See long code below.

    json_attributes_path: "$.data.[7]"

Capture1

How do I fix the json_attributes_path definition? Or is there a better way? I’m a bit of a yaml rookie so any tips would be appreciated.

Thank you. Peter

See code configuration.yaml extract as follows:

# begin

  - platform: rest
    name: sa_data_by_hour
    resource_template: 'https://portal.solaranalytics.com.au/api/v2/site_data/XXXXX?gran=hour&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}'   
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic

    # value_template works
    value_template: "{{ (now()-timedelta(hours=1)).strftime('%-H') | int }}"

    # this doesn't work - what's wrong??? Please help.
    json_attributes_path: "$.data.[sa_data_by_hour.value_template]"

    json_attributes:
      - "energy_consumed"
      - "energy_generated" 
      - "t_stamp"
    scan_interval: 3600
    force_update: true

# end

PeterH24x7 · December 22, 2021, 2:44am

For reference, here’s the output for the ‘get’. You can see the consumed/generated kWh for each hour up until 9pm, where the day is yet to complete.

{
   "data":[
      {
         "energy_consumed":1921,
         "energy_generated":0,
         "t_stamp":"2021-12-18 00:00:00"
      },
      {
         "energy_consumed":1730,
         "energy_generated":0,
         "t_stamp":"2021-12-18 01:00:00"
      },
      {
         "energy_consumed":1745,
         "energy_generated":0,
         "t_stamp":"2021-12-18 02:00:00"
      },
      {
         "energy_consumed":1725,
         "energy_generated":0,
         "t_stamp":"2021-12-18 03:00:00"
      },
      {
         "energy_consumed":1631,
         "energy_generated":0,
         "t_stamp":"2021-12-18 04:00:00"
      },
      {
         "energy_consumed":1709,
         "energy_generated":-3,
         "t_stamp":"2021-12-18 05:00:00"
      },
      {
         "energy_consumed":1967,
         "energy_generated":86,
         "t_stamp":"2021-12-18 06:00:00"
      },
      {
         "energy_consumed":2044,
         "energy_generated":538,
         "t_stamp":"2021-12-18 07:00:00"
      },
      {
         "energy_consumed":2193,
         "energy_generated":1383,
         "t_stamp":"2021-12-18 08:00:00"
      },
      {
         "energy_consumed":3803,
         "energy_generated":3627,
         "t_stamp":"2021-12-18 09:00:00"
      },
      {
         "energy_consumed":2495,
         "energy_generated":3595,
         "t_stamp":"2021-12-18 10:00:00"
      },
      {
         "energy_consumed":292,
         "energy_generated":5317,
         "t_stamp":"2021-12-18 11:00:00"
      },
      {
         "energy_consumed":4635,
         "energy_generated":5867,
         "t_stamp":"2021-12-18 12:00:00"
      },
      {
         "energy_consumed":1539,
         "energy_generated":7169,
         "t_stamp":"2021-12-18 13:00:00"
      },
      {
         "energy_consumed":7616,
         "energy_generated":7105,
         "t_stamp":"2021-12-18 14:00:00"
      },
      {
         "energy_consumed":1626,
         "energy_generated":6781,
         "t_stamp":"2021-12-18 15:00:00"
      },
      {
         "energy_consumed":1505,
         "energy_generated":5231,
         "t_stamp":"2021-12-18 16:00:00"
      },
      {
         "energy_consumed":742,
         "energy_generated":3744,
         "t_stamp":"2021-12-18 17:00:00"
      },
      {
         "energy_consumed":319,
         "energy_generated":1603,
         "t_stamp":"2021-12-18 18:00:00"
      },
      {
         "energy_consumed":1644,
         "energy_generated":334,
         "t_stamp":"2021-12-18 19:00:00"
      },
      {
         "energy_consumed":2105,
         "energy_generated":15,
         "t_stamp":"2021-12-18 20:00:00"
      },
      {
         "energy_consumed":768,
         "energy_generated":-2,
         "t_stamp":"2021-12-18 21:00:00"
      },
      {
         "energy_consumed":"None",
         "energy_generated":"None",
         "t_stamp":"2021-12-18 22:00:00"
      },
      {
         "energy_consumed":"None",
         "energy_generated":"None",
         "t_stamp":"2021-12-18 23:00:00"
      }
   ],
   "t_step":60
}

TheOtherGlen · December 30, 2021, 5:30am

Thanks for posting this. It gave me quite a headstart to integrate SA into my own HA.

If you remove your “json_attributes_path” and just use the following “value_template” then it should work… Well, it did for me at least.

  value_template: "{{ value_json.data[ (now()-timedelta(hours=1)).strftime('%-H') | int ] }}"

Regards
Glen

TheOtherGlen · January 2, 2022, 4:56am

Hi Peter

I’ve been playing with this over the last few days and here’s what I’ve learned…

My previous response didn’t quite work as I thought it did. The correct array item was being extracted but the the json_attributes parameter did not work as it expects the attributes to be at the top level or at the level specified by json_attributes_path. I couldn’t work out how to get json_attributes_path to refer to a single item in an array so I gave up on this idea
(Note: my best attempt was:

json_attributes_path: "$.data[{{ (now().hour - 1 }}]"

but this didn’t seem to work)

Instead of trying to find the correct array item in the rest entity itself, I changed the rest entity to return the whole json result and then used template sensors to extract the attributes I needed from there:

- platform: rest
  name: sa_hour
  resource_template: 'https://portal.solaranalytics.com.au/api/v2/site_data/XXXXXX?gran=hour&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}&trunc=false'
  username: !secret sa_username
  password: !secret sa_password
  authentication: basic
  value_template: "OK"
  json_attributes:
    - "data"
  scan_interval: 3600
  force_update: true
- platform: template
  sensors:
    sa_hour_energy_consumed:
      unique_id: "sa_hour_energy_consumed"
      friendly_name: Energy Consumed Per Hour
      unit_of_measurement: "kWh"
      value_template: "{{ states.sensor.sa_hour.attributes.data[now().hour - 1].energy_consumed }}"
      icon_template: mdi:transmission-tower-export
    sa_hour_energy_produced:
      unique_id: "sa_hour_energy_produced"
      friendly_name: Energy Produced Per Hour
      value_template: "{{ states.sensor.sa_hour.attributes.data[now().hour - 1].energy_generated }}"
      icon_template: mdi:transmission-tower-import

(I made a few minor changes to simplify things a bit)

This worked BUT there were a few things I didn’t like about it.

The hourly scan interval was a bit too infrequent for me
This sensor acts like a “measurement” but it only allows you to take that measurement once the whole block (eg. hour, day, etc) is complete. Hence the “now().hour - 1” index. This means the measurement is nearly always old and out-of-date.

In order to get a more up-to-date measurement, I decided to sum up all of the array items to produce a total measurement for the whole day, at that point in time. This measurement can include the very latest array item (which might only be part of a block) and it would still be accurate.

This is how I achieved it:

sensors.yaml

- platform: rest
  name: sa_hour
  resource_template: 'https://portal.solaranalytics.com.au/api/v2/site_data/XXXXXX?gran=hour&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}&trunc=false'
  username: !secret sa_username
  password: !secret sa_password
  authentication: basic
  value_template: "OK"
  json_attributes:
    - "data"
  scan_interval: 10
  force_update: true
- platform: template
  sensors:
    sa_hour_energy_consumed_total:
      friendly_name: Total Energy Consumed
      unit_of_measurement: "kWh"
      value_template: "{{ states.sensor.sa_hour.attributes.data | rejectattr('energy_consumed', 'equalto', None) | sum(attribute='energy_consumed') }}"
      icon_template: mdi:transmission-tower-export
    sa_hour_energy_produced_total:
      friendly_name: Total Energy Produced
      unit_of_measurement: "kWh"
      value_template: "{{ states.sensor.sa_hour.attributes.data | rejectattr('energy_generated', 'equalto', None) | sum(attribute='energy_generated') }}"
      icon_template: mdi:transmission-tower-import

customize.yaml

sensor.sa_hour_energy_consumed_total:
  device_class: energy
  state_class: total_increasing
sensor.sa_hour_energy_produced_total:
  device_class: energy
  state_class: total_increasing

I considered doing the same thing using the “day” granularity but I noticed that the “day” value doesn’t seem to be as updated (on the server) as often as the “hour” and “minute” granularity so using the “hour” granularity gave the best mix of frequent updates with smaller response payload.
Note that even though I use the hourly granularity, I still request it every 10 secs using the scan_interval parameter. 10 seconds might be too frequent as it seems like the server value for the hourly granularity is only updated every 4-5 mins, but I’ll stick with 10 secs for now and see how it goes.

Anyway, these sensors should get you closer to meeting the requirements for Energy Management.
I’m looking forward to see if you can take what I’ve done and add to it/improve it.

Cheers
Glen

TheOtherGlen · January 2, 2022, 5:25am

And I’ve just realised the values are reported in the API are “Wh” not “kWh”

PeterH24x7 · January 4, 2022, 11:22am

Hi Glen - thank you for your contributions and sorry for my slow follow-up (after not seeing a response from my initial post for some days I’d almost given up). I will review your template approach in more detail tomorrow. In the mean time I had managed to get the hourly data into a list as attributes very similar to your code. While being able to extract the current hour is interesting, I’m keen to work out how to integrate the hour consumption and generation data into the HA Energy Management module. Despite searching I’ve not had much luck finding any intro, tutorial, video, samples etc that might give me at least some hints. I’m keen to collaborate on solving the matter as you see fit.

PeterH24x7 · January 4, 2022, 11:45pm

Hi Glenn - I’ve taken your code and added device_class and state_class (see below). The good news is that the consumed and generated sensors can now be seen in the Energy module. However this is still not complete as the Energy module rather needs the energy imported and exported. Using hourly data this can be roughly calculated as consumed minus generated. However inaccuracies will slip in where you turn on your microwave when a cloud comes over and therefore maybe need to draw from the grid (where there’s net generation for the hour, but some energy drawn from the grid for a brief period of time).

Capture

Nonetheless, if we can do the hour by hour consumed minus generated where the a postive value is import (export = 0) and a negative value is export (import = 0) then we are part way there - how good is your for loop coding to iterate the data by each hour and do the value calcs for import/export?

The good news is that we can get more granular SA data at the 5 minute level for the current day - I’ll work on this rest code later today.

  - platform: template
    sensors:
      sa_todays_energy_consumed_total:
        friendly_name: Total Energy Consumed
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_hour.attributes.data | rejectattr('energy_consumed', 'equalto', None) | sum(attribute='energy_consumed') }}"
        icon_template: mdi:transmission-tower-export
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
      sa_todays_energy_generated_total:
        friendly_name: Total Energy Gnerated
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_hour.attributes.data | rejectattr('energy_generated', 'equalto', None) | sum(attribute='energy_generated') }}"
        icon_template: mdi:transmission-tower-import
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing

PeterH24x7 · January 5, 2022, 12:16am

In the meantime here are some other SA rest related data calls:

Example lovelace card::
Capture

Configuration.yaml code::

sensor:
# Check the status of the Solar Analytics monitoring
  - platform: rest
    name: sa_status

    # Insert Solar Analytics site id - replace XXXX with number - could be 3 or 4 digits
    resource: https://portal.solaranalytics.com.au/api/v3/site_status/XXXX

    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ value_json['data']['mer_status'] }}"
    json_attributes_path: "$.data"
    json_attributes:
      - "dashboard_status"
      - "event_id"
      - "event_list"
      - "fault_status"
      - "mer_percentage"
      - "mer_status"
      - "mer_text"
    scan_interval: 3600
    force_update: true

  - platform: template
    sensors:
      sa_dashboard_status:
        friendly_name: "Dashboard Status"
        value_template: "{{ state_attr('sensor.sa_status', 'dashboard_status') }}"
        entity_id: sensor.sa_status
      sa_mer_status:
        friendly_name: "System Status"
        value_template: "{{ state_attr('sensor.sa_status', 'mer_status') }}"
        entity_id: sensor.sa_status
      sa_mer_percentage: 
        friendly_name: "PV Performance"
        value_template: "{{ state_attr('sensor.sa_status', 'mer_percentage') }}"
        entity_id: sensor.sa_status
        unit_of_measurement: "%"

PeterH24x7 · January 5, 2022, 1:13am

Here’s another data set that returns the Solar Analytics site list for the end-user - usually only one site. Importantly, the attribute data contains the site_id that is required for other SA data gets.

Example HA lovelace view showing the site attributes (assuming there’s only one site)::

Sample configuration.yaml code::

sensor:
# Get the site list details for the Solar Analytics account (uname/pword). 
# Note: the attributes are only shown for a single (the first) site.
  - platform: rest
    name: sa_site_list
    resource: https://portal.solaranalytics.com.au/api/v3/site_list?hardware=true&capacity=true&subscription=true
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ now() }}"
    json_attributes_path: "$.data.[0]"
    json_attributes:
      - "e_status"
      - "fault_class"
      - "fault_id"
      - "has_pv"
      - "mer_status"
      - "overall_status"
      - "retailer_user"
      - "s_cli_site_name"
      - "site_id"
      - "site_inactive"
      - "capacity"
      - "devices"
      - "sub_type"
    scan_interval: 62400

PeterH24x7 · January 5, 2022, 1:43am

Here’s the update that now does the same/similar to the hourly data, but based on 5 minute intervals.

sensor:

  - platform: rest
    name: sa_data_by_5min

# Insert Solar Analytics site id - replace XXXX with number - could be 3 or 4 digits
    resource_template: 'https://portal.solaranalytics.com.au/api/v2/site_data/XXXX?gran=minute&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}'   

    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ now() }}"
    json_attributes:
      - "data"
    state_class: measurement
    device_class: energy
    scan_interval: 300
    force_update: true

  - platform: template
    sensors:
      sa_todays_energy_consumed_total:
        friendly_name: Total Energy Consumed
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_consumed', 'equalto', None) | sum(attribute='energy_consumed') }}"
        icon_template: mdi:transmission-tower-export
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
      sa_todays_energy_generated_total:
        friendly_name: Total Energy Generated
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_generated', 'equalto', None) | sum(attribute='energy_generated') }}"
        icon_template: mdi:transmission-tower-import
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing

TheOtherGlen · January 5, 2022, 2:20am

Hi Peter

I couldn’t work out how to incorporate the state_class into the sensor itself so I did it via the customize.yaml. Well done for working it out. I’ll change mine now.

I too realised the issue with Energy Management requiring “grid import” and “grid export” as opposed to “solar produced” and “energy consumed”
I’ve found that the “raw data” service gives more detailed data per 5 minutes, including values like positive/negative energy and also power.
https://portal.solaranalytics.com.au/api/v2/site_data/XXXXXX?raw=true&trunc=false
I just haven’t had time to work out what each value really signifies and how to use this dataset to calculate the import/export values. Maybe you can take this service and do something with it?

{
  "data": [
    {
      "circuit": XXXXXX,
      "circuit_name": "XXXXXXXXXXXXXXX",
      "data": [
        {
          "apparentPower": 307.0808467305149,
          "current": 1.5855,
          "energy": 23.4028,
          "energyNeg": 0,
          "energyPos": 23.4028,
          "power": 280.8333,
          "powerFactor": 0.914525614313064,
          "reactiveEnergy": -10.3519,
          "reactivePower": -124.2228,
          "time": 1641254100,
          "voltage": 233.35
        },
        {
          ...
        }

However… one thing concerns me with all this is that the way the sensor component works is that it records the state at the time the measurement is taken (ie. when the service is called) and I can’t find a way to use the timestamp in the dataset itself. Maybe it’s not a big deal to most people if the reading in HA is (up to) 5 mins offset to the real reading but the perfectionist in me doesn’t like it at all.

Glen

PeterH24x7 · January 5, 2022, 3:18am

Hi Glenn - the raw data is intersting, but not very useful I’m afraid; it’s also very large and not something you would want to get every 5 minutes. Your time attribute above (in seconds from 1970 converted with [‘time’/24/3600+25569] to days from 1990 - e.g. 1641254100 = 3-Jan-22,11:55 PM). The most recent data you can get is up to 11:55pm for the previous day. It would be good if the regular 5 minute data was not just generated and consumed energy, but also the other SA channels which for me additionally includes hot-water, heating and EV circuits. Maybe if we get a good HA demo up and running we could lobby SA to expand their API to optimally support HA Energy management - including a get of the the raw data that’s in nearer real time and less voluminous.

I wrote some phython code to extract the raw data 5 minute ‘energy’ attribute for each channel and this screen shot is what the tail end of the data looks like in Excel. The numbers exactly tally with the data in the SA web-app.
Capture

I’ve also previously written some test code in python (Idle Python) for the regular 5 minute data (not raw) to get the data then output it to a CSV file with the calcs for import and export energy. See below for your interest - the import and export variables are e_i and e_e. Something similar is needed for HA - together with the subsequent sum of the daily total import and total export. Again these numbers tally with the SA web-app.

#
# Get site data from solar analytics and output the time and energy generated and consumed to a csv file.
# Output the most recent 5 minute values and display the time.
#
# Note: individual channel data not available.
#

import requests
from requests.auth import HTTPBasicAuth

import ast

from datetime import time, datetime, timedelta

# Update these fields for the specific account
username = "[email protected]"
password = "yourpassword"
site_id = "XXXX"

url = "https://portal.solaranalytics.com.au"

urlx_token = "/api/v3/token"
urlx_sitelist = "/api/v3/site_list"
urlx_sitestatus = "/api/v3/site_status/"
urlx_sitedata = "/api/v2/site_data/"

query_min = "&gran=minute" # ** get 5 minute cons/gen data for given date including today **

output_file = "url-file-output18.csv"

myrequest = ""

print("--start--")

mynow = datetime.now()
print(mynow)

# work out today's date and assemble the query for solar analytics
mytoday = mynow.date().isoformat().replace('-', '')
print("Today's date: ", mytoday)
print()

query_date = "?tstart=" + mytoday + "&tend=" + mytoday


#
# Get raw 5 minute detailed data from solar analytics
#
myrequest = requests.get(url+urlx_sitedata+site_id+query_date+query_min, auth=HTTPBasicAuth(username, password))
print('Response code ' + str(myrequest.status_code))

if myrequest.status_code == 200:
      print('Get Site Data Successful')
      print()

#
# Process the data
#

myrequest1 = myrequest.text

# print(len(myrequest1))
# print(myrequest1)

# Remove null data references - causes errors with integer literal reads.
myrequest2 = myrequest1.replace('null', '-1')

myresponse = ast.literal_eval(myrequest2)

print(myresponse)

count = 0

t_s = "" # date-time
e_c = 0  # energy consumed
e_g = 0  # energy generated
e_i = 0  # energy imported - calculated
e_e = 0  # energy exported - calculated


print("start loop")

with open(output_file, "w") as f:
    print(mynow, file=f)
    print("count, date-time, energy_consumed, energy_generated, energy_imported, energy_exported", file=f)  

    while True:
            if myresponse['data'][count]['energy_consumed'] != -1:
                d_t = myresponse['data'][count]['t_stamp']
                e_c = myresponse['data'][count]['energy_consumed']
                e_g = myresponse['data'][count]['energy_generated']

                if e_c >= e_g:
                      e_i = e_c - e_g
                      e_e = 0
                if e_g > e_c:
                      e_i = 0
                      e_e = e_g - e_c

                print(count, ", ", d_t, ", ", e_c, ", ", e_g, ", ", e_i, ", ", e_e, file=f)

                count += 1 

            else: break   
            
    print('Done!', file=f)

print("end loop")    
print()

if count > 0:
    print("Latest energy read: ", mynow)
    print("    Date-time = ", d_t)
    print("    Consumed = ", e_c)
    print("    Generated = ", e_g)
    print("    Imported = ", e_i)
    print("    Exported = ", e_e)
else:
    print("Error: No Data")

print()

endnow = datetime.now()
deltaend = endnow - mynow
print(endnow.time(), " (runtime = ", deltaend, ")")
print("--end--")

TheOtherGlen · January 5, 2022, 4:10am

The most recent data you can get is up to 11:55pm for the previous day.

You’re right. I hadn’t noticed that. So, yeah, it’s useless.

It would be good if the regular 5 minute data was not just generated and consumed energy, but also the other SA channels which for me additionally includes hot-water, heating and EV circuits.

Agreed, But I’d go further and say that the ideal would be a service that includes just the latest values for each circuit. These values could include point-in-time values like power, amperage, voltage, etc as well as the values that are accumulated over a time period (previous 5 mins, hour, day) like energy consumed, generated, exported to grid, imported from grid.
Actually, thinking about it further, these should probably be separate services - current state vs history.

Maybe if we get a good HA demo up and running we could lobby SA to expand their API to optimally support HA Energy management - including a get of the the raw data that’s in nearer real time and less voluminous.

I’m on board!

ddwdiot · January 5, 2022, 4:59am

I’m watching this with interest… having Solar Analytics and recently switching from OpenHab to Home Assistant. I’d like to contribute something positive but you two are far ahead of me in terms of technical skills so I’m likely to be a hanger on for a while at least… happy to help with any troubleshooting I can if needed though… once I get the API reporting.

I can get my hourly (and more detailed if needed) stats in Postman. But that’s using a bearer token rather than a username and password… using that same token gives no results in Home Assistant using your code above. Will spend a bit of time tonight trying my user name and password.

I was in touch with SA recently… was having problems accessing their API documents and it turned out that they had the wrong publically available password on their website (they fixed it while I was on the phone with them). I asked them about any plans they have to allow easy integration with home assistant or other IOT solutions. The person on the end of the phone politely advised that she would pass my “suggestion” on. I guessed that it might not be in their financial interest… but not sure how these things work.

Thanks for all the work you’re doing on this.

PeterH24x7 · January 5, 2022, 5:21am

Hello @ddwdiot, I’m glad you can join us and all contributions are welcomed - starting with your update from Solar Analytics. I think we’re okay from a basic HA Energy Management perspective, but with some addtional API calls it could be much better and more efficient. I think it should be in their interest as there are otherwise alternatives already on the market that can integrate with HA. A friend of mine has a SA 3-channel and has recently installed the following - with 12 channels - and it works with HA Energy Management.

PeterH24x7 · January 5, 2022, 10:53pm

Maybe a rookie question, why do the main accumulation variables sa_today_import and sa_today_export get reset to zero for each execution of the for loop?

Here’s the template code just for a limited time span (108 = 12x9 = 9am, 144 = 12pm) this morning:

{% set sa_today_import = 0 %}
{% set sa_today_export = 0 %}
{% set count = 0 %}
{% for count in range(108,144) %}
  {% if (states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed) == None %}
    {% set e_c = 0 %}
    {% set e_g = 0 %}
  {% else %}
    {% set e_c = states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed %}
    {% set e_g = states.sensor.sa_data_by_5min.attributes.data[count].energy_generated %}
  {% endif %}
  {{ count }}, {{ states.sensor.sa_data_by_5min.attributes.data[count].t_stamp }}
  Current interval {{ e_c }}, {{ e_g }}
  Cumulative import/export totals: {{ sa_today_import }}, {{ sa_today_export }}
  {% if e_c > e_g %}
    consumption is greater
    {% set e_i = (e_c - e_g) %}
    {% set e_e = 0 %}
    Net energy imported {{ e_i }}
    {% set sa_today_import = (sa_today_import + e_i) %}
  {% elif e_c < e_g %}
    generation is greater
    {% set e_e = (e_g - e_c) %}
    {% set e_i = 0 %}
    Net energy exported {{ e_e }}
    {% set sa_today_export = (sa_today_export + e_e) %}
  {% else %}
    equal
  {% endif %}
  New cumulative import/export totals: {{ sa_today_import }} and {{ sa_today_export }}
{% endfor %}

Here are the results::
(Note: the reason why consumption is ~= generation is because my EV is charging and taking all excess solar that would otherwise be exported and putting it into the car)

108, 2022-01-06 09:00:00
  Current interval 190, 161
  Cumulative import/export totals: 0, 0
  
    consumption is greater
    
    
    Net energy imported 29
    
  
  New cumulative import/export totals: 29 and 0

  
    
    
  
  109, 2022-01-06 09:05:00
  Current interval 263, 267
  Cumulative import/export totals: 0, 0   <<<< WHY ARE THE TOTALS BACK TO ZERO AGAIN???
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 0 and 4    <<<<< THIS SHOULD BE 29 and 4 ???

  
    
    
  
  110, 2022-01-06 09:10:00
  Current interval 226, 232
  Cumulative import/export totals: 0, 0    <<<< WHY ARE THE TOTALS BACK TO ZERO AGAIN???
  
    generation is greater
    
    
    Net energy exported 6
    
  
  New cumulative import/export totals: 0 and 6    <<<<< THIS SHOULD BE 29 and 10 ???

  
    
    
  
  111, 2022-01-06 09:15:00
  Current interval 161, 163
  Cumulative import/export totals: 0, 0    <<<< WHY ARE THE TOTALS BACK TO ZERO AGAIN???
  
    generation is greater
    
    
    Net energy exported 2
    
  
  New cumulative import/export totals: 0 and 2     <<<<< THIS SHOULD BE 29 and 12 ???

  
    
    
  
  112, 2022-01-06 09:20:00
  Current interval 169, 182
  Cumulative import/export totals: 0, 0
  
    generation is greater
    
    
    Net energy exported 13
    
  
  New cumulative import/export totals: 0 and 13

  
    
    
  
  113, 2022-01-06 09:25:00
  Current interval 193, 187
  Cumulative import/export totals: 0, 0
  
    consumption is greater
    
    
    Net energy imported 6
    
  
  New cumulative import/export totals: 6 and 0

  
    
    
  
  114, 2022-01-06 09:30:00
  Current interval 199, 203
  Cumulative import/export totals: 0, 0
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 0 and 4

  
    
    
  
  115, 2022-01-06 09:35:00
  Current interval 288, 291
  Cumulative import/export totals: 0, 0
  
    generation is greater
    
    
    Net energy exported 3
    
  
  New cumulative import/export totals: 0 and 3

  
    
    
  
  116, 2022-01-06 09:40:00
  Current interval 215, 218
  Cumulative import/export totals: 0, 0
  
    generation is greater
    
    
    Net energy exported 3
    
  
  New cumulative import/export totals: 0 and 3

  
    
    
  
  117, 2022-01-06 09:45:00
  Current interval 0, 0
  Cumulative import/export totals: 0, 0
  
    equal
  
  New cumulative import/export totals: 0 and 0

  
    
    
  
  118, 2022-01-06 09:50:00
  Current interval 0, 0
  Cumulative import/export totals: 0, 0
  
    equal
  
  New cumulative import/export totals: 0 and 0

TheOtherGlen · January 5, 2022, 11:52pm

Hi @PeterH24x7

I’ve been struggling to find time to play with this lately but I think the quick answer to your question is because of Jinja scoping:
http://jinja.pocoo.org/docs/2.10/templates/#assignments

Try using a namespace as they do in the example in that doco.

Glen

PeterH24x7 · January 6, 2022, 12:35am

Thank you @TheOtherGlen. That was the trick and now working as expected. See below for code and results.

{% set sa_ns = namespace() %}
{% set sa_ns.sa_today_import = 0 %}
{% set sa_ns.sa_today_export = 0 %}
{% set count = 0 %}
{% for count in range(108,144) %}
  {% if (states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed) == None %}
    {% set e_c = 0 %}
    {% set e_g = 0 %}
  {% else %}
    {% set e_c = states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed %}
    {% set e_g = states.sensor.sa_data_by_5min.attributes.data[count].energy_generated %}
  {% endif %}
  {{ count }}, {{ states.sensor.sa_data_by_5min.attributes.data[count].t_stamp }}
  Current interval {{ e_c }}, {{ e_g }}
  Cumulative import/export totals: {{ sa_ns.sa_today_import }}, {{ sa_ns.sa_today_export }}
  {% if e_c > e_g %}
    consumption is greater
    {% set e_i = (e_c - e_g) %}
    {% set e_e = 0 %}
    Net energy imported {{ e_i }}
    {% set sa_ns.sa_today_import = (sa_ns.sa_today_import + e_i) %}
  {% elif e_c < e_g %}
    generation is greater
    {% set e_e = (e_g - e_c) %}
    {% set e_i = 0 %}
    Net energy exported {{ e_e }}
    {% set sa_ns.sa_today_export = (sa_ns.sa_today_export + e_e) %}
  {% else %}
    equal
  {% endif %}
  New cumulative import/export totals: {{ sa_ns.sa_today_import }} and {{ sa_ns.sa_today_export }}
{% endfor %}

108, 2022-01-06 09:00:00
  Current interval 190, 161
  Cumulative import/export totals: 0, 0
  
    consumption is greater
    
    
    Net energy imported 29
    
  
  New cumulative import/export totals: 29 and 0

  
    
    
  
  109, 2022-01-06 09:05:00
  Current interval 263, 267
  Cumulative import/export totals: 29, 0
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 29 and 4

  
    
    
  
  110, 2022-01-06 09:10:00
  Current interval 226, 232
  Cumulative import/export totals: 29, 4
  
    generation is greater
    
    
    Net energy exported 6
    
  
  New cumulative import/export totals: 29 and 10

  
    
    
  
  111, 2022-01-06 09:15:00
  Current interval 161, 163
  Cumulative import/export totals: 29, 10
  
    generation is greater
    
    
    Net energy exported 2
    
  
  New cumulative import/export totals: 29 and 12

  
    
    
  
  112, 2022-01-06 09:20:00
  Current interval 169, 182
  Cumulative import/export totals: 29, 12
  
    generation is greater
    
    
    Net energy exported 13
    
  
  New cumulative import/export totals: 29 and 25

  
    
    
  
  113, 2022-01-06 09:25:00
  Current interval 193, 187
  Cumulative import/export totals: 29, 25
  
    consumption is greater
    
    
    Net energy imported 6
    
  
  New cumulative import/export totals: 35 and 25

  
    
    
  
  114, 2022-01-06 09:30:00
  Current interval 199, 203
  Cumulative import/export totals: 35, 25
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 29

  
    
    
  
  115, 2022-01-06 09:35:00
  Current interval 288, 291
  Cumulative import/export totals: 35, 29
  
    generation is greater
    
    
    Net energy exported 3
    
  
  New cumulative import/export totals: 35 and 32

  
    
    
  
  116, 2022-01-06 09:40:00
  Current interval 215, 218
  Cumulative import/export totals: 35, 32
  
    generation is greater
    
    
    Net energy exported 3
    
  
  New cumulative import/export totals: 35 and 35

  
    
    
  
  117, 2022-01-06 09:45:00
  Current interval 190, 193
  Cumulative import/export totals: 35, 35
  
    generation is greater
    
    
    Net energy exported 3
    
  
  New cumulative import/export totals: 35 and 38

  
    
    
  
  118, 2022-01-06 09:50:00
  Current interval 206, 211
  Cumulative import/export totals: 35, 38
  
    generation is greater
    
    
    Net energy exported 5
    
  
  New cumulative import/export totals: 35 and 43

  
    
    
  
  119, 2022-01-06 09:55:00
  Current interval 200, 204
  Cumulative import/export totals: 35, 43
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 47

  
    
    
  
  120, 2022-01-06 10:00:00
  Current interval 215, 219
  Cumulative import/export totals: 35, 47
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 51

  
    
    
  
  121, 2022-01-06 10:05:00
  Current interval 221, 225
  Cumulative import/export totals: 35, 51
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 55

  
    
    
  
  122, 2022-01-06 10:10:00
  Current interval 188, 192
  Cumulative import/export totals: 35, 55
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 59

  
    
    
  
  123, 2022-01-06 10:15:00
  Current interval 214, 218
  Cumulative import/export totals: 35, 59
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 63

  
    
    
  
  124, 2022-01-06 10:20:00
  Current interval 212, 216
  Cumulative import/export totals: 35, 63
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 67

  
    
    
  
  125, 2022-01-06 10:25:00
  Current interval 223, 227
  Cumulative import/export totals: 35, 67
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 35 and 71

  
    
    
  
  126, 2022-01-06 10:30:00
  Current interval 238, 227
  Cumulative import/export totals: 35, 71
  
    consumption is greater
    
    
    Net energy imported 11
    
  
  New cumulative import/export totals: 46 and 71

  
    
    
  
  127, 2022-01-06 10:35:00
  Current interval 264, 260
  Cumulative import/export totals: 46, 71
  
    consumption is greater
    
    
    Net energy imported 4
    
  
  New cumulative import/export totals: 50 and 71

  
    
    
  
  128, 2022-01-06 10:40:00
  Current interval 380, 381
  Cumulative import/export totals: 50, 71
  
    generation is greater
    
    
    Net energy exported 1
    
  
  New cumulative import/export totals: 50 and 72

  
    
    
  
  129, 2022-01-06 10:45:00
  Current interval 355, 368
  Cumulative import/export totals: 50, 72
  
    generation is greater
    
    
    Net energy exported 13
    
  
  New cumulative import/export totals: 50 and 85

  
    
    
  
  130, 2022-01-06 10:50:00
  Current interval 361, 365
  Cumulative import/export totals: 50, 85
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 89

  
    
    
  
  131, 2022-01-06 10:55:00
  Current interval 317, 319
  Cumulative import/export totals: 50, 89
  
    generation is greater
    
    
    Net energy exported 2
    
  
  New cumulative import/export totals: 50 and 91

  
    
    
  
  132, 2022-01-06 11:00:00
  Current interval 224, 228
  Cumulative import/export totals: 50, 91
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 95

  
    
    
  
  133, 2022-01-06 11:05:00
  Current interval 297, 301
  Cumulative import/export totals: 50, 95
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 99

  
    
    
  
  134, 2022-01-06 11:10:00
  Current interval 234, 238
  Cumulative import/export totals: 50, 99
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 103

  
    
    
  
  135, 2022-01-06 11:15:00
  Current interval 246, 250
  Cumulative import/export totals: 50, 103
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 107

  
    
    
  
  136, 2022-01-06 11:20:00
  Current interval 230, 234
  Cumulative import/export totals: 50, 107
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 111

  
    
    
  
  137, 2022-01-06 11:25:00
  Current interval 230, 234
  Cumulative import/export totals: 50, 111
  
    generation is greater
    
    
    Net energy exported 4
    
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  138, 2022-01-06 11:30:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  139, 2022-01-06 11:35:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  140, 2022-01-06 11:40:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  141, 2022-01-06 11:45:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  142, 2022-01-06 11:50:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

  
    
    
  
  143, 2022-01-06 11:55:00
  Current interval 0, 0
  Cumulative import/export totals: 50, 115
  
    equal
  
  New cumulative import/export totals: 50 and 115

TheOtherGlen · January 6, 2022, 2:40am

Hi @PeterH24x7

Try the following template in Developer Tools->Template:

{% set energy = namespace(exported=0, imported=0) %}

{% for sensor_data in states.sensor.sa_hour.attributes.data | rejectattr('energy_generated', 'equalto', None) if sensor_data.energy_generated > sensor_data.energy_consumed %}
{% set energy.exported = energy.exported + sensor_data.energy_generated - sensor_data.energy_consumed %}
{% endfor %}
Energy exported: {{ energy.exported }}

{% for sensor_data in states.sensor.sa_hour.attributes.data | rejectattr('energy_consumed', 'equalto', None) if sensor_data.energy_consumed > sensor_data.energy_generated %}
{% set energy.imported = energy.imported + sensor_data.energy_consumed - sensor_data.energy_generated %}
{% endfor %}
Energy imported: {{ energy.imported }}

The values that a calculated are close to the values reported in SA so I think they are accurate… or as accurate as I can possibly calculate with the hour granularity I’m currently using. Will try it with 5 minute granularity later.

Notes"

The reason I have two separate for loops is that I believe this is how I will do it in the two sensors (when I get around to it)
We could get rid of the for loops altogether and just use a {{…}} template if we can find a filter (eg. rejectattr or selectattr or something else) that can compare two attributes to each other… but I can live with the for loop above.

PeterH24x7 · January 6, 2022, 2:54am

Thanks Glen - am a bit ahead of you, but your code does look much more efficient and practical than mine so will review with interest.

In the mean time here’s my working code based on 5 minute data so the numbers match the SA web-app. It’s taken an hour or more to start appearing, but I finally have data shown in the HA Energy module dashboard. Woohooo!

# Solar Analytics - get detailed 5 minute data 
# Used to calculate today's total energy for consumed, generated, imported and exported energy for the HA Energy module
# Updated every 5 minutes
  - platform: rest
    name: sa_data_by_5min
    resource_template: 'https://portal.solaranalytics.com.au/api/v2/site_data/36303?gran=minute&tstart={{ now().strftime("%Y%m%d") }}&tend={{ now().strftime("%Y%m%d") }}'   
    username: !secret sa_username
    password: !secret sa_password
    authentication: basic
    value_template: "{{ now() }}"
    json_attributes:
      - "data"
    state_class: measurement
    device_class: energy
    scan_interval: 300
    force_update: true

  - platform: template
    sensors:
      sa_todays_energy_consumed_total:
        friendly_name: Total Energy Consumed
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_consumed', 'equalto', None) | sum(attribute='energy_consumed') }}"
        icon_template: mdi:home-lightning-bolt-outline
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
      sa_todays_energy_generated_total:
        friendly_name: Total Energy Generated
        unit_of_measurement: "Wh"
        value_template: "{{ states.sensor.sa_data_by_5min.attributes.data | rejectattr('energy_generated', 'equalto', None) | sum(attribute='energy_generated') }}"
        icon_template: mdi:solar-power
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
      sa_todays_energy_imported:
        friendly_name: Total Energy Imported
        unit_of_measurement: "Wh"
        value_template: >
          {% set sa_ns = namespace() %}
          {% set sa_ns.sa_today_import = 0 %}
          {% for count in range(0,287) %}
            {% if (states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed) != None %}
              {% set e_c = states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed %}
              {% set e_g = states.sensor.sa_data_by_5min.attributes.data[count].energy_generated %}
              {% if e_g < e_c %}
                {% set sa_ns.sa_today_import = (sa_ns.sa_today_import + e_c - e_g) %}
              {% endif %}
            {% endif %}
          {% endfor %}
          {{ sa_ns.sa_today_import }}
        icon_template: mdi:transmission-tower-export
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing
          
      sa_todays_energy_exported:
        friendly_name: Total Energy Exported
        unit_of_measurement: "Wh"
        value_template: >
          {% set sa_ns = namespace() %}
          {% set sa_ns.sa_today_export = 0 %}
          {% for count in range(0,287) %}
            {% if (states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed) != None %}
              {% set e_c = states.sensor.sa_data_by_5min.attributes.data[count].energy_consumed %}
              {% set e_g = states.sensor.sa_data_by_5min.attributes.data[count].energy_generated %}
              {% if e_g > e_c %}
                {% set sa_ns.sa_today_export = (sa_ns.sa_today_export + e_g - e_c) %}
              {% endif %}
            {% endif %}
          {% endfor %}
          {{ sa_ns.sa_today_export }}
        icon_template: mdi:transmission-tower-import
        device_class: "energy"
        attribute_templates:
          state_class: total_increasing