Enphase Envoy - D7 firmware with JWT - A Different Approach

yesterdayshero · August 22, 2023, 10:55pm

I’ve done a bit of testing but am still getting some issues.

I uninstalled the Enphase integration and removed the HACS repository. Entities were still dropping to 0 due to what I believe to be the rest error I pasted above.

I tried changing the scan interval to 300 thinking that hitting the URL less frequently might help, but was still getting the same results. At the moment I’m sticking to 60.

I also tried pulling data from my inverter API so I could track production from each inverter. I made this scan every 300 seconds, but have a similar issue here with them becoming unavailable sporadically.

Struggling to find any consistency, other than on some scans, the rest request fails, then on others it works. It’s also not my wifi dropping. @Steve61 I think this is different to the issue with the main integration, as it’s not happening at a consistent time.

I’ll keep digging, but let me know if you can think of something else I should try.

del13r · August 23, 2023, 12:44am

Is this in parallel to the rest sensor?
Is this via the native integration/hacs/other?

The reason why I am asking is that we want to eliminate as many possible causes of your drop outs.
Having multiple integrations polling the envoy at the same time is what we are trying to eliminate.
Can you also temporarily try 15 second scan interval?
The reason why is because if other integrations are also contacting the envoy, setting the scan interval to any multiple of 60 seconds (300 / 5 = 60) may cause both polls to occur at the same time at the start of each minute and they both might fail due to the lag that the non rest integrations create by polling production.json
If you set it to 15 seconds, then it’s possible that 3/4 polls per minute might succeed, if my double knockout theory is correct.

del13r · August 23, 2023, 1:11am

You can test this yourself outside of home assistant by using browser tools (f12) in your browser.

Go to https://envoy.local/ivp/meters/readings

It will then show you a screen where you can either:

login to enlighten and generate a temporary 12 hour installer token.
paste in your existing 12 month owner token that is in secrets.yaml

Once you do either step, you should be able to see the json output of https://envoy.local/ivp/meters/readings.
Then you can press f12 to open dev tools and then press ctrl + R (refresh) to see how long each request takes in milliseconds or seconds or even if it times out using browser tools.

You can also tick preserve log and then it will show how long each refresh request took in a table at the bottom. (203ms average across 11 refreshes)

compare that with https://envoy.local/production.json and you will definitely see a slower average for the older API endpoint. (780ms average across 11 refreshes)

This method is pretty much the same thing that the rest sensor does, instead you are triggering the refresh manually whenever you press ctrl + R to refresh the page much the same as the scan interval setting would.

yesterdayshero · August 23, 2023, 11:00am

Is this in parallel to the rest sensor?
Is this via the native integration/hacs/other?

@del13r the issue is using the rest sensors. I essentially replicated your setup l, but also for the inverters. I uninstalled the native integration. So shouldn’t be running in parallel.

I’ve changed the scan for the main API to 15 so I’ll see how that goes. But I was actually getting this issue before I had added the inverter requests, so I’m not to sure that’s the reason.

I’ll also have a play around in browser.

del13r · August 23, 2023, 12:01pm

i would also be interested in your wifi signal strength that the envoy is reporting via https://envoy.local/home.json

As a test, I successfully setup the following sensor which asks for the wifi strength every 5 mins.

rest:
  - headers:
      Authorization: !secret enphase_api
    verify_ssl: False
    scan_interval: 300
    resource: https://envoy.local/home.json
    sensor:
      - name: "envoy wifi strength"
        value_template: "{{value_json.network.interfaces[1].signal_strength }}"

2/5 signal strength = 40% (not very good).

Example of it graphed.

masterfink · August 23, 2023, 7:11pm

I’m also in this exact same situation, cannot figure out how to properly calculate energy import and export. My template for calculating import and export does not work, it never shows any value for import, even when I’m importing the grid. I’m reaching out to my installer to see if I can get my switched away from Load Only, so we’ll see what they say, but would love to have this working in case it’s not possible.

jaaem · August 23, 2023, 10:29pm

You need a Riemann integral helper function to integrate the power import/export sensors. https://www.home-assistant.io/integrations/integration/ That is what I do now.

I am working on making hybrid integration, that gets power locally from the envoy using info from this thread, but gets the energy values from the cloud. Might be noobie-proof method. The cloud seems to know everything, but the envoy does not report it locally!

del13r · August 23, 2023, 11:01pm

Ok, let me try tackle this issue for you.
I cannot guarantee the accuracy as my setup is not the same as yours, so sorry in advance if my solution is not perfect.

Step 1, How can I see what my system is currently setup as?
Go to https://envoy.local/ivp/meters/
look at the value of [1].measurementType
value: net-consumption = Load with solar production.
value: total-consumption = Load only

Step 2, show the difference between load only and net.
This is where things differ.
View the value of [1].activePower at the URL https://envoy.local/ivp/meters/readings

When set to total-consumption/(load only), this value will only ever be positive as it represents what the appliances in house are demanding/consuming.
When set to net-consumption/(load with solar production), you will either see a negative value to indicate power grid export or a positive value to indicate grid import as this value represents what is occurring at the utility/grid/meter.

Example of a user with total-consumption/(load only)

Step 3, what can we calculate with load only?

Source: /ivp/meters/readings	Description	W	kW
value_json[0].activePower	Power Production	2000	2.0
value_json[1].activePower	Power Consumption	1337.395	1.33
template sensor calculation	Power Net (Calculated) (consumption-production)	-662.605	-0.66
template sensor calculation	Power Export (Calculated) (production-consumption)	662.605	0.66
template sensor calculation	Power Import (Calculated) (consumption-production)	0	0

The problem with load only setting is that the value for [1].actEnergyRcvd is empty at https://envoy.local/ivp/meters/readings.

Example of a user with total-consumption/(load only)

The absense of that value causes the energy calculations to no longer be possible.

Source: /ivp/meters/readings	Description	Wh	MWh
value_json[0].actEnergyDlvd	Energy Production	609552.964	0.6
value_json[1].actEnergyDlvd	Energy Consumption	447569.605	0.44
template sensor calculation	Energy Export (Calculated)	not possible
template sensor calculation	Energy Net (Calculated)	not possible
template sensor calculation	Energy Import (Calculated)	not possible

Step 4, How do we fill the gap for load only users?
Create the following sensors for Load Only installations.

rest:
  - headers:
      Authorization: !secret enphase_api
    verify_ssl: False
    scan_interval: 15
    resource: https://envoy.local/ivp/meters/readings
    sensor:

      - name: "Power Production"
        value_template: >
            {% set value = value_json[0].activePower | int(0) %}
            {% if value  <= 5 -%}
                0
            {% elif is_state('sun.sun','below_horizon') %}
                0
            {%- else -%}
                {{ value }}
            {%- endif %}
        device_class: power
        unit_of_measurement: W
        state_class: measurement
        icon: mdi:solar-panel

      - name: "Power Consumption"
        value_template: "{{ value_json[1].activePower | int(0) }}"
        state_class: measurement
        device_class: power
        unit_of_measurement: W
        icon: mdi:home-lightning-bolt

      - name: "Energy Production"
        value_template: "{{ (value_json[0].actEnergyDlvd / 1000 | float(0)) | round(2) }}"
        device_class: energy
        unit_of_measurement: kWh
        state_class: total_increasing
        icon: mdi:solar-panel

      - name: "Energy Consumption"
        value_template: "{{ (value_json[1].actEnergyDlvd / 1000 | float(0)) | round(2) }}"
        device_class: energy
        unit_of_measurement: kWh
        state_class: total_increasing
        icon: mdi:home-lightning-bolt

Then i added these template sensors to calculate power export and power import

template:
  - sensor:
        name: Power Export
        state_class: measurement
        icon: mdi:transmission-tower
        unit_of_measurement: W
        device_class: power
        state : >
          {{ [0, states('sensor.power_production') | int(0) - states('sensor.power_consumption') | int(0) ] | max }}
          
  - sensor:
        name: Power Import
        state_class: measurement
        icon: mdi:transmission-tower
        unit_of_measurement: W
        device_class: power
        state : >
          {{ [0, states('sensor.power_consumption') | int(0) - states('sensor.power_production') | int(0) ] | max }}

  - sensor:
        name: Power Net
        state_class: measurement
        icon: mdi:transmission-tower
        unit_of_measurement: W
        device_class: power
        state : >
          {{ states('sensor.power_consumption') | int(0) - states('sensor.power_production') | int(0)  }}

And then finally, add these 2 sensors to convert:
power in W
to
energy in kWh
using https://www.home-assistant.io/integrations/integration/

sensor:
  - platform: integration
    name: Energy Import
    source: sensor.power_import
    unit_prefix: k
    unit_time: h
    method: left
    
  - platform: integration
    name: Energy Export
    source: sensor.power_export
    unit_prefix: k
    unit_time: h
    method: left

I am unable to test this myself, so feel free to give it a go and let me know.

yesterdayshero · August 23, 2023, 11:01pm

@del13r I have a signal strength of 3. It should be fairly solid. I have a unifi setup and the router is on the other side of the same wall of the house.

I’ve also commented out my inverter rest calls. So basically my setup at the moment is the same as yours. I’ve still noticed some dropped calls. But I’ll track it for a while and see how it goes.

del13r · August 24, 2023, 2:41am

Process of elimination: Try eliminate DNS issues as a possible cause by using the IPv4 address (192.x.x.x) instead of envoy.local

yesterdayshero · August 24, 2023, 8:54am

Ok, so I’ve noticed that even at 15 second intervals. I get about 2 to 3 drops an hour. Not too bad, and definitely an improvement on before. So there’s definitely something to there being an issue if there’s too many concurrent calls. Like when I had the original integration running side by side.

I’ve now swapped over to the IP address and will see how this goes.

yesterdayshero · August 24, 2023, 9:02am

Also played around in browser and definitely get the quicker responses using the URL you suggested over the production .json.

del13r · August 24, 2023, 6:43pm

Yep, that’s really great results. Thanks for taking the time to perform these tests.

yesterdayshero · August 24, 2023, 9:48pm

Process of elimination: Try eliminate DNS issues as a possible cause by using the IPv4 address (192.x.x.x) instead of envoy.local

@del13r ok I think I have some good results to share finally. Since changing to an IP address instead of the envoy.local url, I’ve had no dropped rest calls. So it must have something to do with hitting the envoy.local link.

I’ve been able to turn on the inverter rest calls as well and both calls are working without issues so far over about a 12 hour period.

Thanks so much for helping work it out!

del13r · August 24, 2023, 9:59pm

Ah that’s awesome news. Go team.

jaaem · August 25, 2023, 9:27pm

Not to add another option to get enphase working, but I took the great work of @del13r and made a custom integration. Been working well for a couple days, but probably still has errors. This integration only works for people setup in “load only” or “total-consumption” mode. I will probably try to make it support people with “Load with Solar Production” in the future.

https://github.com/jdeath/enphase_local

Features: Uses endpoints @del13r found to get power locally from the envoy. It gets energy values from enphase cloud, since my system does not report this locally. Avoids needing Reiman sum integrals or any manual sensor setup. Sensors can directly be used in the energy dashboard. Takes username/password so do not need to figure out your token, but do need your serial and siteid. Feel free to submit PRs to add capability.

del13r · August 25, 2023, 10:33pm

I notice your documentation says that it will poll enphase cloud api every 10 mins.

I might be on the wrong track here, but the free developer account only gets 1000 hits per month.

My calculations are about 44 mins per poll to stay under the limit.

Hits
12000.000	365.000	Days/Year
1000.000	30.417	Days/Month
230.769	7.019	Days/Week
32.877	1.000	Day
1.370	1.000	Hour
1.005	44.000	Minutes
0.023	1.000	Minute

jaaem · August 25, 2023, 11:33pm

This integration does not use the official API… No developer account needed. I’ll see if it gets limited (but I doubt it), I have not tested it for more than a day!

del13r · August 25, 2023, 11:33pm

Ah yes. I just noticed this in your code


self.url = "https://enlighten.enphaseenergy.com/pv/systems/" + self.siteID + "/today"

Steve61 · August 26, 2023, 5:21am

@del13r
I had an issue using “INT” in the template sensors:

state : >
          {{ [0, states('sensor.power_production') | int(0) - states('sensor.power_consumption') | int(0) ] | max }}

would throw errors in child sensors, but if I changed it to “float” they disappeared. So:

state : >
          {{ [0, states('sensor.power_production') | float(0) - states('sensor.power_consumption') | float(0) ] | max }}

worked without issue.

I don’t know enough about YAML functions and the only thing I could see was in my version of the source - production and consumption sensors, I converted everything to kW by adding “/1000”.
{% set value = (value_json[0].activePower /1000 | int(0))

I’m not sure if this affects accuracy. Thoughts???