Enphase Envoy - Current state of things

If you are an Enphase customer, you are likely aware that there have been some changes to the Envoy, which is where we gather our data - this thread is long and covers a LOT https://support.enphase.com/s/question/0D53m00006ySLuRCAW/unimpressed-with-loss-of-local-api-connectivity-to-envoys

TL;DR Envoys that are upgraded to firmware D7.x and later no longer have unauthenticated access, default credentials, or an easily crackable installer login - instead, you need to grab a JWT from the enlighten site. So while the good news is that we do still have local access, the bad news is that the token acquisition process is sketchy at best.

I wanted to start this thread as a place to gather thoughts, clarify issues that everyone is having, and pull together a list of issues and feature requests. For clarification, I am currently on Envoy firmware D7.0.107

A special thanks to all of those that have already put in tons of time and effort into the integrations - gtdiehl, jesserizzo, DanBeard, briancmpbll, edasque and numerous others that have contributed, I apologize, I know I’ve missed lots of you.

Token Acquisition Issues

Right now, most of the repos use the following process to gather the token:
httpx session out to either https://entrez.enphaseenergy.com/login or https://enlighten.enphaseenergy.com >> use englighten web credentials to login >> take the session cookie, username and envoy serial number to request the JWT from https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num={} >> use the JWT as bearer auth to the local envoy API endpoints

There is frequently an issue with the 2nd step which results in a 401 unauthorized response from the enlighten service. This issue also frequently occurs with web logins to the enlighten web app, so I believe the issue to be partly (if not completely) on their side. There is no single programatic way that I am aware of to just hit a single endpoint for a token and be done.

It appears that there used to be a JWT validation endpoint on the envoy that needed to be accessed before other API endpoints would respond, but this no longer appears to be the case…maybe this is an older D7 firmware requirement?? Curious if anyone has background here (thanks to Erik for clarification: https://github.com/edasque/enphaseLocalToInfluxDB#authorization-flow)

On my envoy, it appears that a session cookie is generated automatically, without having to first post the JWT to the validation endpoint - the cookie was not required for the GET, only the JWT auth, with the cookie coming after the connection.

The token is valid for 6 months, but the expiration is coded in the exp field of the JWT - needs to be renewed before it expires.

Useful info

Known endpoints for the local Envoy:

https://envoy/backbone/application.js?version=07.0.107 Javascript of the app running (for your version, replace the D with a zero)
https://enlighten.enphaseenergy.com/ Enlighten logon page, will respond with a logon cookie; requires scraping page for login form
https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num={} Returns entrez token as JSON; session must be logged in first, include _enlighten_4_session cookie

Enphase Energy - Enlighten | Sign in to Enlighten Programatic login page, returns login cookie; seems to be more reliable
https://entrez.enphaseenergy.com/tokens Programatic token page, returns JWT as string; takes cookie session, envoy serial number, and enlighten login

These are the two endpoints that I am using (as of a few days ago) in my repo https://github.com/jrutski/home_assistant_envoy_d7_fw , that seem to work a bit more reliably without presenting a 401 error.

Known Issues and Outstanding Feature Requests

  • The data for the energy dashboard is not exported by the API, and needs to be added manually as a few calculated sensors. Not a huge issue, but I see this question come up a lot as new users start add the integration and the energy dashboard. Here are the sensors I use:
  - sensor:
        name: Grid Import Power
        state_class: measurement
        icon: mdi:transmission-tower
        unit_of_measurement: W
        device_class: power
        state: >
            {{ [0, states('sensor.envoy_xxxxxxxxxxxx_current_power_consumption') | int - states('sensor.envoy_xxxxxxxxxxxx_current_power_production') | int ] | max }}
  - sensor:
        name: Grid Export Power
        state_class: measurement
        icon: mdi:transmission-tower
        unit_of_measurement: W
        device_class: power
        state: >
            {{ [0, states('sensor.envoy_xxxxxxxxxxxx_current_power_production') | int - states('sensor.envoy_xxxxxxxxxxxx_current_power_consumption') | int ] | max }}

  • For the systems that have an Enpower smart switch (ATS) Grid Status needs to be added - it appears to be exposed both in the ensemble inventory endpoint (/ivp/ensemble/production.json) or the home endpoint, though I don’t think I’ve seen it being queried (/home.json)

While I think it would be nice to be able to automatically drop loads when the grid is down, I don’t think the current polling interval is short enough to prevent a shutdown. I had this happen the other week, with the dryer and A\C running, I exceeded the battery capacity while the solar micros were “rebooting”…there’s only 10 seconds of peak output.

  • Systems that include batteries do not appear to expose meters for in/out to the batteries, or at least not the newer encharge 3/10 systems. The /production.json endpoint does have SOME storage data, but only for the older AC Battery systems (thus the ‘acb’)

What does this mean? It means your consumption readings will be completely off - becasuse the battery is essentially a home load. So there is no way to tell what the actual loads in the house are vs charging and discharging the battery. I’ve dug through tons of the known endpoints and have yet to figure out where this data might be from an encharge system.

As I mentioned I just wanted to get my thoughts in one place, and hopefully we can find some more answers. I’m happy to provide any API responses that anyone would like to see, or help in any way I can.

Thanks,
Jacob

3 Likes

Been digging through the APIs and I couldn’t figure out where the battery data comes from - when the battery is charging, the consumption CT will read both the home load and the battery charging load, even though it’s all coming from PV…the production and consumption will read roughly the same. None of the existing (known) APIs show ANY of this data, only the individual battery charge level for each of the encharge base units.

Then I went digging through the enlighten apps, and found the following in the enlighten manager:

So the data comes from the IQ8X-BAT micros and surprise surprise - these are NOT exposed in any of the existing APIs.

The search continues…

My system is not yet installed, so can’t help, but have you tried MITM the android app? If the cloud/local token is the same, emulating that login process (with the right user-agent, etc) might be a little more robust. Also, there is always the option to get some of the data via the cloud versus local. Looks like their free v4 API does not have much functionality, but if it the data is in the App, it should be available to get into home assistant.

I’ve been able to that for a simple cloud service in the past.

edit: was able to mitm the app and possibly figured out login process. Don’t yet have an account, so can’t take it further. Happy to PM anyone the code. The app actually first logs in with a default account, so I can reproduce that.

edit: found the jwt token. This method seems fool proof, not sure if it is the same jwt token, but seems to work. I PM the OP the code

Sorry for such a n00b question, but I just got my Envoy online the other day and started looking into things. In all your long post about the current state (which I highly appreciate), I noticed that you never mention the actual Enphase API documented here: The Enlighten Systems API

Is that not a good solution for what we’re trying to do with HA? I haven’t started digging in yet as I’m still reading about current state of things and what’s possible.

Thanks, especially for your great summary of the current state.

Most people here prefer local access, i.e. access the data from the Envoy device straight from Home Assistant rather than taking a detour via Enphase’s servers.

Got it, I wasn’t entirely sure if that API was through their servers or was somehow getting redirected to the device itself, or something.

And ya, I prefer local access too.

Thanks again for all the info.

1 Like

As @exxamalte points out, that API is their cloud API and the more we can keep this local the better…the data comes from the envoy itself, so it should be available local.

I might be willing to sign up for a free account, but:

  1. No
  2. Even the free account limits polling of the API, and I think it ends up being something like once every 15 minutes or so? No thanks
  3. I’m pretty sure that similar data is available to what is available from the envoy, just in a “more supported” means…so we still don’t have the data from the IQ8-BAT micros to define battery charging\discharging

It’s still very much a work in progress, especially if you have encharge storage.

Hi @jrutski,

Great write up of the current state of things.
Even though I am on V5 firmware and don’t need a token to access my envoy yet, I did some testing anyway. Here is what I found.

Step 1
I did some experimenting with postman and then cURL and found how to generate the session_id.

Login and generate session_id via cURL

curl https://enlighten.enphaseenergy.com/login/login.json \
-d 'user[email][email protected]&user[password]=MyPassword' \
| jq -r '.session_id'

Results

32alphanumericcharacterSession_ID

Step 2

cURL command that generates the JWT using the session ID from Step 1.

curl 'https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=myserialnumber' \
  -H 'cookie: _enlighten_4_session=32alphanumericcharacterSession_IDFromStep1'

Results

{"generation_time":1660805107,"token":"Stringof408Characters","expires_at":1692341107}

The token successfully generated the above json formatted output that contained a 408 character token.

I then decided to decode that JWt token at https://jwt.io/as well as convert the epoch timestamps at https://www.epochconverter.com/.
What I noticed is that the token is now a year in length.

I then thought I would see how home assistant might go about generating a token and came up with this solution as I found that the sensor state has a 255 character limit but the sensor attributes do not have the same limit. This is important as the token is 408 characters in length.

sensor:
  - platform: rest
    resource: https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=myserialnumber
    name: envoy JWT
    scan_interval: 604800
    json_attributes:
      - generation_time
      - token
      - expires_at
    value_template: 'ok'
    headers:
      cookie: _enlighten_4_session=32alphanumericcharacters

which then creates this sensor

I can then convert the epoch timestamps from the attributes as well as display the token using this code in Dev tools / template

Generated
{{states.sensor.envoy_jwt.attributes["generation_time"] | timestamp_local }}

Expires
{{states.sensor.envoy_jwt.attributes["expires_at"] | timestamp_local }}

Token
{{states.sensor.envoy_jwt.attributes["token"] }}

Results in Dev tools / template

image

That’s as far as I got with it.

I am not sure how long my Enphase enlighten session_id will last for but I learned some new things while testing this.

Hi @del13r this is great info. The current HACS repos out there do essentially this, and work pretty well. The problems have been on the enphase side, they would issue a token and for whatever reason, it would be invalid, or the token site would return a 401 even though the session had just been successfully created.

Hi @jrutski

Thanks.
I think I was doing this testing and writing this to help people who were affected by any issues, and myself, better understand what is going on under the hood.

The reason why I reverse engineered this is because I haven’t seen anyone break down how it works in plain english yet. Sure, there are existing python scripts, but if you don’t know python well, it’s not that easy to follow how it all works.

As I found, the JWT token validity is now a year in length so even if a user has to manually obtain a JWT token, by using the web browser gui or even by using cURL commands, that will give them 365 days of time to generate a new one.

If I read this correctly, the token site at https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=myserialnumber is generating 401 errors.

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

The HyperText Transfer Protocol (HTTP) 401 Unauthorized response status code indicates that the client request has not been completed because it lacks valid authentication credentials for the requested resource.

Given that we know the only inputs for that token generator are envoy serial number and session_id, it should be fairly easy to narrow down and eliminate which input is the problem.

Since your serial number is static and will never change, it can only be the session_id, which can unpredictably expire, that is causing the 401 error.

In F12 tools in chrome, I can see that the session_id has an expiry/max age of ‘Session’.

From https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#define_the_lifetime_of_a_cookie

Session cookies are deleted when the current session ends. The browser defines when the “current session” ends.

This means that the session_id expiry time is rather unpredictable and should be regenerated via a POST to https://enlighten.enphaseenergy.com/login/login.json before each JWT token generation attempt.

This is the cURL command I am using for testing

curl -v 'https://enlighten.enphaseenergy.com/entrez-auth-token?serial_num=myserialnumber' \
-H 'cookie: _enlighten_4_session=32alphanumericcharacters'

Result for correct serial number and session_id,

HTTP/2 200
status: 200 OK

Result for deliberately false serial number

HTTP/2 500
status: 500 Internal Server Error
{"message":"Invalid serial number(myserialnumber)"}

Result for a deliberately false session_id

HTTP/2 401
status: 401 Unauthorized
{"message":"You need to sign in first"}

Agreed.

Some of the other repos had been using a different method to gather the session token, which seems to be far less reliable, and I believe to have been causing a number of the errors seen. As you mention, https://enlighten.enphaseenergy.com/login/login.json is a programmatic method to gather a session, and seems to work much better - my repo uses this method, instead of the scraping the login page.

1 Like

Hi. Your integration works great on Enphase IQ Combiner 4 with firmware D7.0.153 (5957ef). Thanks! The fields in the config flow are not labeled for me, but I could look in the config_flow.py to figure out the field order.

1 Like

For the same firmware .153 I get an error as below.

2022-09-12 15:49:03.422 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] HTTP GET Attempt #3: https://192.168.1.21/auth/check_jwt: Header:{‘Authorization’: ‘Bearer eyJraTRkMGQtYmNiNC0yNDRmOThlZTE1bGciOiJFUzI1NiJ9…dkEZuZSchwlbYOP-JpxxGKDxhF9LLbBeNkx1Q’}

2022-09-12 15:49:56.750 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Checking Token value:

2022-09-12 15:49:56.751 DEBUG (MainThread) [custom_components.enphase_envoy.envoy_reader] Found empty token:

Are you able to browse to the envoy at that URL? I wonder if they removed the /auth/check_jwt endpoint in that firmware. Can your envoy reach out to the internet?

Sorry for late reply. I was out of town.

I can access Enphase Authentication | Login page.
I can not access https://192.168.1.21/auth/check_jwt gives me a 404 error.

you want me to check any other links?

I get the same error as @TerroBladeZ, on both version 7.0.101 and 7.0.153. Three observations:

  1. Going to the Enphase authentication login page, I have a choice to “Select System”. I apparently have two of them, likely from having two “logical” groups of panels set up in the app. (All the panels go to the same combiner, but on my roof they look like two different “physical” groups.) Selecting the first system renders zero “Select Gateway” options and I get an error attempting to generate an access token. Selecting my second system I am able to select the gateway and also generate an access token just fine.
  • @TerroBladeZ are you having the same issue?
  • @jrutski perhaps the code doesn’t support multiple systems?
  1. I am able to go to the check_jwt endpoint if I use my envoy’s IPv4 address, which TerroBladeZ was not able to do. This leads me to my next point…

  2. My last observation is that the envoy add-on is using an IPv6 address. This wasn’t always the case and I had this error when it was an IPv4 address as well. Just thought I should mention it here

P.S. let me know if I can help in anyway. I can get my hands dirty, debug, and submit a PR; but I just need to be pointed in the right direction.

@unibeck thanks for the feedback - when you add the system, the serial number to use as the gateway ID is your Envoy S/N. Assuming you only have 1 envoy with 2 groups? Interesting that you have multiple options on the select gateway screen.

@TerroBladeZ thanks for checking. It does indeed look like the /check_jwt endpoint has been removed for you. When debugging my system, I did notice that endpoint never being used, but on my system, browsing to that endpoint provides a 401 and dumps back to the page with the JWT form, so it’s still there. It appears that the endpoints are doing JWT checking in-line, without having to go through that. I think that step may need to be removed for newer firmwares.

Fantastic work! Happy to help test as well. My version is D7.0.85, no batteries.