Enphase Envoy with Energy Dashboard

[del13r]

Glad you got it sorted.

I think these links pretty much sums up what I have been trying to say in most posts of this topic.

Expected data source not listed

Which then links to

Sensor Entity | Home Assistant Developer Docs

And then this link says you can customise any sensor that doesn’t already have what is needed

Sensor

The type of data a sensor returns impacts how it is displayed in the frontend. This is controlled by the sensor’s device class designation. Built-in sensors and many created from an integration will have this designation predefined. Those can be modified in the customize section. When manually creating a new sensor the device class may be optionally assigned.

These links need to be a lot easier to find, aggregated into 1 simple link and should be promoted more regarding issues with energy dashboard.

Beyond that, I feel the energy dashboard should be updated in the next version to be able to accept current_power_production and current_power_consumption sensors and do the calculations for us instead of having to do everything mentioned in my first post

[vk2him]

Hi all - I did some google research today and found a website https://thecomputerperson.wordpress.com/2016/08/03/enphase-envoy-s-data-scraping that had lots of interesting and unpublished urls within our Envoys that display instantaneous values for power, reactive power, apparent power, voltage, current , power factor, lifetime power, last 7 days power and heaps of others for Consumption, Production and Net Consumption. It’s a nicely formatted json file which is updated with current values every time you call this url:

http://envoy.local/production.json

Here’s an example from my system:

{"production":[{"type":"inverters","activeCount":18,"readingTime":1629011546,"wNow":48,"whLifetime":15040561},{"type":"eim","activeCount":1,"measurementType":"production","readingTime":1629011749,"wNow":23.678,"whLifetime":15003847.261,"varhLeadLifetime":0.002,"varhLagLifetime":4480231.873,"vahLifetime":17614612.48,"rmsCurrent":1.289,"rmsVoltage":241.538,"reactPwr":305.709,"apprntPwr":311.372,"pwrFactor":0.07,"whToday":24605.261,"whLastSevenDays":154546.261,"vahToday":26747.48,"varhLeadToday":0.002,"varhLagToday":4836.873}],"consumption":[{"type":"eim","activeCount":1,"measurementType":"total-consumption","readingTime":1629011749,"wNow":667.827,"whLifetime":13996625.56,"varhLeadLifetime":9752177.355,"varhLagLifetime":4501845.22,"vahLifetime":24310859.886,"rmsCurrent":5.589,"rmsVoltage":241.653,"reactPwr":-961.189,"apprntPwr":1350.597,"pwrFactor":0.49,"whToday":14845.56,"whLastSevenDays":169982.56,"vahToday":25662.886,"varhLeadToday":11539.355,"varhLagToday":4837.22},{"type":"eim","activeCount":1,"measurementType":"net-consumption","readingTime":1629011749,"wNow":644.15,"whLifetime":9080345.931,"varhLeadLifetime":9752177.353,"varhLagLifetime":21613.347,"vahLifetime":24310859.886,"rmsCurrent":4.3,"rmsVoltage":241.768,"reactPwr":-655.479,"apprntPwr":1040.031,"pwrFactor":0.64,"whToday":0,"whLastSevenDays":0,"vahToday":0,"varhLeadToday":0,"varhLagToday":0}],"storage":[{"type":"acb","activeCount":0,"readingTime":0,"wNow":0,"whNow":0,"state":"idle"}]}

There is another url that displays similar information on a continuous streaming basis, however there are a at least 3 points to be aware of:

1. You require the installer password for this - obtain this by running a simple python script from here GitHub - sarnau/EnphaseEnergy: Various collected information about Enphase Energy solar system
2. The data is json formatted, except that the first curly bracket is missing from the “data” key, so you need to add that bracket and the quotation marks before the JSON can be parsed
3. The data includes consumption/production details for power, current, power factor etc for three phases. If your system is single phase, the non-present phases read zero

http://installer:"installer password"@envoy.local/stream/meter

Here’s an example from my system which updates once per second:

data: {"production":{"ph-a":{"p":-2.24,"q":236.363,"s":239.815,"v":243.884,"i":0.984,"pf":0.0,"f":50.06},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"net-consumption":{"ph-a":{"p":785.708,"q":-648.654,"s":1144.564,"v":244.111,"i":4.69,"pf":0.68,"f":50.12},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"total-consumption":{"ph-a":{"p":783.468,"q":-885.017,"s":1384.502,"v":243.997,"i":5.674,"pf":0.57,"f":50.09},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}}}

data: {"production":{"ph-a":{"p":-2.127,"q":237.415,"s":240.205,"v":244.042,"i":0.985,"pf":0.0,"f":50.12},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"net-consumption":{"ph-a":{"p":787.869,"q":-649.53,"s":1147.731,"v":244.266,"i":4.707,"pf":0.68,"f":50.06},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"total-consumption":{"ph-a":{"p":785.742,"q":-886.946,"s":1389.779,"v":244.154,"i":5.692,"pf":0.57,"f":50.09},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}}}

data: {"production":{"ph-a":{"p":-1.949,"q":236.617,"s":240.536,"v":244.283,"i":0.985,"pf":0.0,"f":50.12},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"net-consumption":{"ph-a":{"p":781.671,"q":-649.608,"s":1144.144,"v":244.487,"i":4.674,"pf":0.68,"f":50.12},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"total-consumption":{"ph-a":{"p":779.722,"q":-886.225,"s":1382.896,"v":244.385,"i":5.659,"pf":0.56,"f":50.12},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}}}

Note:

It’s night here and you can see the phantom negative power values being generated.

Also note:

• s = Apparent power
• p = Real power (this is the one we are interested in)
• q = Reactive power

I’m not very well versed with Home Assistant - I’d like to create 3 x sensors from either of these urls to display real-time Consumption, Production and Exported power. Does anyone know how to create a sensor from json data?

[dgaust]

So back to this - I can now compare a full day of use with the AGL portal (which is the stuff they bill me on, so very accurate)

AGL stats

Energy Stats

Align almost exactly (just a .1kWh difference on import). Overall, pretty happy with the accuracy of the sensors now.

[Steve61 (Steve)]

Ok a noob question here. You say it’s okay if my sensor says energy

@del13r

Don’t panic if yours says energy in the name, it wont break functionality. Its just confusing naming a sensor with the word energy which is meant to be only used for numbers in kWh.

1. Does that mean I change the code in the yaml to “energy” where it says “power” or is it okay to leave it as power?

2. Also my sensors are “sensors.enphase…” I assume I change the entry from “sensors.envoy…” to “sensors.enphase”?

And one more beginner question please.

3. In the code where you have “sensor.envoy_SERIALNUMBER_current_power_consumption”, do I replace the SERIALNUMBER with the serial number of my Envoy or is that just calling the serial number from the already integrated system?

[del13r]

@Steve61

Does that mean I change the code in the yaml to “energy” where it says “power” or is it okay to leave it as power?

Yeah, when I originally started this project, as I had added my enphase envoy to Home Assistant in June 2021, mine were named …_current_energy_consumption and …_current_energy_production.
Someone mentioned the names had changed in late July 2021 however my names did not update (I assume this is only applied to newly added devices to the updated integration so it wont break anything on existing devices).
I then decided to test this and removed my enphase integration and re-added it and the names changed to …_current_power_consumption and …_current_power_production

In your case all you have to do is do a find/replace on my code whereever it says _current_power and replace with _current_energy
Alternatively, you could go into configuration / integrations / enphase / … / delete and re-setup your enphase envoy.

@Steve61

Also my sensors are “sensors.enphase…” I assume I change the entry from “sensors.envoy…” to “sensors.enphase”?

Yes, change the sensor names in the code to exactly match whatever your sensor names in your home assistant are. Go to developer tools / states / and confirm and make note of the sensor names containing _current

@Steve61

In the code where you have “sensor.envoy_SERIALNUMBER_current_power_consumption”, do I replace the SERIALNUMBER with the serial number of my Envoy or is that just calling the serial number from the already integrated system?

I only put SERIALNUMBER in the code as my sensor names actually had my physical envoy serial number in the sensor name. I wasnt comfortable broadcasting my envoy serial number here so I replaced my 12 digit serial number like 123456789012 with SERIALNUMBER. Go to developer tools / states / and confirm and make note of the sensor names containing _current. Your sensors may or may not have the serial number in the name.

[del13r]

@vk2him

http://envoy.local/production.json

I have accessed this http://envoy.local/production.json file before but I never felt the need to fully understand how I would extract data from it until I read your post.

Here is some of the output when I view the .JSON file from my envoy.

Tonight, I decided to read up on JSON format and tried to teach myself something new.

This is what I came up with

sensor:
  - platform: rest
    resource: http://envoy.local/production.json
    name: "Consumption Voltage"
    value_template: '{{ value_json.consumption[0].rmsVoltage }}'
    device_class: voltage
    unit_of_measurement: V

  - platform: rest
    resource: http://envoy.local/production.json
    name: "Consumption Power Factor"
    value_template: '{{ value_json.consumption[0].pwrFactor }}'
    device_class: power_factor
    unit_of_measurement: '%'

and this was the end result

[shampeon]

I’m getting a little lost here on what I ought to be doing with my Envoy-C model that’s integrated into HA. This is an older model that doesn’t have consumption data, and doesn’t expose envoy.local/production.json.

With an Envoy-C I have the following sensors:

• sensor.envoy_{serial number}_current_energy_production
• sensor.envoy_{serial number}_today_s_energy_production
• sensor.envoy_{serial number}_last_seven_days_energy_production
• sensor.envoy_{serial number}_lifetime_energy_production

Is the expectation that HA’s Enphase integration will expose these sensors with Energy in a future update?

If that is the case, am I correct that I would need some other device to monitor consumption and export?

[vk2him]

Yes thats correct - you need a newer Envoy to allow monitor consumption or use a separate device. I have an Envoy that records consumption and production, but I also have a second way of getting this data from a device called a Flukso that counts pulses from two power meters

[del13r]

My envoy-s is mounted on the side of my house next to the switchboard. My Envoy-S has a CT clamp clamped around the mains wire coming out of my meter to measure consumption which then leads into the envoy-s which measures the consumption data coming from this clamp.

Envoy-S stating it does consumption

CT Clamp

The Envoy C appears to be a little more portable, only meant for indoors and I assume it only talks to the microinverters at the back of your solar panels.

Envoy-C not mentioning anything about the ability to read consumption.

Only the Envoy-S metered version has the Consumption Monitoring capabilty.

I found this link that confirms this.
https://www4.enphase.com/en-uk/products-and-services/envoy/family

[del13r]

@vk2him

Does anyone know how to create a sensor from json data?

I am not in a hurry to bruteforce my Envoy Installer Password, but I was able to create 2 sensors based on the .json that doesnt need authentication at Enphase Envoy with Energy Dashboard - #100 by del13r

If you want to connect to the installer .json, I would check this link

RESTful Sensor

and try these configuration variables

authentication string (optional)
Type of the HTTP authentication. basic or digest.

username string (optional)
The username for accessing the REST endpoint.

password string (optional)
The password for accessing the REST endpoint.

[vk2him]

Thanks for your efforts @del13r - I’ve used your code example and I also have setup sensors using the production.json file. Your help was invaluable so thanks again! I changed the frequency the file is read by including

scan_interval: 1

Also thanks for the link to the RESTful sensor - I’ve spent a few hours and have been able to access the password protected installer .json however I have come across an issue I don’t know how to fix - the json data coming from that installer json isn’t 100% formatted correctly as json - this is what comes out - note it’s missing the start and end { } and the word data isn’t enclosed in quotes "

data: {"production":{"ph-a":{"p":3621.602,"q":305.704,"s":3627.168,"v":248.652,"i":14.582,"pf":1.0,"f":50.0},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"net-consumption":{"ph-a":{"p":-3195.826,"q":-686.842,"s":3277.01,"v":248.674,"i":13.17,"pf":-0.98,"f":50.0},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}},"total-consumption":{"ph-a":{"p":425.776,"q":-992.546,"s":351.08,"v":248.663,"i":1.412,"pf":1.0,"f":50.0},"ph-b":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0},"ph-c":{"p":0.0,"q":0.0,"s":0.0,"v":0.0,"i":0.0,"pf":0.0,"f":0.0}}}

It should look like this:

{"data":{"production":{"ph-a":{"p":3621.602,"q":305.704,"s":3627.168,"v":248.652,"i":14.582,"pf":1,"f":50},"ph-b":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0},"ph-c":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0}},"net-consumption":{"ph-a":{"p":-3195.826,"q":-686.842,"s":3277.01,"v":248.674,"i":13.17,"pf":-0.98,"f":50},"ph-b":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0},"ph-c":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0}},"total-consumption":{"ph-a":{"p":425.776,"q":-992.546,"s":351.08,"v":248.663,"i":1.412,"pf":1,"f":50},"ph-b":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0},"ph-c":{"p":0,"q":0,"s":0,"v":0,"i":0,"pf":0,"f":0}}}}

Which then formats correctly as:

{
  "data": {
    "production": {
      "ph-a": {
        "p": 3621.602,
        "q": 305.704,
        "s": 3627.168,
        "v": 248.652,
        "i": 14.582,
        "pf": 1,
        "f": 50
      },
      "ph-b": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      },
      "ph-c": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      }
    },
    "net-consumption": {
      "ph-a": {
        "p": -3195.826,
        "q": -686.842,
        "s": 3277.01,
        "v": 248.674,
        "i": 13.17,
        "pf": -0.98,
        "f": 50
      },
      "ph-b": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      },
      "ph-c": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      }
    },
    "total-consumption": {
      "ph-a": {
        "p": 425.776,
        "q": -992.546,
        "s": 351.08,
        "v": 248.663,
        "i": 1.412,
        "pf": 1,
        "f": 50
      },
      "ph-b": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      },
      "ph-c": {
        "p": 0,
        "q": 0,
        "s": 0,
        "v": 0,
        "i": 0,
        "pf": 0,
        "f": 0
      }
    }
  }
}

My level of skills on Home Assistant are quite low - I know how to fix this in php or in a bash script but I’m struggling on how to fix it in Home Assistant - would I use a value template or something else?

Cheers

[del13r]

Instead of focusing on trying to insert a missing character at the start, try figuring out how to ignore the string data: or the first 5 characters (6 including the space). The great news is that data: only appears once at the very beginning.

ive removed that string from the beginning of the JSON and your JSON is valid after that.

This approach might be easier as I feel that filtering a response is easier than modifying a response.

Maybe something like

sensor:
  platform: command_line
  command: curl ... | jq 

Command line Sensor

https://curl.se/docs/manual.html

https://stedolan.github.io/jq/manual/

[vk2him]

Thanks so much for the pointer @del13r - yes your suggestion to simply trim is a much better solution - I’ll investigate this now.

[vk2him]

Thanks for that - I just did some testing too - I can’t use jq as it requires a properly formatted json input, however I haev had luck with sed

curl --anyauth -u installer:installerpassord --silent envoy.local/stream/meter --stderr - | sed 's/data://'

This strips the data: and leaves nicely formatted json as you predicted. I’m just about to create a sensor with this.

[del13r]

maybe try adding ^ so that it only checks the start of the string and a space after data to be like this so it gets rid of the space

sed 's/^data: //'

even this first 6 character delete will work for what you are trying to do

sed 's/^......//'

[vk2him]

Not much luck - seems teh curl command fails

2021-08-18 14:17:47 ERROR (SyncWorker_1) [homeassistant.components.command_line] Timeout for command: curl --anyauth -u installer:Password --silent envoy/local/stream/meter --stderr - | sed 's/data: //'
2021-08-18 14:17:47 WARNING (SyncWorker_1) [homeassistant.components.command_line.sensor] Empty reply found when expecting JSON data

I noticed if I run that entire curl command from terminal within HA or another unix system that it sometimes runs and other times it doesn’t. Strange cause going to the url in a browser works 100% every time. I wonder if I need to fool it to think I’m accessing via a web-browser - I’ll check to see if curl allows this.

[del13r]

try removing --anyauth and test

and then try --digest
https://curl.se/docs/manpage.html#--digest

qop refers to digest

Digest access authentication

[vk2him]

I’m still getting timeout when I try --digest, however from terminal it works 100% every time so that’s weird … I’m going to try a REST sensor instead of a command_line and see what happens

[del13r]

maybe try replacing envoy.local with static IP (I hope it is static) of envoy just incase its a dns issue

[vk2him]

Yes I have that already - I change it to envoy.local when posting here to not confuse people with my internal ip … and of course rest: won’t work as I can’t use curl with that so I’m stumped