Enphase Envoy - D7 firmware with JWT - A Different Approach

That statement only applies to the developer tools page. This does not apply to sensors or if statements which are instantly updated as soon as a state changes.

@del13r
I was previously using the Enphase DEV integretion… definitely like this one for the local resources (quicker)

one thing I see different is this sensor missing.
wondering where in the API this could be found to create it.

image

Total overkill.
In my experience, the if statement works flawlessly without it.

Here is my current setup for the enphase style view.

Setup the helpers in configuration.yaml

input_select:
  statistics_chart_range:
    initial: day
    options:
      - day
      - week
      - month
      - year

input_datetime:
  statistics_chart_date:
    has_date: true
    has_time: false

Verified they were setup
image

Then setup the template sensors in configuration.yaml

template:

  - sensor:
      - name: statistics_chart_grouping
        unique_id: statistics_chart_grouping
        state: |
          {% set grouping={
            'day':'h',
            'week':'d',
            'month':'d',
            'year':'d'}
          %}
          {% set range = states('input_select.statistics_chart_range') %}
          {{ grouping[range] }}
      - name: statistics_chart_span
        unique_id: statistics_chart_span
        state: |
          {% macro last_date_in_range(date_str,range) -%}
            {% set date=as_timestamp(date_str)|timestamp_local()|as_datetime() -%}
            {% if range=='day' -%}
              {% set end_date=date -%}
            {% elif range=='week' -%}
              {% set end_date=date + timedelta(days=(6-date.weekday())) -%}
            {% elif range=='month' -%}
              {% set month=(date.month % 12) + 1 -%}
              {% set year=date.year+iif(date.month==12,1,0) -%}
              {% set end_date=strptime(year|string + '-' + month|string + '-01', '%Y-%m-%d')-timedelta(days=1) -%}
            {% elif range=='year' -%}
              {% set end_date=strptime(date.year|string + '-12-31', '%Y-%m-%d') -%}
            {% endif -%}
            {{ as_timestamp(end_date)|timestamp_local() -}}
          {% endmacro -%}
          {% set range = states('input_select.statistics_chart_range') %}
          {% set date_str = states('input_datetime.statistics_chart_date') %}
          {% set spans={
            'day':'24',
            'week':'7',
            'month':(last_date_in_range (strptime(date_str, '%Y-%m-%d'),range)|as_datetime()).day,
            'year':last_date_in_range (strptime(date_str, '%Y-%m-%d'),range)|as_timestamp(0)|timestamp_custom('%j',0)|int +1}
          %}
          {{ spans[range] }}
      - name: statistics_chart_period
        unique_id: statistics_chart_period
        state: |
          {% set periods={
            'day':'hour',
            'week':'day',
            'month':'day',
            'year':'day'}
          %}
          {% set range = states('input_select.statistics_chart_range') %}
          {{ periods[range] }}
          
  - sensor:
      - name: statistics_chart_offset
        unique_id: statistics_chart_offset
        state: |
          {% macro last_date_in_range(date_str,range) -%}
            {% set date=as_timestamp(date_str)|timestamp_local()|as_datetime() -%}
            {% if range=='day' -%}
              {% set end_date = date + timedelta(days=1) -%}
            {% elif range=='week' -%}
              {% set end_date=date + timedelta(days=(6-date.weekday())) -%}
            {% elif range=='month' -%}
              {% set month=(date.month % 12) + 1 -%}
              {% set year=date.year+iif(date.month==12,1,0) -%}
              {% set end_date=strptime(year|string + '-' + month|string + '-01', '%Y-%m-%d')-timedelta(days=1) -%}
            {% elif range=='year' -%}
              {% set end_date=strptime(date.year|string + '-12-31', '%Y-%m-%d') -%}
            {% endif -%}
            {{ as_timestamp(end_date) -}}
          {% endmacro -%}
          {% set range = states('input_select.statistics_chart_range') %}
          {% set date_str = states('input_datetime.statistics_chart_date') %}
          {% set offset = last_date_in_range(strptime(date_str, '%Y-%m-%d'),range)|int - now()|as_timestamp(0) -%}
          {% if states('sensor.statistics_chart_grouping') == 'h' %}
            {% set offset=(offset/60/60)|round(method='ceil') %}
          {% else %}
            {% set offset=(offset/60/60/24)|round(method='ceil') + 1 %}
          {% endif %}
          {{ iif(offset >= 0,'+','') }}{{ offset }}{{ states('sensor.statistics_chart_grouping') }}

Verified they were setup
image

Then added this to scripts.yaml

statistics_chart_date_browse:
  alias: statistics_chart_date_browse
  sequence:
  - service: input_datetime.set_datetime
    target:
      entity_id: input_datetime.statistics_chart_date
    data:
      datetime: "{% set range = states('input_select.statistics_chart_range') %}\n{%
        set date = states('input_datetime.statistics_chart_date')|as_datetime %}\n{%
        set direction = direction %}\n{% if range == 'day' %}\n  {% if direction >
        0 %}\n    {% set target = date + timedelta(days=1) %}\n  {% else %}\n    {%
        set target = date - timedelta(days=1) %}\n  {% endif %}\n{% elif range ==
        'week' -%}\n  {% if direction > 0 %}\n    {% set target = date + timedelta(days=7)
        %}\n  {% else %}\n    {% set target = date - timedelta(days=7) %}\n  {% endif
        %}\n{% elif range == 'month' -%}\n  {% if direction > 0 %}\n    {% set month=((date.month)
        % 12) + 1 %}\n    {% set year=date.year+iif(date.month==12,1,0) %}\n    {%
        set target=strptime(year|string + '-' + month|string + '-01', '%Y-%m-%d')
        -%}\n  {% else %}\n    {% set month=((date.month - 2) % 12) + 1 %}\n    {%
        set year=date.year+iif(date.month==1,-1,0) %}\n    {% set target=strptime(year|string
        + '-' + month|string + '-01', '%Y-%m-%d') -%}\n  {% endif %}\n{% elif range=='year'
        -%}\n  {% if direction > 0 %}\n    {% set target=strptime((date.year|int +
        1)|string + '-01-01', '%Y-%m-%d') -%}\n  {% else %}\n    {% set target=strptime((date.year|int
        - 1)|string + '-01-01', '%Y-%m-%d') -%}\n  {% endif %}\n{% endif -%}\n{% if
        direction == 0 %}\n    {% set target = now() %}\n{% endif -%}\n{{ target }}\n"
  mode: single
  icon: mdi:arrow-right-bold-outline

Verified that was setup

I made an extra automation to rollover the graph to todays date at 12:01AM

alias: Update day of apexchart at midnight
description: ""
trigger:
  - platform: time
    at: "00:01:00"
condition: []
action:
  - service: script.statistics_chart_date_browse
    data:
      direction: 0
mode: single

Then created a dashboard view

  - theme: Backend-selected
    title: Enphase
    path: enphase
    icon: mdi:solar-power
    type: sidebar
    badges: []
    cards:
      - type: custom:config-template-card
        variables:
          offset: states['sensor.statistics_chart_offset'].state
          span: states['sensor.statistics_chart_span'].state
          period: states['sensor.statistics_chart_period'].state
          grouping: states['sensor.statistics_chart_grouping'].state
        entities:
          - sensor.statistics_chart_offset
          - sensor.statistics_chart_span
          - sensor.statistics_chart_period
          - sensor.statistics_chart_grouping
        card:
          type: custom:apexcharts-card
          graph_span: 1d
          span:
            offset: ${offset}
          stacked: true
          header:
            show: false
            title: Total Power
          series:
            - entity: sensor.power_production
              type: column
              name: Produced
              color: '#01B4DE'
              group_by:
                func: avg
                duration: 5min
            - entity: sensor.power_consumption
              transform: return x *-1 ;
              type: column
              name: Consumed
              color: '#F37320'
              group_by:
                func: avg
                duration: 5min
            - entity: sensor.power_net
              type: column
              name: Imported(+)/Exported(-)
              color: '#545456'
              group_by:
                func: avg
                duration: 5min
          yaxis:
            - min: -7000
              max: 7000
              decimals: 0
      - type: custom:mod-card
        card:
          type: horizontal-stack
          cards:
            - type: entities
              entities:
                - entity: input_datetime.statistics_chart_date
                  name: date
            - type: glance
              show_name: false
              show_state: false
              entities:
                - entity: script.statistics_chart_date_browse
                  icon: mdi:chevron-left
                  tap_action:
                    action: call-service
                    service: script.statistics_chart_date_browse
                    data:
                      direction: -1
                - entity: script.statistics_chart_date_browse
                  icon: mdi:calendar-today
                  tap_action:
                    action: call-service
                    service: script.statistics_chart_date_browse
                    data:
                      direction: 0
                - entity: script.statistics_chart_date_browse
                  icon: mdi:chevron-right
                  tap_action:
                    action: call-service
                    service: script.statistics_chart_date_browse
                    data:
                      direction: 1

Verified that was good

Alternatively, if all of this is too complex, you can just have a simple card that just shows the last 24 hours using some of my older original instructions here.
https://community.home-assistant.io/t/enphase-envoy-with-energy-dashboard/328668/618?u=del13r

Hi @fireheadman,Which specific DEV integration are you referring to?
There are many to choose from:

briancmpbll/home_assistant_custom_envoy
posixx/home_assistant_custom_envoy (archived on Jul 14, 2023)
jrutski/home_assistant_envoy_d7_fw
DanBeard/enphase_envoy
jesserizzo/envoy_reader
vincentwolsink/home_assistant_enphase_envoy_installer

Unfortunately they are all referred to as DEV, so i need more specifics so I can try look at the code and work out what the source of binary_sensor.grid_status is.

I had a look at the code for the most popular DEV integration and found the following

I searched my https://envoy.local/home.json file for both enpower and grid_status but unfortunately I dont have batteries, so that is as far as I can test/investigate.

I did find this post where someone else had the same idea.

Great. Thank you for your insight.

1 Like

It was the campbell integration that I used… and good gosh!! Last I only knew of 5 custom enphase integrations, they are like gremlins now!

As for batteries, I do not have them either. He was able to pull out the grid status aside from batteries, but this might just be a tail chase, so not a big deal right now. I’m just happy to be able to pull data locally.

Thanks for the step by step guide, I will re-attempt this tomorrow.

1 Like

@del13r
I got it working… had to make a couple adjustments to get my values from getting chopped off. I have a 15.59kW system.

How can I reduce the size? I have to scroll to get to the selector, if I try “control -” to zoom out, it dynamically readjusts on me
Nevermind… found my issue, didn’t have type: sidebar, was using 1-Panel, not it works better. You ROCK! This is very awesome.

BTW: Is it normal behavior that when I have this card opened on my computer and at the same time view it on my iPhone, whenever I select another day on my phone, it updates the screen to that same view on my computer?

Only thing left to figure out is the tariffs. We have a funky deal between 4p-8p where the energy company takes the peak of each day out of 30 days to charge you $2/kWh. So if I decided to run the oven + dryer + sauna at the same time, then I get smacked with $2/kWh for that day between 4p-8p.

1 Like

Yes. The button calls the script to change a sensor related to what day you are viewing.

Hello everyone!

Thank you for all this valuable information! :muscle:t2: When I read the previous post and the post here, I still have a hard time understanding if there is a simple solution to integrate the Envoy (consumption, net energy, and production) in the “Energy” dashboard?

If not, why aren’t your developments or improvements directly committed to the main project?

It’s important for me to understand the context before getting into this I don’t want to wast my time with code or committing… errors :smile:

Thanks in advance for your help! :blush:

Hi @ALh4

Having initially struggled myself with energy dashboard when it was first released years ago, I can understand what you are getting at. Here is my attempt to answer your question.

When I wrote this, I had a very narrow list of requirements that were only meant to fix my own issues with my own hardware setup.
That list was:

  • I was originally only interested in getting data from 2 enphase sensors. power consumption and power production. From there I could use template sensors to calculate power import and power export and then use Integration - Riemann sum integral to record energy over time needed for energy dashboard. This is all I was interested in when I had D5 firmware.
  • get around the D7 firmware with cloud token not being supported by the native integration at the time.
  • poll the device faster than the once per 60 seconds I was previously getting with the native integration.
  • as a bonus, try to find a way to avoid using the Integration - Riemann sum integral which was used to convert power to energy over time.

The simplest solution right now is to use the native enphase integration.

Note: With all Enphase integrations, you will always need to setup template sensors to calculate power import and power export and then use Integration - Riemann sum integral to record energy over time needed for energy dashboard as the envoy production.json file that the Enphase Integrations use does not provide this information.

Why do I suggest this?
At that moment in time when I wrote this topic, the native integration did not support D7 firmware.

D7 firmware is now supported and if you are looking for the easiest and lowest maintenance solution, the native integration is the way to go because it will:

  • periodically refresh your enphase cloud token for you
  • support the increasing number of different models of enphase device (envoy/gateway/batteries)
  • work with either type of consumption CT clamp position/setup
  • get data on individual micro-inverters

A few reasons.

  • I lack any real development skills. Imagine someone who can read a language but cannot write it.

  • What I found via the replies I was getting to this topic, was that there are 2 types of consumption ct clamp position/setting. The position/setting of the consumption ct clamp was the functional difference between whether my solution needed the Integration - Riemann sum integral or not.
    This fragmentation across the installed enphase systems out there is the main reason I have not taken this further.

  • All of the other enphase related integrations that I am aware of, they all appear to rely on scraping data from the production.json file because every time that production.json file is requested, the envoy performs calculations to generate a standardised output regardless of the ct clamp setup. I can see how this standardised output would be attractive to someone creating an integration as it makes the ct clamp position one less condition to cater for.

The only real downside of using the production.json is that the envoy is slower to provide this production.json file each time it is requested. This lag only really matters when you are attempting to get data more often than the standard every 60 seconds from the envoy.

This production.json file lag/delay is what prompted me to look for other faster performing local enphase api endpoints so I could refresh the data more often and have the data arrive much faster with lower latency.
The alternative endpoint which I found that is more efficient, and still use today myself, is mainly what prompted me to write this topic so I could document my findings in the hope that someone with better development skills might see the benefit and take it further.

1 Like

A huge thank you for all this valuable information! :pray:t2: In fact I now understand my main mistake, I thought it was a “live” dashboard, but in fact it’s only a daily dashboard?

1 Like

This is not true. I use a HACS custom version based on production.json, that has the daily total and full total, as well as per panel production. I do not have clamps, and get grid import export from a P1 energy meter. I do not need any templates or Riemann.

The below integration also has the option to use the faster endpoints, but you will indeed lose the totals then. For Riemann to be accurate, you will really need the faster updates too ( and make sure you use left Riemann), but as you do not need Riemann, the slower updates are just fine.

1 Like

Energy dashboard is a daily energy dashboard which uses kWh

… made with hourly energy bars.

You can optionally make your own live power dashboard using data from an enphase device like I have which uses W

You can also graph the historical power data which also uses W

When I make a statement on this topic, it is implied that we are trying to obtain the following measurements from the enphase device:

  • energy import (grid consumption)
  • energy export (return to grid)
  • energy production (solar production)

These 3 measurements are what energy dashboard requires as input.

Sure, you can buy another device to fill in the gaps created by your choice to not buy an Enphase consumption ct clamp, however what I am trying to achieve here is allowing people to obtain all the required information exclusively from an Enphase device.

If people have already paid for an Enphase consumption ct clamp to be installed, then why would someone want spend more money on another device?

I have to assume that most people interested in setting up an enphase device on energy dashboard would understand they need an enphase consumption ct clamp to achieve this.

1 Like

I am also getting that information from production.json, however the only useful measurement for energy dashboard is production today which can be used to satisfy (solar production).

Consumption today is not useful as that only shows what your house consumed. Consumption today does not differentiate how much energy was imported and exported from the grid, nor does it show how much energy was consumed exclusively from the panels.

You still need either an enphase ct clamp or a p1 energy meter to provide energy export and energy import. Without either of those, you will only be able to see energy production in the energy dashboard. The rest will not be visible.

1 Like

The help is really appreciated, it’s given me some good starting points to get ready with Home Assistant. Thank you so much! :blush:

1 Like

I wasn’t advising to do it the same or having an opinion about your preferred setup. I was just highighting my different setup then yours in case it was relevant.

And as for cost, you assume clamps are already paid for. They cost extra for me. Also clamps did not fit in my current installation, so it would have cost me over 500€ to refit the installation to make the room. Add to that a smart energy meter was already present (as in most homes in our country) and already connected at very low cost. Less than they charged for installing clamps, excluding the retrofit). Not all installations are the same.

Maybe the integration is adding the daily totals at the end of the day, maybe it is getting it from a different endpoint. The code is there in the repo I linked to, might you be interested how it is done. The official Emphase app itself is also providing the lifetime production information, per panel and summed. So it is there somewhere. Using the Enphase provided value is possibly more accurate than a Rieman sum, unless you have really frequent updates. That is because wattages can fluctuate pretty quickly.

Ah ok, that makes sense now.

Yes, thanks for that.
I did have a look at the code and found the endpoints that potentially might be used when the option do_not_use_production_json=True

ENDPOINT_URL_METERS = "http{}://{}/ivp/meters"
ENDPOINT_URL_METERS_REPORTS = "http{}://{}/ivp/meters/reports"
ENDPOINT_URL_METERS_READINGS = "http{}://{}/ivp/meters/readings"

The code also checks for the CT Clamp Position/Setting which is good.

I am not aware of any Enphase integrations that record then add measurements together.
Recording and adding is better suited to Home Assistant’s recorder/history feature which stores the results in the database and calculations are usually done with template sensors.
I may be wrong.

If do_not_use_production_json=False, I assume that you are using the Enphase production today for (solar production) from production.json

image

If do_not_use_production_json=True, then only the lifetime total could come from:
https://envoy.local/ivp/meters/reports

image

Energy Dashboard can work with lifetime totals and just record the difference between the increase in lifetime readings at each hour of the day.

As you don’t have an Enphase CT Clamp, the other 2 metrics can only come from your 3rd party p1 meter:

  • energy import (grid consumption)
  • energy export (return to grid)