Extend the Solar Forecast and import the actual data to be able to manipulate

Trying to make use of the actual forecast data, and not only be able to glance at the Graph, or find a single max value, I’ve been exploring the actual Free api made available by https://knutkohl.consulting.

This allows me to compare the provided data, (which changes during the day in showing either tomorrow and today, or yesterday and today, depending on the subset selected)

I’ve set it up in a way, that the actual sensor value template show todays (max) value where applicable, and has all hourly data in its attributes. Made those into a single dictionary for now. Maybe it can be useful to template them out into single entities at a later time.

My Frontend card uses custom:template-entity-row, to show the max of those attributes over the 2 days.
This way, we can come closer to deciding whether the dishwasher should be used now, or better wait until tomorrow during sun production. Or, when that would be low, use the low rate time period. what ever suits your situation

@petro and @thefes helped me out tremendously on the template section, thx again, you make this community shine!
original author and code-owner of the Solar Forecast integration @klaasnicolaas also shared his feedback in the initial stages of this little project. Thanks Klaas!
As that resulted in revealing the unlikelihood of Core HA providing the data set as it is available below in the near future, I pursued here.


Challenge:

as you can see I’ve not yet decided which entity I need to set the automation to update.
There is a strict ratelimit, and not wanting to bash the free service provider (Thanks!) I’ve set the scan_interval to once a day, and only want to update during daytime.

I could add all 5 sensors on this single resource, but havent been able to establish what that does with the ratelimit. Preferably we would be able to update the resource only once, and the sensors would follow suit? Not sure yet.


Anyways, this is what it looks like for now (might add a nice Apexcharts card with the graphed data, but since we have that in core Dashboard, didnt spend much time on that yet. note this holds more data-points which was the whole reason to start this)

some more info-s:

backend package

##########################################################################################
# Package Solar Forecast Estimate rest sensor met alle attributes
# zie https://swagger.forecast.solar
# @mariusthvdb is also my Github and community handle.
##########################################################################################

##########################################################################################
# Automation
# https://www.home-assistant.io/integrations/automation/
##########################################################################################


automation:

  id: update_solar_forecast_estimate
  alias: Update solar forecast estimate
  trigger:
    platform: time_pattern
    minutes: 0
  condition:
    condition: state
    entity_id: sun.sun
    state: above_horizon
  action:
    service: homeassistant.update_entity
    target:
      entity_id:
        -

##########################################################################################
# Rest
# https://www.home-assistant.io/integrations/rest
##########################################################################################

# {% set midnight = today_at() %}
# {% set tomorrow = midnight + timedelta(days=1) %}
# {% set data = value_json.result.watts.items()
# | rejectattr('0', '<=', midnight.isoformat())
# | rejectattr('0', '>=', tomorrow .isoformat())
# | sort(attribute='1', reverse=True) %}
# {{ (data | first | default(("Unknown",0)))[1] }}
# Thank you @Petro !

# {{ (value_json.result.watts.items()
# | list | sort(attribute='1', reverse=True)
# | first | default(("Unknown",0)))[1] }}

# {{ value_json.result.watts.items()
# | rejectattr('0', '<', today_at().isoformat())
# | rejectattr('0', '>=', (today_at() + timedelta(days=1)).isoformat())
# | map(attribute='1') | max }}

# items are templastes in KVP's. [0] selects timestamp, [1] selects value

# for Frontend:
# {% set peak = (data | first | default(("Unknown",0))) %}
# {{ peak[1] }} W om {{ as_datetime(peak[0]).strftime('%-H') }} uur

# {% set data =
#  value_json.result.watt_hours_day.items() | list
# | sort(attribute='1', reverse=True)  %}
# {% set peak = (data | first | default(("Unknown",0))) %}
# {{ peak[1] }} W op {{ as_datetime(peak[0]).strftime('%-d-%-m') }}

# thank you @TheFes !

# rest sensors calculating the 'todays' value, frontend showing the highest of 2 days
# in secondary
rest:

  - resource: !secret solar_forecast_api
    scan_interval: 86400
    sensor:

      - unique_id: solar_forecast_estimate_watts
        name: Solar Forecast estimate watts
        icon: mdi:solar-power
        unit_of_measurement: W
        value_template: >
          {% set today = now().strftime('%Y-%m-%d') %}
          {% set data = value_json.result.watts.items()
            |selectattr('0','search',today)
            |sort(attribute='1',reverse=True) %}
          {{(data|first|default(('Unknown',0)))[1]}}
        json_attributes_path: "$.result"
        json_attributes:
          - watts

      - unique_id: solar_forecast_estimate_watt_hours_period
        name: Solar Forecast estimate watt hours period
        icon: mdi:solar-power-variant
        unit_of_measurement: Wh
        value_template: >
          {% set today = now().strftime('%Y-%m-%d') %}
          {% set data = value_json.result.watt_hours_period.items()
            |selectattr('0','search',today)
            |sort(attribute='1',reverse=True) %}
          {{(data|first|default(('Unknown',0)))[1]}}
        json_attributes_path: "$.result"
        json_attributes:
          - watt_hours_period

      - unique_id: solar_forecast_estimate_watt_hours
        name: Solar Forecast estimate watt hours
        icon: mdi:sun-clock-outline
        unit_of_measurement: Wh
        value_template: >
          {% set today = now().strftime('%Y-%m-%d') %}
          {% set data = value_json.result.watt_hours.items()
            |selectattr('0','search',today)
            |sort(attribute='1',reverse=True) %}
          {{(data|first|default(('Unknown',0)))[1]}}
        json_attributes_path: "$.result"
        json_attributes:
          - watt_hours

      - unique_id: solar_forecast_estimate_watt_hours_day
        name: Solar Forecast estimate watt hours day
        icon: mdi:calendar-today
        unit_of_measurement: Wh
        value_template: >
          {% set today = now().strftime('%Y-%m-%d') %}
          {{value_json.result.watt_hours_day[today]}}
#           {% set today = now().strftime('%Y-%m-%d') %}
#           {{(value_json.result.watt_hours_day.items()|list
#             |selectattr('0','search',today)
#             |sort(attribute='1',reverse=True)
#             |first|default(('Unknown',0)))[1]}}
        json_attributes_path: "$.result"
        json_attributes:
          - watt_hours_day

      - unique_id: solar_forecast_estimate_message
        name: Solar Forecast estimate message
        icon: mdi:message-text-outline
        value_template: >
          {% set type = value_json.message.type %}
          {{type if type == 'success' else value_json.message.text}}
        json_attributes_path: "$.message"
        json_attributes:
          - code
          - type
          - text
          - info
          - ratelimit

##########################################################################################
# Template
# https://www.home-assistant.io/integrations/template/
##########################################################################################

template:

  - sensor:

      - unique_id: solar_forecast_estimate_current_hour
        name: Solar forecast current hour
        state: >
          {% set nu = now().replace(minute=0, second=0 , microsecond=0).isoformat() %}
          {% set watts = state_attr('sensor.solar_forecast_estimate_watts','watts')%}
          {% set data = watts.items() %}
          {{watts[nu] if nu in watts else 'No production estimated'}}
        icon: mdi:sun-clock
        unit_of_measurement: W

# guard against `now()` not being in the dictionary because no data is provided by the api at sun = down.
# could probably also test for sun state too ;-)

##########################################################################################
# Rest
# https://www.home-assistant.io/integrations/sensor.rest/
##########################################################################################

# sensor:
#
#   - platform: rest
#     unique_id: solar_forecast_estimate_rest_sensor
#     name: Solar Forecast estimate
#     resource: !secret solar_forecast_api
#     scan_interval: 3600
#     value_template: >
#       {{now()}}
#     headers:
#       Accept: "application/json"
#     json_attributes_path: "$.result"
#     json_attributes:
#       - watts
#       - watt_hours_period
#       - watt_hours
#       - watt_hours_day

and Dashboard card:

type: entities
card_mod:
  class: class-header-margin
title: Solar Forecast estimates
state_color: true
entities:
  - type: custom:template-entity-row
    entity: sensor.solar_forecast_current_hour
    secondary: >
      Actuele opbrengst: {{states('sensor.zp_actuele_opbrengst')|int(default=0)}} W
  - type: divider

  - type: custom:template-entity-row
    entity: sensor.solar_forecast_estimate_watts
    secondary: >
      {% if states[config.entity] is not none %}
        {% set attr =
            state_attr(config.entity,'watts').items()
            |sort(attribute='1',reverse=True) %}
        {% set peak = (attr|first| default(('Unknown',0))) %}
         {{peak[1]}} W op {{as_datetime(peak[0]).strftime('%-d-%m')}} om {{as_datetime(peak[0]).strftime('%-H')}} uur
      {% else %} Initializing
      {% endif %}
  - type: custom:template-entity-row
    entity: sensor.solar_forecast_estimate_watt_hours_period
    secondary: >
      {% if states[config.entity] is not none %}
        {% set attr =
            state_attr(config.entity,'watt_hours_period').items()
            |sort(attribute='1',reverse=True) %}
        {% set peak = (attr|first| default(('Unknown',0))) %}
         {{peak[1]}} W op {{as_datetime(peak[0]).strftime('%-d-%m')}} om {{as_datetime(peak[0]).strftime('%-H')}} uur
      {% else %} Initializing
      {% endif %}
  - type: custom:template-entity-row
    entity: sensor.solar_forecast_estimate_watt_hours
    secondary: >
      {% if states[config.entity] is not none %}
        {% set attr =
            state_attr(config.entity,'watt_hours').items()
            |sort(attribute='1',reverse=True) %}
        {% set peak = (attr|first| default(('Unknown',0))) %}
         {{peak[1]}} W op {{as_datetime(peak[0]).strftime('%-d-%-m')}}
      {% else %} Initializing
      {% endif %}
  - type: custom:template-entity-row
    entity: sensor.solar_forecast_estimate_watt_hours_day
    secondary: >
      {% if states[config.entity] is not none %}
        {% set attr =
            state_attr(config.entity,'watt_hours_day').items()
            |sort(attribute='1',reverse=True) %}
            {% set peak = (attr|first| default(('Unknown',0))) %}
        {{peak[1]}} W op {{as_datetime(peak[0]).strftime('%-d-%-m')}}
      {% else %} Initializing
      {% endif %}
  - type: custom:template-entity-row
    entity: sensor.solar_forecast_estimate_message
    secondary: >
      {% if states[config.entity] is not none %}
        {% set limit = state_attr(config.entity,'ratelimit').limit %}
        {% set left = state_attr(config.entity,'ratelimit').remaining %}
        Limit: {{limit}}, Remaining: {{left}}
      {% else %} Initializing
      {% endif %}
  - entity: automation.update_solar_forecast_estimate
    secondary_info: last-triggered

and an initial Apex-charts card:

  - type: custom:apexcharts-card
    graph_span: 48h
    update_interval: 1h
    header:
      show: true
      title: Solar Forecast
    experimental:
      color_threshold: true
    span:
      start: day
    now:
      show: true
      label: Now
    yaxis:
      - id: estimate
        decimals: 0
      - id: actueel
        decimals: 0
        show: false
    apex_config:
      plotOptions:
        bar:
          columnWidth: 100%
    series:
      - entity: sensor.solar_forecast_estimate_watts
        name: Solar power
        yaxis_id: estimate
        data_generator: |
          let res = []; for (const [key, value] of
            Object.entries(entity.attributes.watts)) {
            res.push([new Date(key).getTime(), value]);
            } return res;
        color_threshold:
          - value: 0
            color: moccasin
          - value: 500
            color: khaki
          - value: 1000
            color: yellow
          - value: 1500
            color: gold
          - value: 2000
            color: orange
          - value: 2500
            color: darkorange
          - value: 3000
            color: tomato
          - value: 3500
            color: sandybrown
          - value: 4000
            color: peru
          - value: 4500
            color: orangered
          - value: 5000
            color: maroon
          - value: 5500
            color: purple
          - value: 6000
            color: darkred
        type: column
        show:
          extremas: true
#          legend_value: true
#         opacity: 1
#         float_precision: 0
        stroke_width: 8

      - entity: sensor.zp_actuele_opbrengst
        yaxis_id: actueel
#         transform: return x /1000;
#         unit: kW
        type: line
        name: Actueel
        stroke_width: 1
        extend_to: false
        color: grey
        group_by:
          func: min
          duration: 60m
#        show:
#          legend_value: true
#        float_precision: 0

there’s some extra detail to work out there (better color thresholds, and,

why does the legend show 0…

thanks @vingerha and @studioIngrid for helping me find the construction for the data_generator, converting the dict to a list!

combining those with

and one can see the foreshadows of some better logic, for deciding when to fire which utility

2 Likes

It looks pretty interesting.
Tried to reproduce, but I cannot get the API to work, the API Page doesn’t load on my side (too many redirects on multiple browsers).
Did you find the API specs on another place?

seems you use forecast.solar and not swagger.forecast.solar ?

It’s back online, seems it was a temporary issue. Thanks

Still causing issues here now …

It didn’t update during the weekend indeed, update seems back ok this morning.
Only one issue left, the value of the forecast for current hour is ‘n’

yes, check the opening post, I updated that sensor to guard it doesnt update when sun is not showing :wink:

noticed the history graph of my watts sensor, to which I have set:

unit_of_measurement: W

doesnt show correctly, and only follows state, not does it show the unit on the more-info:

Yep, actually it seems I still have some updating challenge.
My data lags 24h behind.

While the API page seems correct and up-to-date

Trace of the automation gives an error on config step:

service: homeassistant.update_entity
target:
  entity_id:
    - null

gives

Executed: 27 mars 2023 à 10:00:00
Error: Template rendered invalid entity IDs: [None]

not seeing that here:

could it be you’ve exceeded the api call limit? it should state so in the error message of the rest sensor

solved:

updated template sensor to:

          {% set nu = now().replace(minute=0, second=0 , microsecond=0).isoformat() %}
          {% set watts = state_attr('sensor.solar_forecast_estimate_watts','watts')%}
          {% set data = watts.items() %}
          {{watts[nu] if nu in watts else 'No production estimated'}}

see: Templating - Home Assistant for some explanation

Got it to work on my end.
Had to replace the first line with:

{% set nu = now().replace(minute=0, second=0, microsecond=0).strftime('%Y-%m-%d %H:%M:%S') %}

The isoformat() puts a T between date & time what isn’t present in the sensor’s attributes:

watts:
  '2023-04-04 07:14:00': 0
  '2023-04-04 08:00:00': 713
  '2023-04-04 09:00:00': 756
  '2023-04-04 10:00:00': 1400
  '2023-04-04 11:00:00': 1619
  '2023-04-04 12:00:00': 1656
  '2023-04-04 13:00:00': 1608
[...]
  '2023-04-05 20:00:00': 5
  '2023-04-05 20:23:18': 0
icon: mdi:solar-power
friendly_name: Solar Forecast estimate watts

Therefore no match and no value for current hour