REST API command/platform fails on POST to external URL (solcast)

I have updated my range of Solcast Sensors to mimic those created by the Forecast.Solar integration.

I hope this is useful once they enable other solar forecasting sensors to be used by the Energy Panel.

Feel free to check on my calculations, but I think they are all Ok.

sensor:
  - platform: rest
    name: "Solcast Forecast Data"
    json_attributes:
      - forecasts
    resource: https://api.solcast.com.au/rooftop_sites/SOLCAST_RESOURCE_ID/forecasts?format=json&api_key=SOLCAST_API_KEY&hours=72
    method: GET
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    unit_of_measurement: "kW"
    device_class: power
    scan_interval: 00:30
    force_update: true

template:
  - sensor:
      - name: "Solcast Forecast Energy Today"
        unique_id: solcast_forecast_energy_today
        unit_of_measurement: "kWh"
        device_class: energy
        state: >
          {% set ns = namespace (fc_today = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% set ns.fc_today = ns.fc_today + (forecast.pv_estimate/2)|float %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today|round(2) }}
      - name: "Solcast Forecast Energy Tomorrow"
        unique_id: solcast_forecast_energy_tomorrow
        unit_of_measurement: "kWh"
        device_class: energy
        state: >
          {% set ns = namespace (fc_tommorrow = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% set ns.fc_tommorrow = ns.fc_tommorrow + (forecast.pv_estimate/2)|float %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tommorrow|round(2) }}
      - name: "Solcast Forecast Peak Power Today"
        unique_id: solcast_forecast_peak_power_today
        unit_of_measurement: "kW"
        device_class: power
        state_class: measurement
        state: >
          {% set ns = namespace (fc_today_max = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% if ns.fc_today_max < forecast.pv_estimate|float %}
                {% set ns.fc_today_max = forecast.pv_estimate|float %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today_max|round(2) }}
      - name: "Solcast Forecast Peak Power Tomorrow"
        unique_id: solcast_forecast_peak_power_tomorrow
        unit_of_measurement: "kW"
        device_class: power
        state_class: measurement
        state: >
          {% set ns = namespace (fc_tomorrow_max = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% if ns.fc_tomorrow_max < forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max = forecast.pv_estimate|float %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tomorrow_max|round(2) }}
      - name: "Solcast Forecast Peak Time Today"
        unique_id: solcast_forecast_peak_time_today
        state: >
          {% set ns = namespace (fc_today_max = 0, fc_today_max_time = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% if ns.fc_today_max < forecast.pv_estimate|float %}
                {% set ns.fc_today_max = forecast.pv_estimate|float %}
                {% set ns.fc_today_max_time = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).time() %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today_max_time }}
      - name: "Solcast Forecast Peak Time Tomorrow"
        unique_id: solcast_forecast_peak_time_tomorrow
        state: >
          {% set ns = namespace (fc_tomorrow_max = 0, fc_tomorrow_max_time = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% if ns.fc_tomorrow_max < forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max = forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max_time = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).time() %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tomorrow_max_time }}
      - name: "Solcast Forecast Power Current Hour"
        unique_id: solcast_forecast_power_current_hour
        state: >
          {{ ((state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined') + state_attr('sensor.solcast_forecast_data', 'forecasts')[1].pv_estimate|default('variable is not defined'))/2)|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Power Next 12 Hours"
        unique_id: solcast_forecast_power_next_12_hours
        state: >
          {% set ns = namespace (fc_next12 = 0) %}
          {% for i in range(0, 24) %}
            {% set ns.fc_next12 = ns.fc_next12 + states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']|float %}
          {%- endfor %}
          {{ ns.fc_next12|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Power Next 24 Hours"
        unique_id: solcast_forecast_power_next_24_hours
        state: >
          {% set ns = namespace (fc_next24 = 0) %}
          {% for i in range(0, 48) %}
            {% set ns.fc_next24 = ns.fc_next24 + states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']|float %}
          {%- endfor %}
          {{ ns.fc_next24|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Energy Current Hour"
        unique_id: solcast_forecast_energy_current_hour
        state: >
          {% set ns = namespace (fc_energy_current = 0) %}
          {% for i in range(0, 2) %}
            {% set ns.fc_energy_current = ns.fc_energy_current + (states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']/2)|float %}
          {%- endfor %}
          {{ ns.fc_energy_current|round(2) }}
        unit_of_measurement: 'kWh'
        device_class: energy
      - name: "Solcast Forecast Energy Next Hour"
        unique_id: solcast_forecast_enery_next_hour
        state: >
          {% set ns = namespace (fc_energy_next = 0) %}
          {% for i in range(2, 4) %}
            {% set ns.fc_energy_next = ns.fc_energy_next + (states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']/2)|float %}
          {%- endfor %}
          {{ ns.fc_energy_next|round(2) }}
        unit_of_measurement: 'kWh'
        device_class: energy
1 Like

I’m using the new rest integration to make one API call to get the forecast and 10/90 predictions for the graph above:

rest:
- resource: https://api.solcast.com.au/rooftop_sites/RESOURCE_ID/forecasts?format=json&api_key=API_KEY&hours=72
  scan_interval: '00:30:00' # RATE LIMIT!
  sensor:
  - name: "Solcast Forecast Data"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    json_attributes:
    - forecasts
  - name: "Solcast Forecast 10"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate10|round(2) }}"
  - name: "Solcast Forecast 90"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate90|round(2) }}"
2 Likes

You could also use the other sensor formatting with value templates such as:

{{ ((state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate10|round(2) }}

Same difference.

Do you mean with template sensors?

That would have the issue with not having “force_update”, leading to weird apex charts when there is no update in the state value overnight.

1 Like

I’m wrong. You are right to get the full forecast for the percentiles.

My rest sensors only grab a state every 30 minutes, not the full list of forecast attributes. But the force_update option in the rest sensor fixes the issue I had above, REST API command/platform fails on POST to external URL (solcast) - #128 by tom_l (it is your solution to use the rest sensor, I just added more).

EDIT: seems to be working ok:

Screenshot 2021-09-14 at 11-15-19 Administration - Home Assistant

1 Like

How did you get them to stack? I have your same original problem where the forecast graphs for the three pv_estimates show, but only one historical (not the 10 or 90).

Thank you @tom_l your rest sensor config also fixed the issue for me where the line was wrong from midnight to the 1st data point in the morning.

Now it looks nice:

Screenshot 2021-09-15 at 10.59.25

@safepay
You need to add extra data points for the historic ones.

Using @tom_l 's config:

rest:
- resource: !secret solcast_forecast_resource
  #resource: https://api.solcast.com.au/rooftop_sites/RESOURCE_ID/forecasts?format=json&api_key=API_KEY&hours=72
  scan_interval: '00:30:00' # RATE LIMIT!
  sensor:
  - name: "Solcast Forecast Data"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    json_attributes:
    - forecasts
  - name: "Solcast Forecast 10"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate10|round(2) }}"
  - name: "Solcast Forecast 90"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate90|round(2) }}"

Then the apex chart code:

type: custom:apexcharts-card
graph_span: 36h
span:
  start: day
  offset: '-6h'
header:
  show: true
  title: Solar Production vs. forecast
  show_states: true
now:
  show: true
  label: now
apex_config:
  legend:
    show: false
series:
  - entity: sensor.goodwe_ppv
    stroke_width: 2
    show:
      extremas: false
    color: '#FFF700'
    name: Actual
    unit: W
    fill_raw: last
    extend_to_end: false
    group_by:
      func: avg
      duration: 30min
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
    color: '#3498DB'
    transform: return x * 1000;
    name: Forecast
    unit: W
    fill_raw: last
    extend_to_end: false
  - entity: sensor.solcast_forecast_10
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    transform: return x * 1000;
    name: Forecast 10
    unit: W
    fill_raw: last
    extend_to_end: false
    opacity: 0.4
  - entity: sensor.solcast_forecast_90
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    transform: return x * 1000;
    name: Forecast 90
    unit: W
    fill_raw: last
    extend_to_end: false
    opacity: 0.4
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#E74C3C'
    type: line
    extend_to_end: false
    unit: W
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate*1000];
       });
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    type: line
    extend_to_end: false
    unit: W
    opacity: 0.4
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate10*1000];
       });
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    type: line
    extend_to_end: false
    unit: W
    opacity: 0.4
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate90*1000];
       });

Just use the additional 10/90 sensors for historic data and then I guess you already have the additional data_generator sensors to get them for forecast.

1 Like

I’ve added your sensor code, and most things look great. Thank you! Unless I’m missing something, though, there does seem to be one anomaly - the forecast power for the next 24 hours. I’ve attached a screenshot. This was taken at around 18.25 local time. Figures look very sane, except that the figure for the next 24 hours seems to be just double that for tomorrow. There could well be something I’m not understanding here though, because one measurement is energy, and the other power.

Because the data is in 30 minute intervals, the calculation for energy (kWh) vs power (kW) is energy divided by 2.

kWh = kW * h

For 30 minute intervals:

kWh = kW * 0.5

or

kWh = kW / 2

Got it. Thank you.

Don’t forget to vote for a Solcast integration.

I’m not sure why this template:

  state: >
    {% set ns = namespace (fc_tommorrow = 0) %}
    {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
      {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %}
      {% if daydiff.days == 1 %}
        {% set ns.fc_tommorrow = ns.fc_tommorrow + (forecast.pv_estimate/2)|float(0) %}
      {%- endif %}
    {%- endfor %}
    {{ ns.fc_tommorrow|round(2) }}

is generating this error every now and then:

Logger: homeassistant.components.template.template_entity
Source: components/template/template_entity.py:73
Integration: Template (documentation, issues)
First occurred: 10:38:21 (1 occurrences)
Last logged: 10:38:21

TemplateError('TypeError: 'NoneType' object is not iterable') while processing template 'Template("{% set ns = namespace (fc_tommorrow = 0) %} {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %} {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} {% if daydiff.days == 1 %} {% set ns.fc_tommorrow = ns.fc_tommorrow + (forecast.pv_estimate/2)|float(0) %} {%- endif %} {%- endfor %} {{ ns.fc_tommorrow|round(2) }}")' for attribute '_attr_native_value' in entity 'sensor.solcast_forecast_tomorrow'

There are defaults in case the forecast is not defined :thinking:

Do we need to change 'variable is not defined' to 0?

I thing None is valid as far as default goes. You might want to add the “true” option, which defaults if the input evaluates to false.

Is anyone here aware of this solcast integration? https://github.com/oziee/ha-solcast-solar

@safepay wondering whether you’ve looked into it. Seems to do most of what is implemented in this thread, with the exception of 10/90% probabilities, with the added advantage that it integrates with the Energy component, so the forecast shows on Energy’s solar graph.

Yes. I have been keeping an eye on it.

When it is properly in HACS I’ll install it. (Says it in HACS but it wasn’t last time I looked)

I also want 1/2 hour updates.

It seems to be a little over-zealous with API calls at the moment (there’s an open issue). Someone also says it does 2 API calls every hour, one for live, one for forecast, but I thought the method on this thread did everything in one call. Is that right? Integration doesn’t poll overnight though.

You can install it using HACS by adding its Github page as a custom repository.

I think it does one API call on startup to get history. Seems unnecessary to me as this is about forecasting solar data.

The most recent forecast is the “live” or current one, so one API call at a time is fine.

If you have more than one location, then 1 hour per location would allow updates without exceeding your API limit.

Still an outstanding issue. It seems to make 2 calls an hour, but that’s increasing by an extra 2 every hour, at least for me, so it’s multiplying.

Hi again @safepay. I’ve been running the solcast integration in parallel for a little while now. Its older version, 1.0.5, as described on the thread, seems to be OK with api calls. Looking at its sensors has brought up a couple of issues.

A sensor called energy_production_forecast_today has a fixed value all day. The sensor solcast_forecast_energy_today in the templates above doesn’t. Is it right that it contains the forecast for the rest of today? I’m not sure which approach is the best, or whether the Energy platform on whose sensors you seem to have based the templates dictates one approach over the other.

The template sensors solcast_forecast_energy_current_hour and solcast_forecast_energy_next_hour contain very different values to the integration’s energy_this_hour and energy_next_hour. In fact, they are about half the integration’s values, and the integration’s values seem to make more sense. I can see the templates for these sensors above take the first two “slots” in the solcast data for the current hour, and the next two slots for the next hour, and then divides by two, eg:

  - name: "Solcast Forecast Energy Current Hour"
    unique_id: solcast_forecast_energy_current_hour
    state: >
      {% set ns = namespace (fc_energy_current = 0) %}
      {% for i in range(0, 1) %}
        {% set ns.fc_energy_current = ns.fc_energy_current + (states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']/2)|float %}
      {%- endfor %}
      {{ ns.fc_energy_current|round(2) }}
    unit_of_measurement: 'kWh'
    device_class: energy

Given that it is calculating the power for an hour, I’m not sure it should be doing the division by two. Or am I way off here? Thank you.