REST API command/platform fails on POST to external URL (solcast)

Beyond anyone’s at the moment. The PVGIS data set that Forecast Solar uses does not cover the area. I contacted PVGIS and got this response:

we are working on updating the various datasets of PVGIS to include data up to 2020 and also to extend the spatial coverage.

It would be great to be able to provide sastellite based solar radiation data for Australia and New Zealand, but for the time being, that area is not among our priorities. However, we intend to provide data from the reanalysis product ERA5 that would cover that part of the world.

We would like to have this update ready by the end of this year.

So there is the possibility of hourly forecasts soon.

Cheers, Tom, but I was referring to Solcast’s capabilities, not Forecast Solar’s, which are inadequate.

Solcast will give 30 minute forecasts and has a simple API.

It can give HA’s energy panel the capability to be fully utilised by everyone.

This is particularly relevant given that:

Here is my current Solcast graph in HA using the REST integration:

Yeah I implemented Solcast yesterday thanks to this helpful thread. Only have the Apex chart to do today.

2021.9 Adds support for alternative Solar Forecasts, (commit: 54576) its now just a matter of integrating the Solcast API to use the documented energy forecast panel integration.

Great news! Thanks for that.

Vote for a SolCast integration:

1 Like

Using the apex chart config you have kindly provided, every morning when the new day’s data comes in my forecast data graph goes a bit wrong. There’s no way I’m going to generate power at midnight, and the data from the day before certainly did not predict that.

Screenshot 2021-09-09 at 13-04-19 Overview - Home Assistant

Do you have any idea why this happens?

I was thinking it might be a timezone thing as the forecast data is in UTC (https://hatebin.com/aogimlkrfj) but that cant be it as the actual and forecast data for the rest of the day lines up.

type: custom:apexcharts-card
graph_span: 36h
span:
  start: day
  offset: '-6h'
header:
  show: true
  title: Solar Forecast
  show_states: true
now:
  show: true
  label: now
apex_config:
  legend:
    show: true
series:
  - entity: sensor.sma_inverter_power
    name: Actual
    unit: W
    fill_raw: last
    color: '#0da035'
    stroke_width: 2
    extend_to_end: false
    group_by:
      func: avg
      duration: 30min
    show:
      legend_value: false
  - entity: sensor.solcast_forecast_average_30min
    transform: return x * 1000;
    name: Past Forecast
    unit: W
    fill_raw: last
    color: '#e0b400'
    stroke_width: 2
    extend_to_end: false
    show:
      legend_value: false
  - entity: sensor.solcast_forecast_data
    type: line
    name: Future Forecast
    extend_to_end: false
    stroke_width: 2
    unit: W
    show:
      in_header: false
      legend_value: false
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate*1000];
       });

Edit: I think I’ve worked it out. This sensor (for the past data) does not update while the value is a t 0 all night:

- name: solcast_forecast_average_30min
  unit_of_measurement: "kW"
  device_class: power
  state: >
    {{ state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined')|round(2) }}

So the graph draws a curve from the last data point. Setting the graph to step line shows this:

Screenshot 2021-09-09 at 13-00-32 Overview - Home Assistant

So I think I need to force the template sensor for the 30min average to update, maybe by including now() like this:

- name: solcast_forecast_average_30min
  unit_of_measurement: "kW"
  device_class: power
  state: >
    {% set force_update = now() %}
    {{ state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined')|round(2) }}

If that does not work I’ll try adding a random number in the 0.1 to 0.2W range every 1 minute.

I have the same behavior. The apex chart card does some kind of low pass filtering on the data to generate smoother plots. If you compare the apex-chart with your actual history of your production, you will see a lot more peaks in the raw data. Whatever smoothing apex-chart card does together with the fact that the sensor data values are not stored when the forecasts stays at 0 during the night, results in that graph you are seeing. I haven’t looked into that, but indeed your solution to either adding a random number could work or we could check with the developer of apex-chart if there are other smoothing functions we can try?

The default for the apexcharts curve option is smooth which draws a curve between the points. Not ideal when your last point was 12 hours ago. The other options are straight (direct line between points) or stepline (flat line until next point then straight up or down).

Only the last option will not introduce the issue but it’s not the best looking.

I think if the template sensor is forced to update every minute like I have done with now() in the template above it should fix the issue. As there will be a point every 60 seconds even when the value is stuck at 0 for the whole night. I’ll let you know if it works in 12 hours or so.

It did not work.

Screenshot 2021-09-10 at 09-24-44 Overview - Home Assistant

I will try the adding of a small random number if 0.

  state: >
    {% set force_update = now() %}
    {% if state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined')|round(2) == 0 %}
      {{ range(-100,100)|random / 100000 }}
    {% else %}
      {{ state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined')|round(2) }}
    {% endif %}

The other issue is that the simple integrations for for the today and tomorrow energy forecast sensors are often very inaccurate. Like +/-200% inaccurate - even when my actual power follows the forecast power graph closely.

I can see what the template is trying to do, it’s a half hour power prediction, so divide the power by two and add it to the total energy. But when I do this manually using the forecast power data visible in the graph I get a much more accurate answer (±10%). As the power data seems correct I’m assuming it is the date range selection that is at fault. Not sure why you need to replace the TZ info and the predictions appear to be in UTC anyway ( period_end: '2021-09-09T23:30:00.0000000Z' ), and you seem to be comparing it to the local date.

  state: >
    {% set ns = namespace (fc_tommorrow = 0) %}
    {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
      {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %}
      {% if daydiff.days == 1 %}
        {% set ns.fc_tommorrow = ns.fc_tommorrow + (forecast.pv_estimate/2)|float %}
      {%- endif %}
    {%- endfor %}
    {{ ns.fc_tommorrow|round(2) }}

Hey Tom,

I don’t use the template sensor data for the graph. I use the Rest sensor value. This was my mod from the original (and brilliant!!) idea.

Here is the raw sensor data from today:
Capture

And the corresponding graph:

Doesn’t give me any problems.

Worth a try.

1 Like

Thanks I’ll give that a go. Adding a random small value when 0 did work, but it felt a bit kludgy.

Screenshot 2021-09-11 at 12-04-17 Overview - Home Assistant

That fixed it. Thanks. The force_update: true option in the rest sensor is the key.

Going to see if I can add the 10/90 stats to the graph too.

Edit: Damn it! I forgot to limit my rest calls when making new sensors to record the past 10/90 percentile data.

Screenshot 2021-09-12 at 14-36-41 Solcast API Toolkit

Oh well. Testing that will have to wait until tomorrow. What I have so far:

Screenshot 2021-09-12 at 11-17-25 Administration - Home Assistant

There does not seem to be a way to stack individual area series (only all of them) so this won’t work as I want unfortunately:
Screenshot 2021-09-12 at 11-24-01 Administration - Home Assistant

2 Likes

I have updated my range of Solcast Sensors to mimic those created by the Forecast.Solar integration.

I hope this is useful once they enable other solar forecasting sensors to be used by the Energy Panel.

Feel free to check on my calculations, but I think they are all Ok.

sensor:
  - platform: rest
    name: "Solcast Forecast Data"
    json_attributes:
      - forecasts
    resource: https://api.solcast.com.au/rooftop_sites/SOLCAST_RESOURCE_ID/forecasts?format=json&api_key=SOLCAST_API_KEY&hours=72
    method: GET
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    unit_of_measurement: "kW"
    device_class: power
    scan_interval: 00:30
    force_update: true

template:
  - sensor:
      - name: "Solcast Forecast Energy Today"
        unique_id: solcast_forecast_energy_today
        unit_of_measurement: "kWh"
        device_class: energy
        state: >
          {% set ns = namespace (fc_today = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% set ns.fc_today = ns.fc_today + (forecast.pv_estimate/2)|float %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today|round(2) }}
      - name: "Solcast Forecast Energy Tomorrow"
        unique_id: solcast_forecast_energy_tomorrow
        unit_of_measurement: "kWh"
        device_class: energy
        state: >
          {% set ns = namespace (fc_tommorrow = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% set ns.fc_tommorrow = ns.fc_tommorrow + (forecast.pv_estimate/2)|float %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tommorrow|round(2) }}
      - name: "Solcast Forecast Peak Power Today"
        unique_id: solcast_forecast_peak_power_today
        unit_of_measurement: "kW"
        device_class: power
        state_class: measurement
        state: >
          {% set ns = namespace (fc_today_max = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% if ns.fc_today_max < forecast.pv_estimate|float %}
                {% set ns.fc_today_max = forecast.pv_estimate|float %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today_max|round(2) }}
      - name: "Solcast Forecast Peak Power Tomorrow"
        unique_id: solcast_forecast_peak_power_tomorrow
        unit_of_measurement: "kW"
        device_class: power
        state_class: measurement
        state: >
          {% set ns = namespace (fc_tomorrow_max = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% if ns.fc_tomorrow_max < forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max = forecast.pv_estimate|float %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tomorrow_max|round(2) }}
      - name: "Solcast Forecast Peak Time Today"
        unique_id: solcast_forecast_peak_time_today
        state: >
          {% set ns = namespace (fc_today_max = 0, fc_today_max_time = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 0 %}
              {% if ns.fc_today_max < forecast.pv_estimate|float %}
                {% set ns.fc_today_max = forecast.pv_estimate|float %}
                {% set ns.fc_today_max_time = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).time() %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_today_max_time }}
      - name: "Solcast Forecast Peak Time Tomorrow"
        unique_id: solcast_forecast_peak_time_tomorrow
        state: >
          {% set ns = namespace (fc_tomorrow_max = 0, fc_tomorrow_max_time = 0) %}
          {% for forecast in state_attr('sensor.solcast_forecast_data', 'forecasts')|default('variable is not defined') %}
            {% set daydiff = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).date() - as_local(utcnow()).date() %} 
            {% if daydiff.days == 1 %}
              {% if ns.fc_tomorrow_max < forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max = forecast.pv_estimate|float %}
                {% set ns.fc_tomorrow_max_time = as_local(strptime(forecast.period_end, '%Y-%m-%dT%H:%M:%S.%f0Z').replace(tzinfo=utcnow().tzinfo)).time() %}
              {%- endif %}
            {%- endif %}
          {%- endfor %}
          {{ ns.fc_tomorrow_max_time }}
      - name: "Solcast Forecast Power Current Hour"
        unique_id: solcast_forecast_power_current_hour
        state: >
          {{ ((state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate|default('variable is not defined') + state_attr('sensor.solcast_forecast_data', 'forecasts')[1].pv_estimate|default('variable is not defined'))/2)|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Power Next 12 Hours"
        unique_id: solcast_forecast_power_next_12_hours
        state: >
          {% set ns = namespace (fc_next12 = 0) %}
          {% for i in range(0, 24) %}
            {% set ns.fc_next12 = ns.fc_next12 + states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']|float %}
          {%- endfor %}
          {{ ns.fc_next12|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Power Next 24 Hours"
        unique_id: solcast_forecast_power_next_24_hours
        state: >
          {% set ns = namespace (fc_next24 = 0) %}
          {% for i in range(0, 48) %}
            {% set ns.fc_next24 = ns.fc_next24 + states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']|float %}
          {%- endfor %}
          {{ ns.fc_next24|round(2) }}
        unit_of_measurement: 'kW'
        device_class: power
      - name: "Solcast Forecast Energy Current Hour"
        unique_id: solcast_forecast_energy_current_hour
        state: >
          {% set ns = namespace (fc_energy_current = 0) %}
          {% for i in range(0, 2) %}
            {% set ns.fc_energy_current = ns.fc_energy_current + (states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']/2)|float %}
          {%- endfor %}
          {{ ns.fc_energy_current|round(2) }}
        unit_of_measurement: 'kWh'
        device_class: energy
      - name: "Solcast Forecast Energy Next Hour"
        unique_id: solcast_forecast_enery_next_hour
        state: >
          {% set ns = namespace (fc_energy_next = 0) %}
          {% for i in range(2, 4) %}
            {% set ns.fc_energy_next = ns.fc_energy_next + (states.sensor.solcast_forecast_data.attributes['forecasts'][i]['pv_estimate']/2)|float %}
          {%- endfor %}
          {{ ns.fc_energy_next|round(2) }}
        unit_of_measurement: 'kWh'
        device_class: energy
1 Like

I’m using the new rest integration to make one API call to get the forecast and 10/90 predictions for the graph above:

rest:
- resource: https://api.solcast.com.au/rooftop_sites/RESOURCE_ID/forecasts?format=json&api_key=API_KEY&hours=72
  scan_interval: '00:30:00' # RATE LIMIT!
  sensor:
  - name: "Solcast Forecast Data"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    json_attributes:
    - forecasts
  - name: "Solcast Forecast 10"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate10|round(2) }}"
  - name: "Solcast Forecast 90"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate90|round(2) }}"
2 Likes

You could also use the other sensor formatting with value templates such as:

{{ ((state_attr('sensor.solcast_forecast_data', 'forecasts')[0].pv_estimate10|round(2) }}

Same difference.

Do you mean with template sensors?

That would have the issue with not having “force_update”, leading to weird apex charts when there is no update in the state value overnight.

1 Like

I’m wrong. You are right to get the full forecast for the percentiles.

My rest sensors only grab a state every 30 minutes, not the full list of forecast attributes. But the force_update option in the rest sensor fixes the issue I had above, REST API command/platform fails on POST to external URL (solcast) - #128 by tom_l (it is your solution to use the rest sensor, I just added more).

EDIT: seems to be working ok:

Screenshot 2021-09-14 at 11-15-19 Administration - Home Assistant

1 Like

How did you get them to stack? I have your same original problem where the forecast graphs for the three pv_estimates show, but only one historical (not the 10 or 90).

Thank you @tom_l your rest sensor config also fixed the issue for me where the line was wrong from midnight to the 1st data point in the morning.

Now it looks nice:

Screenshot 2021-09-15 at 10.59.25

@safepay
You need to add extra data points for the historic ones.

Using @tom_l 's config:

rest:
- resource: !secret solcast_forecast_resource
  #resource: https://api.solcast.com.au/rooftop_sites/RESOURCE_ID/forecasts?format=json&api_key=API_KEY&hours=72
  scan_interval: '00:30:00' # RATE LIMIT!
  sensor:
  - name: "Solcast Forecast Data"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate|round(2) }}"
    json_attributes:
    - forecasts
  - name: "Solcast Forecast 10"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate10|round(2) }}"
  - name: "Solcast Forecast 90"
    force_update: true
    value_template: "{{ value_json.forecasts[0].pv_estimate90|round(2) }}"

Then the apex chart code:

type: custom:apexcharts-card
graph_span: 36h
span:
  start: day
  offset: '-6h'
header:
  show: true
  title: Solar Production vs. forecast
  show_states: true
now:
  show: true
  label: now
apex_config:
  legend:
    show: false
series:
  - entity: sensor.goodwe_ppv
    stroke_width: 2
    show:
      extremas: false
    color: '#FFF700'
    name: Actual
    unit: W
    fill_raw: last
    extend_to_end: false
    group_by:
      func: avg
      duration: 30min
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
    color: '#3498DB'
    transform: return x * 1000;
    name: Forecast
    unit: W
    fill_raw: last
    extend_to_end: false
  - entity: sensor.solcast_forecast_10
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    transform: return x * 1000;
    name: Forecast 10
    unit: W
    fill_raw: last
    extend_to_end: false
    opacity: 0.4
  - entity: sensor.solcast_forecast_90
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    transform: return x * 1000;
    name: Forecast 90
    unit: W
    fill_raw: last
    extend_to_end: false
    opacity: 0.4
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#E74C3C'
    type: line
    extend_to_end: false
    unit: W
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate*1000];
       });
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    type: line
    extend_to_end: false
    unit: W
    opacity: 0.4
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate10*1000];
       });
  - entity: sensor.solcast_forecast_data
    stroke_width: 2
    show:
      extremas: false
      in_header: false
    color: '#797D7F'
    type: line
    extend_to_end: false
    unit: W
    opacity: 0.4
    data_generator: |
      return entity.attributes.forecasts.map((entry) => {
         return [new Date(entry.period_end), entry.pv_estimate90*1000];
       });

Just use the additional 10/90 sensors for historic data and then I guess you already have the additional data_generator sensors to get them for forecast.

1 Like