REST API command/platform fails on POST to external URL (solcast)

Hi all
I’m attempting to use a platform: rest sensor (and doing testing using rest_command) to POST the 5-minute averages of my solar system’s output to solcast.com to tune it’s predictions of my production.

I’ve been using a platform: rest sensor successfully for a while now to GET predicted output from solcast.com, but no success with POST.

My yaml for the post_command is as follows (xxxx represents my the solcast rooftop ID, yyyy represents my API key):

rest_command:
  solcast_tune:
    url: https://api.solcast.com.au/rooftop_sites/xxxx/measurements?api_key=yyyy
    method: POST
    headers:
      Content-Type: application/json
    payload: states('sensor.solcast_post')

The payload in sensor.solcast_post is a string generated through a template sensor to create the JSON-coded info needed for the API. It looks like this:
{"measurement": {"period_end": "2019-10-19T03:35:00Z", "period": "PT5M","total_power": 1.19}}

I’ve tested the API POST using Postman (with the URL above, and strings straight from sensor.solcast_post and it works fine (i.e. no errors in Postman, and the measurement shows up on solcast.com).

However, whenever I can service rest_command.solcast_tune in Developer Tools I get a 400 error (per the solcast.com API, this indicates “The measurement does not pass validation (only for single measurement).

Does anyone have suggestions?

You are sending the literal string “states(‘sensor.solcast_post’)”. You need to use a template to send the value of the sensor.

payload: "{{ states('sensor.solcast_post') }}"
1 Like

Perfect—much appreciated!

Hi Ben,
would it be possible to also share your platform: rest sensor configuration for fetching the data from solcast.com?
BR
Philipp

For sure… Will post it tonight.

1 Like

Sorry all… having some network issues at the moment and can’t access my config. Will try again tonight.

Hey @dannerph… here you go… in stages, as its a bit complicated as I have it set up to attempt to limit API calls to my free 20 hits per day… (usually woks unless I restart HA a bunch of times).

REST sensor (runs once per year… i.e. basically never):

  - platform: rest
    resource: https://api.solcast.com.au/rooftop_sites/***your_ID_here***/estimated_actuals?format=json&api_key=***your_API_key_here***
    method: GET
    name: "SolCast agg"
    value_template: >-
      {{ (value_json.estimated_actuals[0].pv_estimate | float ) }}
    unit_of_measurement: kW
    scan_interval: 31536000
    force_update: true

Automation to “run” the sensor above every 60 minutes (only during daytime):

#Run SolCast predicted sensor every 60 minutes
  - alias: solcast-update
    trigger:
      platform: time_pattern
      minutes: "/60"
    condition:
      condition: and
      conditions:
        - condition: sun
          before: sunset
        - condition: sun
          after: sunrise
    action:
      service: homeassistant.update_entity
      entity_id: sensor.solcast_agg

Automation to set the value of the sensor above to zero 5 minutes after sunset (so it doesn’t stay at the last non-zero value called before sunset):

  - alias: solcast-zero
    trigger:
      platform: sun
      event: sunset
      offset: "+00:05:00"
    action:
      service: python_script.set_state
      data_template:
        entity_id: sensor.solcast_agg
        state: 0
1 Like

thanks a lot!
What is your impression about accuracy of the forecast from Solcast? In specific with and without pushing measurements to their servers?

To get the context: I am planning to integrate PV forecasts in the charging algorithm I use for my BMW i3.

Really depends on your setup. Solcast doesn’t allow you to ‘set up’ a rooftop system with panels facing in different directions (e.g. I have both north and west facing panels)—so initially I set up two accounts, with two systems and aggregated the predictions (it was fairly good except for some early morning and mid afternoon shading effects). So if your panels face only one way and you don’t have any unusual local shading issues, its very good even without calibration. Otherwise, its must better with calibration—correlation on my (now single account, with tuning) is 98%.

Hi all,

This is fab. Was wondering if the sensor could be amended (or a new one created) to show a forecast of tomorrow’s potential PV output.

https://api.solcast.com.au/rooftop_sites/site/forecasts?format=json&api_key=api-key

The json format comes out in 30 mins slots, so any thoughts on how I could pull that together into a single figure?

ie, run at say 9pm and pull out the total kwh forecast for tomorrow?

Any clues would be appreciated… new to all this. :slight_smile:

Hi @Zarch… almost certainly possible… and something I was considered (or rather, the opposite… using a single API call for the previous days data, so I go the bull data set rather than hourly spots).

However, I kinda put it in the to-hard basket because of the difficulty in passing the full chunk of data with time conversions in the “history” of a sensor.

Hi all,

It should be feasible to write a small module for home assistant that pulls the data only once or twice a day and converts the data to the correct timezone. I will put it on my todo list!

For historical view that should not be a problem as this module will probably implement a senor that pushes the previously fetched data at the time of the forecast (use case of @AussieByrd). For a preview for the upcoming day, I do not know any home assistant internal data model that could be used here (use case of @Zarch). The most similar data model is the one of weather forecasts (only temperature, humidity nothing about energy). Home Assistant, for my understanding, is a real-time system that does not provide an option to insert predicted values in sensors.

BR
Philipp

2 Likes

@dannerph if you can pull something like that together, that’d be awesome! Let me know if there’s anything I can do to help out.

1 Like

I now have a basic module consuming only the api_key and the resource_id of the rooftop site:

solcast:
  api_key: xxx
  resource_id: xxx

It can already fetch forecasts and the estimated actuals (POST of measurements are prepared but WIP. My idea is to configure an entity_id in the config file from which the script fetches the data for automatically pushing it to solcast).
From the forecasts, it calculates the expected total energy of the next day as separate sensor in kWh (I have no idea how to store and show a timeseries, so I aggregated the energy as this provides a good starting point to see how the energy production for the next day will be).
From the estimated actuals values I could easily show the latest value, but then I need to update every 30 min to get the data in highest resolution which will overshoot the API limit. As far as I have seen, it is not possible to store historical values within Home Assistant, always only the current state. One option is to reduce the fetching rate in order not to overshoot the API limit, second option I need to dig deeper into Home Assistant Core. Any ideas?

1 Like

Reduced fetch rate was the way I went when I ran into the same kind of issues—so not sure if I’ll be much help!

Sounds like you’re going great though—total forecast should work pretty well for @Zarch’s use case.

1 Like

Short update:
I now checked how state values are stored in the database (in the recorder module). The listening EVENT_STATE_CHANGED event, that triggers storing the data from an entity (sensor) to the database and hence for the history graph, is created in core.py:984. The important last_changed datetime, however, is set to the processing time.
As short conclusion: If the solcast component cannot push directly to the event bus, we cannot get historical data in the database (e.g. for only one request to the solcast API per day). According to the documentation it should be possible to fire directly on the eventbus. I will have a look on it the next days.

UPDATE: I got a small test code running that feeds in past data, so technically it is feasible! Future data however is not automatically shown in the history graph. I will probably use the aggregated energy per day as explained before.

2 Likes

This all sounds fantastic, great work. Please let me know if you need any help testing / checking.

I’m pretty new to the intricacies of HA, but will try. :+1:

Short update [EDIT]:
I found some time to finetune the module. Here is the current state.

What is running:

  • Inserting estimated actual (historical data) with only one API call
    grafik
  • Calculating the daily forecasted energy for the next days (performed every day at midnight)
    grafik
  • smart fetching interval (optimal usage of the API limit)

Todos:

  • automatic pushing of measured PV production for tuning of the solcast forecast

Once the Todos are finished I will put the code on GitHub so you could test it!

1 Like

This is looking GREAT! Thanks @dannerph

@AussieByrd: Do you know, if pushing measured data values (HTTP POST) is also counted for the API limit?