Making solar forecast work with Octopus integration

I have solar panels and a heat pump, which heats water in a hot water tank. I use the Octopus Agile tariff, which means that my energy price varies every half hour.

There is a very good Octopus integration for Home Assistant, which I have set up. One of its features is a sensor which turns on when the electricity import price is the lowest for a fixed period in the next 24 hours. This can be used to turn on the heat pump’s water heating function in the best 2:30h slot in any given day.

There is also an action in this integration that allows the price weightings to be adjusted (“register_rate_weightings”). This allows for each half hour’s price to be adjusted by a factor depending on some data.

What I want to do is use a solar forecast to register these rate weightings: for every half hour in which the forecast says that the solar panels will be producing more than 2kW of power, I want the effective price to be halved for the purposes of calculating the optimum time. This is because, if I am producing more than 2kW, the water heating is likely to be run entirely from the solar panels, and the price that I will be paying will not be the import price, but rather the opportunity cost of not exporting energy from the solar panels in that period. Typically export prices are about half import prices. Unfortuantely, the Octopus integration does not allow using the export prices directly in only some hours for detecting a target rate.

However, I am struggling to find a way to make this work. In particular, I am struggling to find any reliable way of extracting half hourly (or even hourly) solar forecast data. I have Forecast.solar set up, and can see the forecast, hour by hour, in the energy tab, which is, although not perfectly accurate, accurate enough for me to use for this purpose.

The actual sensors available from Forecast.Solar, however, do not allow for hourly power production forecasts, even though the Forecast.Solar API provides these data.

I have spent many hours looking at topics dealing with this issue, such as this discussion here, but I cannot make any of the code snippets given in those topics work, either because some knowledge that I do not have (and cannot find because there is no clue as to what sort of information is missing) is implicitly assumed by whoever posted them, because there is an error in the code, or because the code is no longer compatible with the current version of Home Assistant: I know not which.

For example, in this post, there is a code snippet for creating a “rest” sensor. Adding that to configuarion.yaml does not work and produces errors. No clue is given in the post about where to add this code snippet. There is a second code snippet given, but it is unclear how this relates to the first nor to which file or files that this snippet should be added and what, if any, further steps are necessary to allow this code to be executed at the appropraite time(s). Adding it immediately after the first produces syntax errors.

I have not yet got to the stage of working out how to make the data work with the Octopus integration as I have not got as far as exposing the data yet.

Does anyone have any idea about where even to begin with this? This seems as though it is far more difficult than it should be.

This is too complicated for my skill level, but I can give you what I got (:

Just to understand this, you want to change your tarrif, because you use your solar power to heat?
Is this even necessary, because your costs are calculated from consumption data and if you dont use grid power you dont need to pay for it.
Or do you want to control your heating when the forecast is good? Then you can take the “energy - next hour” sensor from forecast.solar (you may need to enable the entity) and then run an automation everytime that value changes and control your heater.

General answers:

The Rest sensor needs to go under configuration.yaml like this:

sensor:
  - platform: rest
    name: "Solcast Forecast Data"
...

and the second one under the rest one can go under your settings, devices, helpers and there to a new template sensor.
As the post ist 3 years old you may need to check if the api call is still up to date: API [Forecast.Solar]

When you create the template sensor you will see " This template listens for the following state changed events: " so every time the source (rest) sensor updates your template will run.
For debugging purposes you can also paste the code under developer tools template and strip out parts of the code to understand what it does or provide a demo input.

Also, to lead you to more options, other good or even better forecasts are solcast and open meteo solar forecast.

Thank you for your reply. I have attempted to add the rest sensor to configuration.yaml thus:

sensor:
- platform: rest
name: "Solar Forecast Data"
json_attributes:
- forecasts
# TODO: Give more accurate details for this
resource: https://api.forecast.solar/estimate[LOCATION REDACTED]/0/0/8
method: GET
value_template: "{{ (value_json.forecasts[0].pv_estimate)|round(2) }}"
unit_of_measurement: "W"
device_class: power
scan_interval: 8000
force_update: true

But this gives the following error when my Home Assistant is rebooted:

Invalid config
The following integrations and platforms could not be set up:

Please check your config and logs.

Do I need to add the second code snippet before the first will work?

To answer your questions:

Just to understand this, you want to change your tarrif, because you use your solar power to heat?
Is this even necessary, because your costs are calculated from consumption data and if you dont use grid power you dont need to pay for it.
Or do you want to control your heating when the forecast is good? Then you can take the “energy - next hour” sensor from forecast.solar (you may need to enable the entity) and then run an automation everytime that value changes and control your heater.

What I want to do is work out the optimum time for my water heater to come on in advance, taking into account both the varying tariff and the estimated solar production. For example, if the import tariff between 03:00 and 06:00 is 15p/kWh and that is the lowest tariff in the day for a block of ~3 hours, then, without taking into account solar, the water heater should turn on at 03:00h for 3 hours. But suppose that the the import tariff between 12:00h and 15:00h is 20p/kWh and the export tariff between those times is 10p/kWh. If I will be generating enough solar energy between those times to run the water heater, then it will be cheaper to run the water heater between 12:00h and 15:00h rather than 03:00h and 06:00h because it will only cost me 10p/kWh, as I will be running the water heater only from my own production, and thus only losing out on export at 10p/kWh rather than paying the 20p/kWh import price. This means that the water heater should not run overnight, but wait until the afternoon.

But, if the export price during those times were, for example, 16p/kWh, then, for the same level of solar forecast/production, it would still be better to run the water heater overnight. Alternatively, if the prices were the same as in the first example, but solar production were forecast to be very poor during the day, then it would still be better to run the water heater overnight.

So, the system needs to know hours in advance whether, taking into account both price and solar production, the best time to run the water heater is at night or during the day (and, specifically, at what time that it is best to run the heater).

The Octopus integration already provides code to find the optimum rate for exactly this purpose. It also has an action called “update rate weightings” to apply a multiplier to the rate at certain times of day, and the documentation mentions that one of the things for which it is intended to be used is solar forecasting (i.e., exactly my case), although the code example given is for grid carbon intensity forecasting.

All that I am trying to do is use this Octopus integration feature with solar forecasting in the way that it was designed to optimise my water heating timing.

Ok I see. First about your code, you need to format it with spaces

sensor:
  - platform: rest
    name: "Solar Forecast Data"
    json_attributes:
      - forecasts
    # TODO: Give more accurate details for this
    resource: https://api.forecast.solar/estimate[LOCATION REDACTED]/0/0/8
    method: GET
    value_template: "{{ (value_json.forecasts[0].pv_estimate)|round(2) }}"
    unit_of_measurement: "W"
    device_class: power
    scan_interval: 8000
    force_update: true

No, the second one is based on data of the first one.

For the other things I think that it is a little bit complex for Home Assistant alone. Did you take a look at EMHASS? It seems a little bit complicated to set up, but from there it should do what you want. It is well maintained and if you need more help you can ask the specialists there.

1 Like

Although Home Assistant, at the main entry website, continues to promote the concept of home energy management with solar production predictions, actually using the HA solar forecast in any meaningful way is ‘difficult’, to put it mildly.

The standard HA solar forecast integration - forecast.solar - does work but as you comment, I also find it too inaccurate to be of practical value. The HA integration works by pulling down the latest forecast each hour, computing the value for the energy display and the given forecast entity sensors, then throwing the data away. There is (currently) no way of either holding or obtaining the time-power forecast array.

Other than the next hour prediction, all solar forecast work of any practical value necessitates doing all the work yourself.

There are three things required here

  • the solar forecast for the day ahead
  • the utility pricing for the day ahead
  • a computation algorithm for processing the data

It is easy enough to run forecast.solar API call yourself. I have tinkered with rest sensors in HA, but given the difficulty in debugging and then dealing with the data, I use Node-RED throughout.

I have a Node-RED flow to pull and maintain solar forecasting. I run this for forecast.solar, but have switched to using Solcast which is accurate, and importantly works on half-hour periods not hourly.

I have a Node-RED flow to pull and maintain Octopus agile pricing. I use the (great) Octopus Integration, but I don’t have Agile as my tariff, so this is for interest purposes only - but it works and provides a full day-ahead pricing in an array, and also provides the ‘best period’ information for the lowest time periods.

Holding all of this in Node-RED means that it is not difficult to pull both together.

Actually, I think this is the most difficult part. If you can translate this into an algorithm, I am sure that we could make it work…

If you want to go down the rest-sensor path, then firstly you need to get the API call to forecast.solar correct. Your call appears to be missing most of the parameters, and will reject unless correct.

If you want to use Node-RED (which is not popular in the HA world, but I contend is so much easier to use than Jinja and YAML) then this flow will call and process the solar forecast array for you. [you will need to change the location, elevation and azimuth, and array power values of course]

[{"id":"23d7b6836ea4b7e8","type":"change","z":"dc25556591dc08be","name":"Parameters","rules":[{"t":"set","p":"parm.latitude","pt":"msg","to":"51.5132","tot":"num"},{"t":"set","p":"parm.longitude","pt":"msg","to":"0.1589","tot":"num"},{"t":"set","p":"parm.elevation","pt":"msg","to":"35","tot":"num"},{"t":"set","p":"parm.azimuth","pt":"msg","to":"-107","tot":"num"},{"t":"set","p":"parm.power","pt":"msg","to":"2.19","tot":"num"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":1540,"wires":[["fe85feeb01d19e78"]]},{"id":"fe85feeb01d19e78","type":"http request","z":"dc25556591dc08be","name":"API FCS","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.forecast.solar/estimate/{{{parm.latitude}}}/{{{parm.longitude}}}/{{{parm.elevation}}}/{{{parm.azimuth}}}/{{{parm.power}}}","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":440,"y":1540,"wires":[["5a3ad33d60a7a2aa"]]},{"id":"5a3ad33d60a7a2aa","type":"switch","z":"dc25556591dc08be","name":"OK?","property":"payload.message.code","propertyType":"msg","rules":[{"t":"eq","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":570,"y":1540,"wires":[["a1d21e482bb9ccb3"],[]]},{"id":"0ddcf6c2516d884e","type":"inject","z":"dc25556591dc08be","name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":false,"onceDelay":0.1,"topic":"","payload":"","payloadType":"date","x":140,"y":1540,"wires":[["23d7b6836ea4b7e8"]]},{"id":"28f09ddeefc61d70","type":"debug","z":"dc25556591dc08be","name":"debug 31","active":true,"tosidebar":true,"console":false,"tostatus":false,"complete":"payload","targetType":"msg","statusVal":"","statusType":"auto","x":980,"y":1540,"wires":[]},{"id":"a1d21e482bb9ccb3","type":"change","z":"dc25556591dc08be","name":"Get array of time-power","rules":[{"t":"set","p":"payload","pt":"msg","to":"(\t    \t/* lift results, prune for today and build array */\t\t    $p:=payload.result.watts;\t    $today:=($keys($p).$substringBefore(\" \")~>$distinct)[0];\t    $spread($p).(\t        $entry:=$;\t        $date:=$substringBefore($keys(),\" \");\t        $time:=$substringAfter($keys(),\" \")~>$substring(0,5);\t        $power:=$.*;\t        {\"date\": $date, \"time\": $time, \"power\": $power}\t        )[date=$today and power >0]\t)","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":770,"y":1540,"wires":[["28f09ddeefc61d70"]]}]

Thank you both for your help.

Dealing first with the code snippets from the other thread, I have now fixed the errors with the code in configuration.yaml using the correct whitespacing as indicated.

However, on attempting to create the template sensor from the second code snippet, I am getting new errors. I use the following code when creating a template sensor through the UI as suggested:


{%- set awattar_all_list = state_attr('sensor.solar_forecast_data', 'forecasts') | map(attribute='pv_estimate') | list %}
          {%- set values_all = namespace(all=[]) %}
          {% for i in range(awattar_all_list | length) %}
           {%- set v = (awattar_all_list[i] | float |multiply(1000) ) | int(0) %}
            {%- set values_all.all = values_all.all + [ v ] %}
          {%- endfor %} {{ ([states('sensor.APF_Generation_Entity')|int(0)] + values_all.all)[:48] }}

This results in the following error message being given in the template sensor preview field:

Sensor None has device class ‘None’, state class ‘measurement’ unit ‘W’ and suggested precision ‘None’ thus indicating it has a numeric value; however, it has the non-numeric value: ‘[0]’ (<class ‘homeassistant.helpers.template.gen_result_wrapper..Wrapper’>)

Have I misunderstood the steps necessary to create a template sensor from this set of data? (The actual get command is correct, incidentally: I just redacted my location data in the posted snippet. I have tested it through a web browser and I get each hour’s production forecast in watts as needed).

Turning to the more general points raised by Biscuit Geoff, I am aware that using the solar forecast data is very difficult - that is why I am posting this thread! It is far more difficult than it should be and it is not really clear why it was done that way. I do not find a major problem with the accuracy of the data (obviously, it is not perfectly accurate, but I think that it is close enough for my purposes): the problem is more the accessibility of the hourly data.

Actually, I think this is the most difficult part. If you can translate this into an algorithm, I am sure that we could make it work…

I can write some pseudocode for this if that would help.

for each forecast in forecasts
 {
  if(forecast.watts > 500 && forecast.watts <= 1000)
  {
    octopus_integration.update_rate_weightings(forecast.time, 85%);
  }
  else if(forecast.watts > 1000 && forecast.watts <= 1500)
  { 
    octopus_integration.update_rate_weightings(forecast.time, 75%);
  {
  else if(forecast.watts > 1500 && forecast.watts <= 2000)
  {
    octopus_integration.update_rate_weightings(forecast.time, 66%);
  }
  else if(forecast.watts > 2000)
  {
    octopus_integration.update_rate_weightings(forecast.time, 50%);
  }
}

In this pseudo code, I imagine that update_rate_weightings takes two parameters, first of all the time for which the rate takes effect and secondly the percentage of the price to apply: e.g., if 50% is specified, the price will be assumed to be half the base price for the purposes of optimisation. I also imagine that forecasts is a collection class populated by data from the solar forecast giving a time/date and a number of forecast watts for that time/date.

A more sophisticated version would have some arithmetic to perform a linear interpolation between the 85% and 50% values rather than using fixed thresholds as here, but this pseudocode is probably simpler to understand.

The basic algorithm is not the difficult part: the difficult part is understanding how to expose the data necessary for the alogrithm to work and make it work in the correct way for the Octopus integration (e.g. turn hourly data into half hourly data by repeating the hourly data twice for each half-hourly slot).

For the other things I think that it is a little bit complex for Home Assistant alone. Did you take a look at EMHASS? It seems a little bit complicated to set up, but from there it should do what you want. It is well maintained and if you need more help you can ask the specialists there.

I did look at this - but I would still need to do all the work of exposing the solar forecast by hour to make this work and do all the work of creating the optimum rate finding algorithm which is already included in the Octopus integration, so this does not seem as though it is capable of solving the underlying problem of exposing and formatting the solar forecast data in a way that the Octopus integration’s update rate weightings algorithm can process.

Thank you both again for all of your help so far.