Access data from Forecast.Solar

[Biscuit (Geoff)]

I have revisited and completely rewritten the Node-RED flow (using JSONata). Improvements mean that it automatically picks up any number of planes, and I have improved the forecast analysis (power levels) to include tomorrow as well as today.

Currently going through testing - JSONata can throw up some surprises. Takes a long time to test as it takes two full days to build a complete data array. When I have finished testing my plan was to re-publish as a new ‘my project’ thread.

As to the one-hour-out issue, this seems to be a long running problem where it is seen with some setups and not others. First place is to check the forecast.

• what location are you (roughly - UK/ Europe)?
• what timezone are you on?

If you look at the forecast data, it shows the day start (sunrise) and day end (sunset). From your screenshot the period is 6.51 to 20.04. First question is, does this line up with your local sun times?

If not, then the issue is with the forecast.
If yes, then the issue is with the actual data.

[ffm777 (Carlo)]

@Biscuit

I have revisited and completely rewritten the Node-RED flow (using JSONata).

That is great to hear! I’m very much looking forward to your revisited code.

I’m located in Frankfurt. So this is CET. The sunrise and sunset times in the forecast data are right! So there must be an issue with the actual data. Can you give me a hint where to search for the error? Thanks!

Quick update: I just checked the actual data. This one is correct, too.

[Biscuit (Geoff)]

The hours and hours that I have spent on this just does not bear thinking about!

First check - solar forecast start/end matches local solar day.
You (currently) expect sunrise at 06:50, and sunset at 20:05 (rounded). This agrees with the solar sun calculator suncalc.org for Frankfurt. I am in Bristol UK on BST (UTC+1), being about 10 degrees west of you my solar day is much the same, currently sunrise 06:32, sunset 19:52.

Next, ‘actual power’.

Here is my solar power history for Tuesday. A sunny day generated 20kWh total.
Top line is solar power sensor from inverter. Power started around 07:00, and ended around 20:00 (which is OK for a solar day from 6:30 - 20:00). The power line is correct, as it comes from my inverter and agrees with what I see on the inverter UI screen.

I use this sensor in a Reimann sum to get solar energy (second line). This is correct - difference start to end of the day is 20kWh. The sensor records total energy, and this agrees (within tolerance) with my ongoing inverter monthly/yearly records.

Now I want to plot the ‘actual’ on my graph for each hour.

Here I show the utility meter sensor for energy hourly at +30 minutes offset.

Aside: why offset 30 minutes?
The solar forecast is power and I also want to plot actual energy to easily compare them. Energy is not power, however the average power for a one hour period is the energy, so I can plot both together, as long as I line up the periods. Actual power at 9:00 has to be compared with energy from 8:30 to 9:30 to match.

Check utility meter: From power graph, actual power at 8:30 is ~1330 W, and at 9:30 it is ~1630 W, so an average is 1480 W, or approximately 1.5kW over 1 hour = 1.5kWh.
If I update the solar forecast at 10:05, and I read the utility meter entity, the entity 'last period’ is the value when it was last reset at 9:30, and this gives 1.5kWh, so I can use that to plot 1.5kWh on the graph at 9:00.
Double check - power at 9:00 is ~1500W (as expected on a linear interpolation between 8:30 and 9:30) so that matches with average power and energy between 8:30 and 9:30. I can plot power 1500W at 9:00 and energy (1 hour) 1500Wh at 9:00.

So my actual energy values are correct.

Here is a HA history plot of solar power for the east and the west plane, and a plot of the solar forecast (power) for the current hour.

To make this clearer here is an excel scatter plot for just the east-north-east plane showing actual and forecast.

In theory, at this point, the ‘actual energy’ and the ‘forecast power’ should now line up. They don’t - you can clearly see the huge disparity between forecast and actual. The expected rapid rise at the start of the day from sunlight directly on the east facing plane is curtailed (by horizon shading) for about 60-90 minutes. Also, my planes are on a garage roof with a nearby south-facing house wall reflecting indirect sunlight back onto the planes. This increases the midday and early afternoon generation.

Now I look at your graph.
I note that your solar day starts at 06:51, so at 7:00 the sun is just rising. On your graph your actual for 7:00 is about zero, which is what I would expect. However, and this is the interesting part, your forecast for power suggests 5500 W.

Whatever your situation, I cannot believe that your system is going to be generating over 5kW just 9 minutes after the sun has risen!

Another check is to look at solar noon - for a south facing plane both forecast and actual should peak at solar noon. For east/west planes this will not be the case, but my two equal capacity, and almost equally offset from due south, when added together should peak around solar noon. Your solar noon should be approx 13:30. If you have balanced planes then your actual looks reasonable peaking at 13:00, but the forecast peaks before 12:00.

So I suspect the forecast is the problem. Is this being plotted one, even two hours earlier than it should be?

My thoughts are that planes that do not face due south produce poor forecasts, with the error being at the start for east and end for west, and becoming more pronounced further away from due south. There seems to be no easy fix for this error (I just can’t add in or subtract, or shift the curve left or right). I have managed to get a reasonable match for myself by getting forecast power and actual energy to time-match, by using the damping factor, and by having two planes 180 degrees opposing which goes some way to cancelling out the errors between them.

[ffm777 (Carlo)]

@Biscuit

So I suspect the forecast is the problem. Is this being plotted one, even two hours earlier than it should be?

That is exactly the problem. I have 4 planes. The south plane is not facing perfectly to the south but is angled 20 degrees to the west (and the other planes are “off axis” by 20 degrees as well). That leads to a peak at around 13:00. You can see the power of the 4 different planes from April, 5th (which is the day in the chart I posted above) here:

What makes matters even more complicated in my case is that I’m close to a forest which shades my east facing modules completely in first 2 hours after sunrise (in April). You can clearly see this in the graph above. The south plane is also fairly complex as I have a couple of modules sitting on my garage which are less angled than the ones on the roof and get shade from my own house in the afternoon (handled by optimizers). So maybe my situation is just too complex to produce a good forecast. Still, I find it somewhat puzzling that the forecast on some days looks almost perfect with the exception that it appears to be shifted two hours to the left.

[Biscuit (Geoff)]

This thread started with a question about getting the data out of forecast.solar integration, and whilst this is slightly off-topic, for me it still relates back to a basic question of what can we do with, and how can we improve the solar forecast integration, other than look at it in the energy dashboard?

There are, I suggest, several challenges with using the integration and forecast. Most of them I have now managed to understand, and I have some ideas as to how to overcome some of them, but not all.

What I perceive is

• the energy dashboard display is one hour out due to plotting values at the end of the period not at the start of the period (which I why I get and process my own data from API calls directly, and plot the forecast where I think it should be)
• being a forecast, there can only be a potential good match to ‘actual’ for entirely sunny days (no cloud) or entirely overcast days (no direct sun). On days where there is ‘weather’ the forecast will naturally be random. The forecast also changes during the day (understandably, as the weather forecast changes) so a match of forecast-to-actual is always going to be a moving target
• the forecast is problematic for planes not facing south due to shading issues

On non-south-facing planes, my current theory is this.

Solar pv power generated for each plane comes from a combination of indirect sunlight and sunlight shining directly on the plane.

Unless it is a very dark cloudy day, indirect solar pv is fairly independent of cloud cover, related slightly to sun azimuth but mostly to the sun elevation. This results in a smooth power curve from sunrise to sunset, approximately 10-15% of a full-sun forecast.

Direct solar pv comes from the sunlight incident on the plane. For a due-south facing plane this will start obliquely at sunrise, peak at solar noon, and ends when the sun sets. For a due-east facing plane however, the direct-sun curve is shifted so that sunrise at due east (at the equinox) gives rise to facing sun immediately, resulting in an almost step-function start to the solar day.

Now, a solar forecast assumes sun incident on a theoretical plane with a flat horizon. For most practical situations horizon shading will exist (houses, trees, hills) causing a ‘cut-off’. For a due-south plane this ‘cut-off’ occurs at sunrise / sunset when the direct-sun forecast is near zero. For a due-east plane, this ‘cut-off’ occurs at the point of significant direct sunlight, and therefore it has a major consequence on the forecast (for sunny days) for the first hour or so.

To try and model this, I have created the following two excel plots.

• the sky curve shows the power forecast for the full day from ‘general sky’ generation
• the direct curve shows an estimate for the direct (full sun) power forecast. Note that an east facing plane at sunrise (at the equinox) is already half-way through a full curve
• the ‘cut-off’ periods model horizon shading that blocks early/late direct sunlight
• the sum curve shows the indirect and direct added together
• the actual curve shows the effect of adding the indirect solar forecast to only the effective part of the direct solar forecast

The south facing, with minimal impact from shading, which can be adjusted for in the forecast using the damping factor. Timescale is for around the equinox - solar day about 12 hours, solar noon (for my local time DST) around 13:00

And now for an east facing plane. The sky curve moves to peak slight earlier in the day (as the plane faces east). The direct curve now centres somewhere between sunrise (sun face-on but zero elevation) and noon (oblique but maximum elevation). The direct curve ‘theoretically’ extends beyond sunrise, so the start of the day has the rapid rise from zero. The morning shading now has a significant impact on the first hour or two, and results in the step-change we see in the actual power generation.

Using this crude model, I can match each of your planes quite well, assuming shading on the east.

To improve solar forecast accuracy for planes not facing due-south, I believe that it would be beneficial to

• break the forecast into direct and indirect components
• allow for a cut-off window (ideally based on solar elevation above local horizon) that removes the direct solar element
• recombine the full forecast from indirect, and modified direct

A split (direct/indirect) forecast would also have the benefit of indicating the likely difference due to cloud cover. Full sun will generate at the higher level (clear-sky forecast). With cloud cover the generation power will drop to the lower (indirect only) level.

My solar power generation for a day with passing cloud, with an (hand drawn) estimate of the indirect and indirect + direct curves showing how the power drops from one to the other when the sun is covered by a passing cloud. This curve shows very clearly the morning shading effect too.

For some of the shift between forecast and actual, I can now attribute to the shading impact. However without significant further study I cannot explain the two-hour shift in your earlier posting. I don’t have the skill and knowledge to be able to pursue this further, but I had hoped to be able to find an API that I could use to at least obtain a basic clear-sky power forecast for my time/location/plane. That would at least allow some general checking of the forecast values against time.

EDIT PS
In answer to my issue regarding horizon shading, of course the forecast.solar API offers the use of ‘horizon’ parameter to implement a practical horizon, something that I did not pick up on last year.
Room for yet more experimentation!

[Makkie1969 (Marius Kruger)]

Hi Geoff
I am using Forecast.Solar, outside HA, to manually check against my actual pv production. I am very new to HA and want to integrate it into my HA. Do you mind sharing your full code from getting the Forecast.Solar data and creating your graph?

[Biscuit (Geoff)]

Code is in Node-RED. I have recently completely re-written this and am still testing it. When I am happy with it I will publish as a new ‘my project’ thread.

[comm]

Accidently deleted:
Hi Geoff,
I wondered if your re-written Solar Forecast module was ready for sharing?
Thanks in advance.

[Biscuit (Geoff)]

I started to pull this together again just yesterday, so you are in luck!

For anyone else interested… @ffm777 , @Beny_44_44, @Makkie1969 et al

I had a very long revisit of my Node-RED code, completely updated it and set it up for testing last May. Every time I thought it was good to go, another ‘critical update’ or small problem came along, and I had to wait to see if it all still worked, hence the rather long wait.

I have moved on to other things, but this is my latest code for calling the Forecast Solar API directly. This is now entirely in JSONata, which has improved the ability to handle 1 to n planes, added more analysis output data for both today and tomorrow, and fixed a couple of bugs I found.

Disclaimer: this is working for me, but it may not work for you and you will have to deal with setting up Node-RED and all the various sensors and settings yourself.

If you are updating from my earlier version my suggestion is to disable all the old trigger, sensors, and configuration nodes, to redeploy and restart NR (so as to clear the sensor entities from HA), import the new code, and then update the parameter settings and HA server in the sensor configuration nodes.

All the context data should also be deleted so as to force the code to regenerate a fresh copy.

The sensors should reappear, but check that they are correctly named and not ‘_2’ copies: a full disable, deploy, restart HA and HR may be required to clear the old sensor (HA entities) in such cases.

Requirements: as before, this needs up to date Node-RED. It also requires the Cron-plus node (node-red-contrib-cron-plus) if you want to schedule the exact time in the hour for API call update, and it requires the time-date integration so as to be able to read the current local and ISO time.

• cronplus node is the one I prefer to use to trigger the hourly update, but this can be swapped for the standard inject node with a repeat set to every 60 minutes
• the time-date https://www.home-assistant.io/integrations/time_date/ integration should be added to configuration YAML file, the date_time_iso value is the one used by the code
• the half-hour solar actual sensor is explained above in previous posts

Here I am now using the forecast solar API returned UTC and local times to work out the time zone setting, and also using an entity state node to get the date_time_iso so as to obtain time zone settings. This all works, but I may modify the JSONata again to do the UTC to local time another way and avoid using the date-time platform… I am still experimenting with JSONata code to obtain the local time (UTC time is easy - just use $now()).

NOTE that Forecast Solar decides the local time zone / DST given the location settings used for the planes in the API call. As usual, this will all only work correctly where the HA machine local settings match the plane location settings. (I have tested my code for my location in the UK, Frankfurt in Europe, and had a stab at Pacific Daylight Time in LA, but it is challenging to fully test time zone code)

Code:

• All NR code has been scrubbed. You will need to add back in your own HA server to each of the sensor (entity) configuration nodes.
• Location settings for the planes have been invented. You will need to add back in your own settings for your plane(s). I have experimented with the ‘horizon’ setting to see if it improved the accuracy of the forecast. It appears to have no effect whatsoever, so I don’t think this is working, but I am leaving it in as an example - my horizon is very restricted so do remove or adjust this!
• The context variables save & read actions have been moved to change node entries. This makes them visible and much easier to select the context store (memory / file) if you have more than one. I use persistent file storage for the forecast table.
• key nodes have been disabled so as to make importing the code less disruptive. Enable configuration, sensor and display nodes as required.

When imported and configured it should look something like this:

The context data values have been extended to include much more data, particularly for the analysis (power level) table.

Today was not a good solar day. Forecast Solar, at the end of the day, is giving 3.4 kWh forecast (backcast), however my analysis uses the historic record values to give 4.1 kWh, which exactly matches the 4.1 kWh actually obtained.

For myself I use this in HA as a visual display only. I am now looking at the UK Octopus Energy ‘Flux’ tariff, and need to get an accurate forecast for ‘tomorrow’ to decide if I need/can charge my solar battery from the grid during the night, still allowing for space to capture excess solar.

Problems? Requests? Comments? Can’t get it to work?
Please note that I am retired (which means I am very busy). I am doing this just for fun (which means that, although I like to ensure my code is tested and it works, there is no warranty). I can answer questions, however most answers are already out there.

Many thanks to those who are using my code for your positive feedback. Good luck to you all with your solar (forecast) projects.

[Biscuit (Geoff)]

Edit:

The JSON code extract is too big for the block upload limit here. In trying to split the file into two I have also hit a 3-reply limit.

The code is quite sizeable, and I have included a good number of comments in the JSONata (to help me debug it as much as to help anyone else) so I am going to have to find another way to upload / share this.

If anyone knows a good way to add large JSON files to a post here then please do let me know!

[comm]

Thank you so much for sharing - even your descriptor is helpful in my understanding.

Maybe share on a GitHub project? Looking forward to the JSON!

[Biscuit (Geoff)]

I have joined the ‘Node-RED’ forum, which allows for uploading files (but this is not available to new members) so I have tried another approach and added the flow to the Node-RED flows. This uses GitHub, but was easier to do in the Node-RED site. In time I may get round to Node-RED projects; just more to learn!

You can find the flow here: https://flows.nodered.org/flow/bbde517b1e592ce804f553da80dbb4c4

Do let me know if you have issues upload and getting it to work.

I am not intending to spend further time on this flow as I only use it for display which is working nicely for me, and I have other flows that I want to work on.

EDIT April 2024.
I have again updated the code, but am now using Github (much easier to manage releases I now find!)

For anyone interested, the latest Node-RED code for this flow can be found as follows:

Access data from Forecast.Solar

Code Update: If anyone ever reads down as far as here, I have refactored the Node-RED flow I posted here (mostly tidying) and have finally got around to pulling it all together with some documentation. It is now on Github. https://github.com/oatybiscuit/Node-RED-HA-ForecastSolar-JSONata?tab=readme-ov-file#node-red-ha-forecastsolar-jsonata I have removed the need for the cronplus node (just use a standard inject and a timer), the moment nodes, and have also removed the need for a half-hour off…

[DvdE]

Thank you for this post. I don’t like to use Nodered and have the API working in HA using REST sensor

• platform: rest
  resource: !secret forecast_locatie # https://api.forecast.solar/:apikey/estimate/:lat/:lon/:dec/:az/:kwp #replace with your own values
  name: “Solar Forecast (rest)”
  json_attributes_path: “$.result”
  json_attributes:
  • watt_hours_day
    value_template: “OK”
    scan_interval: 600

However, I get stuck on defining the data_generator to produce the apex charts. Any suggestions how to define this?

[Biscuit (Geoff)]

Yes, not many HA users seem to like Node-RED, however it does deal with data manipulation very well.

I don’t like to use HA for API stuff, but I assume that your sensor is only parsing the JSON fields ‘watt_hours_day’. As these are just the total Wh for today and tomorrow, I don’t see that you have much to plot.

If you want a time-graph of data, then you will have to read in something like the watts array (time stamp / watts as a pair). Then you will have to store this in a sensor attribute somewhere, then all it takes is some JS script for the data generator - just build a new array from your stored array, where the array is an array-pair of the date object (millisecond timestamp) and the value (watts) to plot.

You can find the details of what the graph requires here
https://github.com/RomRider/apexcharts-card?tab=readme-ov-file#data_generator-option

it has a good explanation of how to get the data from sensor attributes.

[DvdE]

Thank you for link. It works with following:

data_generator: >
      return Object.entries(entity.attributes.watt_hours_period).map(([time,
      watt]) => [new
      Date(time).getTime(), (watt)]);

[samaerts (Sam)]

Hi Everyone,

New to this topic, still going through documentation and comments which already looks promising
What I didn’t come across is the possiblity to insert your own data. In Belgium, the electricity network provider shares forecasts through an api

Photovoltaic power production estimation and forecast on Belgian grid (Near...

Any ideas on best practices for integration of this data?

[davidusb]

Look at the solution to this thread.
A REST sensor should do the trick for what I think you are trying to do

[AlexanderLL95]

How can you show the data on an apexcharts in HA?
I tried the following configuration, but doesn’t show anything.
Is there somenone who already did it?

type: custom:apexcharts-card
graph_span: 2d
span:
  start: day
  offset: '-1d'
header:
  show: true
  show_states: true
  colorize_states: true
series:
  - entity: sensor.fc_table
    name: Solar forecast
    data_generator: |
      return entity.attributes.Fchours.map((time, index) => {
        return [new Date(time).getTime(), entity.attributes.Fcwatts[index]];
      });

[Biscuit (Geoff)]

It all works for me, but then I did write the Node-RED flow code that you appear to be referencing.

• Are you actually using the Node-RED flow to generate the sensor.fc_table entity?
• Is this table correctly populated with data?
• Does the entity use ‘Fchours’ or ‘fchours’ (I think it did before I changed it to lower case…)
• Do you have at least two-days worth of data? Your settings of span 2d and offset -1d I believe only show ‘yesterday’ and if you have just started using this, there will not be any ‘yesterday’ and thus nothing will show…

Here is my current Apex Chart setting.

And here is a different set of settings, used to just show ‘today’. Note that the
critical settings for the graph ‘window’ onto the data are graph-span (how much to show), and span- end/offset (where to anchor to)

And yes, my one-day graph title needs changing. This is only a test chart I have left in my development area…

Edit: I had missed that you are using start: day (I use end: day) so your graph span settings are correct for showing today and tomorrow.

Back to the first question - do you have the data in the sensor?

[AlexanderLL95]

Stupid mistake.
The attributes where with capital written in the entitie FC table, but for the graph the attributes needed to writen in lower case. Graphs are now working very well in HA.