Dear community.
I seek help on this one. I need to extract the data from the dashed black curve shown in this graph below. I need to use this data in a template. This is the solar PV production forecast from the Forecast.Solar integration as shown when added to the Energy Management dashboard:
Any help on this will be much appreciated.
A solution rather depends on exactly what data you want and how you plan to use it. Do you need the entire forecast, just one data point, the maximum point, or to know when the line is above a set value?
As far as I can tellâŚ
The dotted line is a plot of the watt-hours returned by an internal integration call to the forecast website API, and the returned data is used to populate the graph and the 10 integration sensors/entities but not otherwise stored anywhere in HA.
Three possible solutions that I have tried:
The entities to look at are Estimated Power Production - Next 1/12/24 Hours (in Watt), which is actually the power in watts at the hour 1/12/24 hours ahead. These are disabled by default.
Note.
The power production values are watts at the hour, and the dotted line is watt-hours for each one hour period. Watt-hour (energy) is the average of the two watt (power) figures at the start/end of each hour, for the hour period, and can be easily calculated (more complex for the first/last part-hour after sunrise/before sunset when period is not a full hour).
The dotted line graph seems to be one-hour out (shifted to the right from where it should be). I think this is a mistake.
The integration updates once per hour, randomly, so you may find that you are getting the data at xx:55 when you might like it at xx:05. My HA integration has just updated at 14:21, so the 24 hour ahead entity in HA now has the value for the solar forecast power (watts) at 14:00 tomorrow.
The forecast itself is updated regularly. Not necessarily hourly, but certainly if you read the 24-hour ahead figure from HA history for 14 hours ago it will be the forecast for 10 hours ahead of now, but 14 hours out of date. (if that sounds complicated, itâs because it probably isâŚ)
See bottom left - this is just an HA history graph card for the âestimated power production next 12 hoursâ entity. I have two integrations running (for two solar planes). The WEST line shows that, at 05:21 this morning the entitiy updated to a value of 978 watts. So, at 05:xx the forecast for 17:00 was that the power generated would be 978 watts. The most recent API call suggests 845 watts.
I have struggled with adding template sensors in HA, so I am currently working in Node-Red for this, but I have made all three solutions work (more or less). Of any help to you?
Hi, thanks for the answer.
I need the data in the black dashed curve everyday at 5am. But I need the entire forecast, the whole future observations in that black dashed curve.
So I think that your solution #3 should be the only option.
How will you exactly implement solution #3? With a rest api sensor?
If you want the entire (fresh) forecast at 05:00 then yes an API call.
I have this working, but I have used Node-Red flow to do this rather than HA Rest sensor. I have looked at HA Restful sensor, but I decided that it is too challenging for me at the moment (I am having issues with trying to write the json_attribute_path constructions). The difficulty is not in getting an API call to work, it is in being able to do something useful with the data that is returned.
Node-Red allows me to strictly control the number of API calls (there is a limit of 12 per hour), allows me to store and save the data, and for me is easier to write and test simple functions to manipulate the data. This has been very important as I can call the API once and easily save the json return for testing.
I have several objectives
My ultimate goal is to be able to decide (at 01:00) whether to charge or discharge the battery so as to have store for the 16:00-19:00 peak and not to waste solar power. Still working on that one!
I am still very much working on this, but the first objective is achieved. I am returning an array of the forecast from Node-Red back into an HA entity, which I can use to populate an Apex-charts graph.
The second bit I have been trying to do in HA with templates, but again difficulties with getting this to work are sending me back to Node-Red where I know I can write the necessary code.
As this is in Node-Red, I accept that it is not a solution that suits everyone, but I am planning to tidy it up and offer it to anyone who would find it useful - so let me know!
A table of timestamp (date/hour), watts, actual generation returned to an HA entity and used to plot a graph.
Current flow. Runs in Node-Red as add-on. Calls API every hour, for two separate solar panes and saves the data into a flow variable. Then adds last-hour actual and manipulates the forecast into a 72-hour array. Then pulls data from this array into a graphing format, and a power-level event format, returning these to HA using an entity node.
Ok, thank you very much or your help.
Iâm not very fond of Node-Red, I prefer to do that kind of stuff directly in Python. But your very detailed answer has helped to understand how this should be done.
If youâre using these forecasts to manage your home energy I can suggest to take a look at this package and HA add-on that I developed for just that purpose.
The core package: https://github.com/davidusb-geek/emhass
And the HA add-on: https://github.com/davidusb-geek/emhass-add-on
Cheers
You are welcome.
I started using Node-Red last year when I had the solar pv system installed, and I have grown to like using it. Benefit for me is I have NR on my Raspberry Pi as well as my HA odroid, and I can develop on one and transfer to production when tested.
Thanks for the links to your package. I have to say that I am avoiding GitHub as I find it a little too complicated, but I will certainly look at your work.
Ok 
Well for github for the package it just contains the source code and the documentation. The link for the add-on just contains the information on how to install the add-on, you donât have to do anything else with github. Once installed everything will work on the HA web ui. Otherwise you may find the documentation here: EMHASS: Energy Management for Home Assistant â emhass 0.3.17 documentation
Ok an answer to my original question using a rest sensor here:
The answer is for the SolCast integration but the API call is very similar to Forecast.Solar. Posting this for other people looking to do the same in the future.
I would be highly interested in your Node Red code. I also use Node Red for all of my automations in Home Assistant and would love to find an easy way to retrieve the solar forecast data. Thanks!
This is my current code. I have removed all the settings, so it will not work without some reconfiguration.
The API calls need lat & long etc - go to
https://doc.forecast.solar/doku.php?id=api:estimate
for the details. The azimuth in the API is different to that used in the HA configuration!
Note that there is a limit to the number of free calls per hour, so I set this to run only once per hour.
I convert all times to local, and I use Dat/Time formatter node (just easy to use) - you will need to change the location settings in these.
I use persistent storage for my Forecast array object, but I have removed this you may not have it.
For the actual energy (Wh), I use a Riemann Sum on my inverter power to give energy, then I use a utility sensor with hourly reset and 30 minute offset to measure Wh by the hour. Since the forecast is power on the hour, I want to display energy as well at the hour, so rather than average between the hours and plot at the half hour, I average between the half hours and plot at the hour.
[{"id":"e08d258f206f740d","type":"http request","z":"8dc76c7c28d91dc0","name":"West (75)","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.forecast.solar/estimate/LAT/LONG/TILT/75/2.2","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"x":480,"y":220,"wires":[["4e231fe707a1707e"]]},{"id":"4f23fd40dd993022","type":"debug","z":"8dc76c7c28d91dc0","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","statusVal":"","statusType":"auto","x":790,"y":180,"wires":[]},{"id":"a514e382982b2e2d","type":"http request","z":"8dc76c7c28d91dc0","name":"East (-105)","method":"GET","ret":"obj","paytoqs":"ignore","url":"https://api.forecast.solar/estimate/LAT/LONG/TILT-105/2.2","tls":"","persist":false,"proxy":"","authType":"","senderr":false,"x":470,"y":180,"wires":[["9d4953144b87679b"]]},{"id":"05c6e6c99987756a","type":"function","z":"8dc76c7c28d91dc0","name":"Init/Shift array","func":"var solarFC=flow.get(\"SolarForecast\") || {};\nvar i;\n\n// read date of the first set of records (ie forecast for today)\nvar upDate=Object.keys(msg.payload.result.watt_hours_day)[0];\nvar dateToday=msg.payload.nowTS.substring(0,10);\n\n// initialise object & table if it does not exist\nif (Object.keys(solarFC).length==0) {\n solarFC.solarTable=[];\n setTable(0);\n solarFC.energyToday=0;\n solarFC.energyTomorrow=0;\n solarFC.start=null;\n solarFC.stop=null;\n solarFC.dates={yesterday: msg.payload.dateYesterday, today: dateToday, tomorrow: msg.payload.dateTomorrow};\n solarFC.lastRead={P1: {time: null, watts: null, todayTotal: null, tomorrowTotal: null}, P2: {time: null, watts: null, todayTotal: null, tomorrowTotal: null}};\n flow.set(\"SolarForecast\", solarFC);\n}\n\n// if update is beyond 'tomorrow', clear array and start again\nif (upDate>solarFC.dates.tomorrow) {\n solarFC.solarTable=[];\n setTable(0);\n solarFC.dates={yesterday: msg.payload.dateYesterday, today: dateToday, tomorrow: msg.payload.dateTomorrow};\n flow.set(\"SolarForecast\", solarFC);\n}\n\n// if update = 'tomorrow' we need to move array left & add new 'tomorrow'\nif (upDate==solarFC.dates.tomorrow) {\n for (i=1; i<=24; i++) {solarFC.solarTable.shift()}\n setTable(48);\n solarFC.dates.yesterday=solarFC.dates.today;\n solarFC.dates.today=solarFC.dates.tomorrow;\n solarFC.dates.tomorrow=msg.payload.dateTomorrow;\n flow.set(\"SolarForecast\", solarFC);\n}\n\n// keep 'today' for use later\nmsg.payload.dateToday=dateToday;\nreturn msg;\n\nfunction setTable(from) {\n var j, hr, ts;\n for (j=from; j<72; j++) {\n ts=msg.payload.dateTomorrow;\n if (j<48) {ts=dateToday}\n if (j<24) {ts=msg.payload.dateYesterday}\n hr=j%24;\n ts=ts+\" \"+(100+hr).toString().substring(1,3)+\":00:00\";\n solarFC.solarTable[j] = {hour: hr, timestamp: ts, actualWh: null, efcWh: null, fcW: 0, oldfcW: null };\n }\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":860,"y":280,"wires":[["51d4b5d7426bcdeb"]]},{"id":"75789820f57a598f","type":"moment","z":"8dc76c7c28d91dc0","name":"NowTS","topic":"","input":"","inputType":"date","inTz":"ETC/UTC","adjAmount":0,"adjType":"days","adjDir":"add","format":"YYYY-MM-DD HH:mm","locale":"en-GB","output":"payload.nowTS","outputType":"msg","outTz":"Europe/London","x":920,"y":220,"wires":[["5e3849fade603d4c"]]},{"id":"5e3849fade603d4c","type":"moment","z":"8dc76c7c28d91dc0","name":"Y'day","topic":"","input":"","inputType":"date","inTz":"ETC/UTC","adjAmount":"1","adjType":"days","adjDir":"subtract","format":"YYYY-MM-DD","locale":"en-GB","output":"payload.dateYesterday","outputType":"msg","outTz":"Europe/London","x":1050,"y":220,"wires":[["aae43beb5a958f00"]]},{"id":"9d4953144b87679b","type":"change","z":"8dc76c7c28d91dc0","name":"P1","rules":[{"t":"set","p":"payload.plane","pt":"msg","to":"P1","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":610,"y":180,"wires":[["4f23fd40dd993022","95407afda7f38cef"]]},{"id":"4e231fe707a1707e","type":"change","z":"8dc76c7c28d91dc0","name":"P2","rules":[{"t":"set","p":"payload.plane","pt":"msg","to":"P2","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":610,"y":220,"wires":[["4f23fd40dd993022","95407afda7f38cef"]]},{"id":"aae43beb5a958f00","type":"moment","z":"8dc76c7c28d91dc0","name":"T'row","topic":"","input":"","inputType":"date","inTz":"ETC/UTC","adjAmount":"1","adjType":"days","adjDir":"add","format":"YYYY-MM-DD","locale":"en-GB","output":"payload.dateTomorrow","outputType":"msg","outTz":"Europe/London","x":1170,"y":220,"wires":[["05c6e6c99987756a"]]},{"id":"51d4b5d7426bcdeb","type":"function","z":"8dc76c7c28d91dc0","name":"Save record","func":"var solarFC=flow.get(\"SolarForecast\");\nvar plane=msg.payload.plane;\n\n// save part of API record, according to plane\nsolarFC.lastRead[plane].time=msg.payload.nowTS;\nsolarFC.lastRead[plane].watts=msg.payload.result.watts;\nsolarFC.lastRead[plane].todayTotal=msg.payload.result.watt_hours_day[msg.payload.dateToday];\nsolarFC.lastRead[plane].tomorrowTotal=msg.payload.result.watt_hours_day[msg.payload.dateTomorrow];\nflow.set(\"SolarForecast\", solarFC);\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1030,"y":280,"wires":[[]]},{"id":"95407afda7f38cef","type":"switch","z":"8dc76c7c28d91dc0","name":"Success?","property":"payload.message.code","propertyType":"msg","rules":[{"t":"eq","v":"0","vt":"num"},{"t":"else"}],"checkall":"true","repair":false,"outputs":2,"x":780,"y":220,"wires":[["75789820f57a598f"],[]]},{"id":"edc105c15dfe0888","type":"api-current-state","z":"8dc76c7c28d91dc0","name":"Solar Actual Wh hr30","server":"","version":3,"outputs":1,"halt_if":"","halt_if_type":"str","halt_if_compare":"is","entity_id":"sensor.solar_energy_hr30","state_type":"num","blockInputOverrides":false,"outputProperties":[{"property":"payload","propertyType":"msg","value":"","valueType":"entity"}],"for":"0","forType":"num","forUnits":"minutes","override_topic":false,"state_location":"payload","override_payload":"msg","entity_location":"data","override_data":"msg","x":460,"y":340,"wires":[["240261a5dcdb5d83"]]},{"id":"240261a5dcdb5d83","type":"change","z":"8dc76c7c28d91dc0","name":"values","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.attributes","tot":"msg"},{"t":"set","p":"payload.value","pt":"msg","to":"$number(msg.payload.last_period)*1000","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":630,"y":340,"wires":[["38d3d2589928f9df"]]},{"id":"3a49dbac5ac2a5ef","type":"delay","z":"8dc76c7c28d91dc0","name":"10s","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":310,"y":220,"wires":[["e08d258f206f740d","b9e003618f7c07f6"]]},{"id":"b9e003618f7c07f6","type":"delay","z":"8dc76c7c28d91dc0","name":"10s","pauseType":"delay","timeout":"10","timeoutUnits":"seconds","rate":"1","nbRateUnits":"1","rateUnits":"second","randomFirst":"1","randomLast":"5","randomUnits":"seconds","drop":false,"allowrate":false,"outputs":1,"x":290,"y":340,"wires":[["edc105c15dfe0888"]]},{"id":"7820caeea2c216d4","type":"function","z":"8dc76c7c28d91dc0","name":"Chart Array","func":"var graph = [\n {\n \"series\": [\"Forecast\", \"History\", \"Energy\", \"Actual\"],\n \"data\": [ [], [], [], []],\n \"labels\": []\n }\n];\n\nvar i;\nvar dp;\n\nfor (i = 0; i < msg.payload.length; i++) {\n dp=msg.payload[i];\n graph[0].data[0].push(dp.fcW);\n graph[0].data[1].push(dp.oldfcW);\n graph[0].data[2].push(dp.efcWh);\n graph[0].data[3].push(dp.actualWh);\n \n graph[0].labels.push(dp.hour.toString().concat(\":00\"));\n}\n\nreturn {payload: new Array(graph[0])};\n","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":890,"y":400,"wires":[["6f563040164af065"]]},{"id":"b4bc85f8910e51d7","type":"change","z":"8dc76c7c28d91dc0","name":"Read FC","rules":[{"t":"set","p":"payload","pt":"msg","to":"SolarForecast","tot":"flow"}],"action":"","property":"","from":"","to":"","reg":false,"x":240,"y":400,"wires":[["13ca57d7603b44e2","61a51f448dea4164","53154cd628b93a06","6ad0127144bb6b65","770e87fd5dc16fe4","ce8d74722e05f5c8","8e411ef252b4a030"]]},{"id":"6f563040164af065","type":"ui_chart","z":"8dc76c7c28d91dc0","name":"Forecast","group":"76e876790e22c53e","order":2,"width":0,"height":0,"label":"SolarForecast (Yesterday-Today-Tomorrow)","chartType":"line","legend":"true","xformat":"HH:mm","interpolate":"cubic","nodata":"","dot":true,"ymin":"","ymax":"","removeOlder":1,"removeOlderPoints":"","removeOlderUnit":"3600","cutout":0,"useOneColor":false,"useUTC":false,"colors":["#1f77b4","#d62728","#ff7f0e","#2ca02c","#98df8a","#d62728","#ff9896","#9467bd","#c5b0d5"],"outputs":1,"useDifferentColor":false,"className":"","x":1040,"y":400,"wires":[[]]},{"id":"6f365ba0a9744877","type":"inject","z":"8dc76c7c28d91dc0","d":true,"name":"Startup - refresh displays","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"","crontab":"","once":true,"onceDelay":0.1,"topic":"","payloadType":"date","x":230,"y":440,"wires":[["b4bc85f8910e51d7"]]},{"id":"38d3d2589928f9df","type":"moment","z":"8dc76c7c28d91dc0","name":"TheHour","topic":"","input":"payload.last_reset","inputType":"msg","inTz":"ETC/UTC","adjAmount":0,"adjType":"days","adjDir":"add","format":"HH","locale":"en-GB","output":"payload.hour","outputType":"msg","outTz":"Europe/London","x":760,"y":340,"wires":[["03a5848d489c44e0"]]},{"id":"03a5848d489c44e0","type":"change","z":"8dc76c7c28d91dc0","name":"Index","rules":[{"t":"set","p":"payload.index","pt":"msg","to":"$parseInteger(msg.payload.hour, \"99\")+24\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":890,"y":340,"wires":[["236a759e877f8fa7"]]},{"id":"730cf7a0b350b753","type":"change","z":"8dc76c7c28d91dc0","name":"Clear","rules":[{"t":"set","p":"payload","pt":"msg","to":"\"\"","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":170,"y":180,"wires":[["3a49dbac5ac2a5ef","a514e382982b2e2d"]]},{"id":"6cb12d8d34719e70","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":3,"width":5,"height":1,"name":"","label":"Energy Today","format":"{{msg.payload}} kWh","layout":"row-spread","className":"","x":500,"y":560,"wires":[]},{"id":"53154cd628b93a06","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":5,"width":4,"height":1,"name":"","label":"Date","format":"{{msg.payload.dates.today}}","layout":"row-spread","className":"","x":470,"y":520,"wires":[]},{"id":"ac6df3972954c2f1","type":"ha-entity","z":"8dc76c7c28d91dc0","d":true,"name":"FC Energy Estimate","server":"","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"FC energy estimate"},{"property":"device_class","value":"measurement"},{"property":"icon","value":"mdi:counter"},{"property":"unit_of_measurement","value":"kWh"},{"property":"state_class","value":"energy"},{"property":"last_reset","value":""}],"state":"payload.energy","stateType":"msg","attributes":[{"property":"power","value":"payload.power","valueType":"msg"},{"property":"start","value":"payload.start","valueType":"msg"},{"property":"stop","value":"payload.stop","valueType":"msg"},{"property":"maximum","value":"payload.max","valueType":"msg"},{"property":"minimum","value":"payload.min","valueType":"msg"},{"property":"maxPower","value":"payload.maxPower","valueType":"msg"},{"property":"date","value":"payload.today","valueType":"msg"}],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"block","outputOnStateChange":false,"outputPayload":"","outputPayloadType":"str","x":1080,"y":500,"wires":[[]]},{"id":"1f15dba3805db244","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":7,"width":5,"height":1,"name":"","label":"Energy Tomorrow","format":"{{msg.payload}} kWh","layout":"row-spread","className":"","x":510,"y":600,"wires":[]},{"id":"c4a8903f4a0a4a42","type":"template","z":"8dc76c7c28d91dc0","name":"Period","field":"payload","fieldType":"msg","format":"handlebars","syntax":"mustache","template":"{{payload.start}} to {{payload.stop}}","output":"str","x":430,"y":720,"wires":[["354a2540446d3ced"]]},{"id":"354a2540446d3ced","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":9,"width":4,"height":1,"name":"","label":"Period","format":"{{msg.payload}}","layout":"row-spread","className":"","x":550,"y":720,"wires":[]},{"id":"13ca57d7603b44e2","type":"change","z":"8dc76c7c28d91dc0","name":"Table","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.solarTable","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":710,"y":400,"wires":[["7820caeea2c216d4","e204e3f2caada2a3"]]},{"id":"61a51f448dea4164","type":"function","z":"8dc76c7c28d91dc0","name":"Analyse Forecast","func":"var trend=\"night\";\nvar i, j, hour, FCnow, FCstart, FCstop, maxi=0, mini=0;\nvar FCmax=[], FCmin=[];\nvar FCpower=[{power: 0, period: [{start: 0, stop: 0, minutes: 0}]}];\nvar FCmaxPower=0, FCenergyToday=0;\nvar FCold, FClast=0, FCthisOne, FClastOne=0;\nvar startMins, stopMins;\nvar today=msg.payload.dates.today;\n\n// set up power array: up to 4000 watts in 250 watt increments\nfor (i=0; i<17; i++) {\n FCpower[i]={power: i*250, period: []};\n}\n// write true start/stop to [0] record (ie zero power)\nFCpower[0].period[0]={start: toMinutes(msg.payload.start.substring(11,16)), stop: toMinutes(msg.payload.stop.substring(11,16)), minutes: 0};\nstartMins=parseInt(msg.payload.start.substring(14,16));\nstopMins=parseInt(msg.payload.stop.substring(14,16));\n\n// go through forecast array - for today only\n// use 'history' record of forecast in preference to current forecast where it exists\n// so as to get a more accurate figure as the day progresses\n// *now is forecast, *old is history of forecast (current hour)\n// *thisOne is current hour value by preference, *lastOne is last hour value\n\nfor (hour=0; hour<24; hour++){\n FCnow=msg.payload.solarTable[hour+24].fcW;\n FCold=msg.payload.solarTable[hour+24].oldfcW;\n if (FCold>0) {FCthisOne=FCold} else {FCthisOne=FCnow} // use history in preference\n FCenergyToday+=FCthisOne;\n if (trend==\"night\") { // night\n if (FCthisOne>0) { // night->day : rise, start hour\n trend=\"rise\";\n FCstart=hour*60;\n power(startMins);\n }\n } else { // day (rise|fall)\n if (FCthisOne==0) { // day->night : night, stop hour\n power(60-stopMins);\n trend=\"night\";\n FCstop=(hour-1)*60;\n } else { // day=day\n if (FCthisOne>FClastOne) { // is rising\n if (trend==\"fall\") { //fall->rise, minimum point\n trend=\"rise\";\n FCmin[mini]=toHM((hour-1)*60);\n mini++;\n }\n } else { // is falling|same\n if (trend==\"rise\") { //rise->fall, maximum point\n trend=\"fall\";\n FCmax[maxi]=toHM((hour-1)*60);\n maxi++;\n }\n } // end falling|same\n power(0);\n } // end=day\n } // end day (rise|fall)\n FClastOne=FCthisOne;\n} // end for\n\n// calculate event duration and set minutes to string time\nfor (i=0; i<FCpower.length; i++) {\n if (FCpower[i].period.length>0) {\n FCmaxPower=FCpower[i].power;\n }\n for (j=0; j<FCpower[i].period.length; j++) {\n FCpower[i].period[j].minutes=FCpower[i].period[j].stop-FCpower[i].period[j].start;\n FCpower[i].period[j].start=toHM(FCpower[i].period[j].start);\n FCpower[i].period[j].stop=toHM(FCpower[i].period[j].stop);\n }\n}\n\n// output values for Node Red display & push to HA\nmsg.payload={};\nmsg.payload.start=toHM(FCstart); // start hour HH:00\nmsg.payload.stop=toHM(FCstop); // stop hour HH:00\nmsg.payload.maxPower=FCmaxPower; // maximum power level (*250 W) during day\nmsg.payload.max=FCmax; // array of maximum points (hours HH:00)\nmsg.payload.min=FCmin; // array of minimum points (hours HH:00) if found\nmsg.payload.power=FCpower; // array [i*250 W] of objects {power, period [{start, stop, minutes duration}]\nmsg.payload.today=today;\n// estimate of today energy kWh\nmsg.payload.energy=Math.round(FCenergyToday/100)/10;\n\nreturn msg;\n\n\n// extract total minutes (hour*60 + mins) from a string time\nfunction toMinutes(timeS) {\n return parseInt(timeS.substring(0,2))*60+parseInt(timeS.substring(3,5)); \n}\n\n// turn total minutes to a string (with leading zeros)\nfunction toHM(mins) {\n var temp=mins%60;\n return (100+(mins-temp)/60).toString().substring(1,3)+\":\"+(100+temp).toString().substring(1,3);\n}\n\n// for power levels of 250 watt increments, pro-rata the start/end\n// times based on a straight line: work in total minutes\n\n// loop through power levels between last and this power value\n// for each, pro-rata the minutes and add to last hour\n// save start (create new object) end (add to object) as next item in array of events\n// allow for more than one period in the forecast, adjust first/last for start/stop times\n\nfunction power(offset) { // offset = minutes into first hour at start\n var pwrStart, pwrIndex, mins, event; // 60-minutes from end at stop\n \n if (trend==\"rise\") { \n pwrStart=Math.floor((FClastOne*4)/1000)+1;\n for (pwrIndex=pwrStart; pwrIndex<=FCthisOne/250; pwrIndex++) {\n mins=Math.floor(((pwrIndex*250-FClastOne)/(FCthisOne-FClastOne))*(60-offset));\n event=FCpower[pwrIndex].period.length;\n FCpower[pwrIndex].period[event]={start: (hour-1)*60+mins+offset, stop: 0, minutes: 0}\n }\n } else {\n pwrStart=Math.floor((FClastOne*4)/1000);\n for (pwrIndex=pwrStart; pwrIndex>FCthisOne/250; pwrIndex--) {\n mins=Math.floor(((FClastOne-pwrIndex*250)/(FClastOne-FCthisOne))*(60-offset));\n event=FCpower[pwrIndex].period.length;\n FCpower[pwrIndex].period[event-1].stop=(hour-1)*60+mins;\n }\n }\n return;\n}","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":810,"y":500,"wires":[["ac6df3972954c2f1","1819fa8cbaad4353"]]},{"id":"6599aef1b6a6b158","type":"ha-entity","z":"8dc76c7c28d91dc0","d":true,"name":"FC Solar Table","server":"","version":2,"debugenabled":false,"outputs":1,"entityType":"sensor","config":[{"property":"name","value":"FC table"},{"property":"device_class","value":"datetime"},{"property":"icon","value":"mdi:update"},{"property":"unit_of_measurement","value":""},{"property":"state_class","value":"timestamp"},{"property":"last_reset","value":""}],"state":"payload.update","stateType":"msg","attributes":[{"property":"FChours","value":"payload.time","valueType":"msg"},{"property":"FCwatts","value":"payload.forecast","valueType":"msg"},{"property":"FCactual","value":"payload.actual","valueType":"msg"},{"property":"FCold","value":"payload.old","valueType":"msg"},{"property":"FCwh","value":"payload.energyfc","valueType":"msg"}],"resend":true,"outputLocation":"","outputLocationType":"none","inputOverride":"block","outputOnStateChange":false,"outputPayload":"","outputPayloadType":"str","x":1060,"y":440,"wires":[[]]},{"id":"e204e3f2caada2a3","type":"function","z":"8dc76c7c28d91dc0","name":"HA graph array","func":"var i;\nvar ts=[], fc=[], efc=[], act=[], old=[];\nvar temp;\n\nfor (i=0; i<msg.payload.length; i++){\n temp=msg.payload[i];\n ts.push(temp.timestamp);\n fc.push(temp.fcW);\n efc.push(temp.efcWh);\n act.push(temp.actualWh);\n old.push(temp.oldfcW);\n}\n\ntemp=new Date().getHours()+24;\ntemp=msg.payload[temp].timestamp;\nmsg.payload={time: ts, forecast: fc, energyfc: efc, actual: act, old: old, update: temp};\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":880,"y":440,"wires":[["6599aef1b6a6b158"]]},{"id":"8e411ef252b4a030","type":"change","z":"8dc76c7c28d91dc0","name":"Period","rules":[{"t":"set","p":"payload.start","pt":"msg","to":"$substring(msg.payload.start,11,5)\t","tot":"jsonata"},{"t":"set","p":"payload.stop","pt":"msg","to":"$substring(msg.payload.stop, 11,5)\t","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":310,"y":720,"wires":[["c4a8903f4a0a4a42"]]},{"id":"c12e8f53b88a1fca","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":6,"width":5,"height":1,"name":"","label":"P1 update","format":"{{msg.payload.P1.time}}","layout":"row-spread","className":"","x":490,"y":640,"wires":[]},{"id":"31ba7f6e5d2590fa","type":"ui_text","z":"8dc76c7c28d91dc0","group":"76e876790e22c53e","order":10,"width":5,"height":1,"name":"","label":"P2 update","format":"{{msg.payload.P2.time}}","layout":"row-spread","className":"","x":490,"y":680,"wires":[]},{"id":"ce8d74722e05f5c8","type":"change","z":"8dc76c7c28d91dc0","name":"lastRead","rules":[{"t":"set","p":"payload","pt":"msg","to":"payload.lastRead","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":320,"y":640,"wires":[["31ba7f6e5d2590fa","c12e8f53b88a1fca"]]},{"id":"6ad0127144bb6b65","type":"function","z":"8dc76c7c28d91dc0","name":"to kWh","func":"msg.payload=(msg.payload.energyToday/1000).toFixed(1);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":340,"y":560,"wires":[["6cb12d8d34719e70"]]},{"id":"770e87fd5dc16fe4","type":"function","z":"8dc76c7c28d91dc0","name":"to kWh","func":"msg.payload=(msg.payload.energyTomorrow/1000).toFixed(1);\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":340,"y":600,"wires":[["1f15dba3805db244"]]},{"id":"1819fa8cbaad4353","type":"change","z":"8dc76c7c28d91dc0","name":"Save Power Table","rules":[{"t":"set","p":"SolarPowerTable","pt":"flow","to":"payload.power","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":990,"y":560,"wires":[[]]},{"id":"236a759e877f8fa7","type":"function","z":"8dc76c7c28d91dc0","name":"Update Table","func":"var FC = flow.get(\"SolarForecast\")\n\n// save actual from msg.payload index/value. index = last hour\n// this sould be energy Wh between hh-:30 and hh+:30\nvar index=msg.payload.index;\nFC.solarTable[index].actualWh=msg.payload.value;\n\n// save pre-update forecast for last hour to forecast history\nFC.solarTable[index].oldfcW=FC.solarTable[index].fcW;\n\nvar today=FC.dates.today;\nvar plane2=(FC.lastRead.P2.time.substring(0,10)==today); // true if P2 last update date is today\n\nFC.energyToday=FC.lastRead.P1.todayTotal;\nFC.energyTomorrow=FC.lastRead.P1.tomorrowTotal;\nif (plane2) {\n FC.energyToday+=FC.lastRead.P2.todayTotal;\n FC.energyTomorrow+=FC.lastRead.P2.tomorrowTotal;\n}\n\nvar i, j;\nvar key, value, date, hour;\nvar wattArray=FC.lastRead.P1.watts;\nvar keyArray=Object.keys(wattArray);\n\n// clear all hour forecasts to zero\nfor (i=24; i<72; i++) {FC.solarTable[i].fcW=0}\n\n// save P1 last-read values to table\nfor (i=0; i<keyArray.length; i++) {\n key=keyArray[i];\n value=wattArray[key];\n date=key.substring(0,10);\n hour=parseInt(key.substring(11,13),10);\n j=hour+24;\n if (date==today){\n if (value==0){ // capture start/end hour\n if (hour<12) {FC.start=key}\n else {FC.stop=key}\n }\n } else {j+=24}\n if(value>0) {FC.solarTable[j].fcW=value}\n}\n\n// add in P2 values if required\nif (plane2) {\n wattArray=FC.lastRead.P2.watts;\n keyArray=Object.keys(wattArray);\n for (i=0; i<keyArray.length; i++) {\n key=keyArray[i];\n value=wattArray[key];\n date=key.substring(0,10);\n hour=parseInt(key.substring(11,13),10);\n j=hour+24;\n if (date>today) {j+=24}\n if (value>0) {FC.solarTable[j].fcW+=value}\n }\n}\n\n// recalculate watt-hours for each hr:-30 to hr:+30 and\n// post to hh:00 using a+b+b+c/4\n// run from index (last hour) to end of today\nfor (i=index; i<48; i++) {\n value=FC.solarTable[i-1].fcW;\n if (i==index) {value=FC.solarTable[i-1].oldfcW}\n value+=2*FC.solarTable[i].fcW;\n value+=FC.solarTable[i+1].fcW;\n FC.solarTable[i].efcWh=Math.floor(value/4);\n}\n\nflow.set(\"SolarForecast\", FC);\n\nreturn msg;","outputs":1,"noerr":0,"initialize":"","finalize":"","libs":[],"x":1030,"y":340,"wires":[["b4bc85f8910e51d7"]]},{"id":"3992708ff30a51e8","type":"inject","z":"8dc76c7c28d91dc0","d":true,"name":"","props":[{"p":"payload"},{"p":"topic","vt":"str"}],"repeat":"3600","crontab":"","once":false,"onceDelay":0.1,"topic":"","payloadType":"date","x":150,"y":100,"wires":[["730cf7a0b350b753"]]},{"id":"76e876790e22c53e","type":"ui_group","name":"Solar Forecast","tab":"a7de90402f2427f9","order":7,"disp":true,"width":15,"collapse":false,"className":""},{"id":"a7de90402f2427f9","type":"ui_tab","name":"Solar Forecast","icon":"dashboard","order":1,"disabled":false,"hidden":false}]
You will have to decide how much of the Node-RED dashboard you want, and how much of the uplift to HA you want.
The forecast analysis builds a table of power periods based on 250 watts, and for each provides an array of possible events, each event being start, stop, and duration minutes. There being more than one period at a given power when the forecast goes up and down, I am still working on how best to use this information in HA. So far I have managed a template sensor that, for a given power, identifies the longest event out of a list of events at that power level.
I should add that I have two planes, so this is set up for two API calls and combination. You will have to remove the second call if not required - I did try to write the code so that it will sort out if only one plane, but naturally this is presented âas isâ and some work may be required.
EDIT April 2024 - Code has been updated and is now on Github for anyone interested
Thanks a lot! This is very helpful. I only have one remaining question. I didnât quite get how you setup âsensor.solar_energy_hr30â. I have also setup a sensor which is a Riemann Sum on my inverter power and I could setup a utility sensor with hourly reset. But how can I make sure that it actually shows the energy over the last hour? If it resets mid-hour than depending on when the flow actually runs, the utility sensor only displays the energy generated since the last reset, doesnât it? It would be great if you could share some more insights on how you calculate the sensor. Thank you!
Yup I just knew you were going to ask that one sooner or later.
So, the solar forecast is power, measured at the hour, and I plot this on my graph - power level, each hour.
To get energy from the forecast power, I can get the average power, measured across one hour, which then becomes watt-hours. Eg 9:00 power 800 W, 10:00 power 1200 W, simple average is 1000 W, so average 1000 Watts for one hour = 1000Wh. Now, this value should be plotted at 9:30 as it represents the period 9-10, not plotted at the hour 9:00 or hour 10:00. But I am plotting (power) at the hour and I donât want to plot half way between. So, I decided to take the power at 9:30, the power at 10:30, and average them to get the energy between 9:30 -10:30, and plot this energy value at 10:00. The power at 9:30 is the average power between 9 and 10, and for 10:30 between 10 and 11. Thus the power 9:30 to 10:30 is ( power(9:00) + 2*power(10:00) + power(11:00) ) / 4
This gives the forecast average power, for the half hour to half hour period, which can be plotted at the hour.
Now I also want the âactualâ as power in watt-hours, for the period 9:30 to 10:30, to plot at 10:00 in the same way.
I have:
Inverter power sensor â inverter energy sensor using Riemann Sum.
I added:
Utility meter, based on the inverter energy, which has to be set up in the configuration file, as hourly with an offset of 30 minutes. The helper Utility Meter cannot cope with offsets.
This utility meter now records the energy between, for example, 9:30 and 10:30, and resets to zero at 10:30.
If, at around 11:10 I read this meter, it returns in the entity state the current energy between 10:30 and 11:10, but in the payload.attributes it has the value âlast_periodâ which happens to be the total energy for the period ending at the last reset, or for 9:30 to 10:30. I grab this, and push it to the array for the last_reset hour, being 10:00 (from 10:30 reset).
Yes, it only works correctly for an hourly update between hour:01 and hour:29.
Correction: works always. If, at around 11:50 I read this meter, it returns energy 10:30 to 11:30 in the last_period, and last_reset is 11:30 so hour is 11:00. Posts 10:30 to 11:30 energy at 11:00.
I use a different type of node to trigger, and set this to run at hour:04. However, being community minded I am aware that if everyone wanted their flow to update at hour:04 the âfreeâ API call would not cope. In the HA ForecastSolar integration, the update is ârandomâ, thus preserving an even spread for demand on the API. I was just mindful when publishing this not to encourage bunching, but clearly that is up to the end users.
This all seems to work - I get the power forecast, which changes during the day, a history record of the power forecast now in the past, an energy forecast, and the actual energy. I have a really good match (on full sun days) with all four plots for the afternoon, but a difference in the morning, from my East facing panels. I am still working on why - some of this is my panel orientation and placement (being North of East) but I still think the forecast is out.
And yes, I know the colours on my Node-RED dashboard graph donât match those on my HA Apex Charts graph. I have been working on this stuff for months!
Thanks again! I did get it to work. I now just have to extend it to 4 planes. 
Hopefully, 4 additional API calls per hour together with the forecast.solar integration will not exceed the limit of 12 calls per hour. Do you know by any chance what the number of API calls is that the integrated forecast.solar integration triggers per hour?
The integration does one call per integration per hour. This is at the ârandomâ time of the update. Current allowance is 12 per hour per IP address. I donât know if it is a rolling hour or fixed hour. With two planes, two integrations, and two of my own API calls, I consume 4/12 per hour so room for the odd testing call as well.
The full return from the API call sends back location as well as how many API calls you have left. During testing I have checked this number and it seems reasonably accurate, so it could be used to manage or track useage.
Everything is working perfectly right now. Incl. persistant storage in HA and the 4 planes. I only have one small question left: How can I access the values stored in sensor.fc_table in HA? I see that the values are displayed in the attributes of the sensor and are separated by commas, but is there any way to access them directly, letâs say to use the values in an apexcharts card?
Edit: Answered this myself. Itâs an array so that I can easily access the values in the apexcharts data generator. Perfect!
I bought HA to use it to monitor and control my Solar PV system. The feature that âsoldâ it to me was the integrated Solar Forecast - however I have found it to be not entirely accurate and of little practical use as we canât get at the data.
The big question for Solar Forecasting is not âhow to get itâ but âwhat to do with it once you have itâ. I was inspired by
https://community.home-assistant.io/t/forecast-solar-predict-when-output-power-will-be-above-predefined-level/424348
to start working on Solar Forecast, with a view to answering this very question - to decide when output power will be above a predefined level.
Getting the API calls to work was the easy bit. Next was a graph, and that is what sensor.fc_table provides. As you have answered - this holds all the values in arrays in the sensor attributes and therefore was designed to run almost directly into apex charts. I had not used apex charts before, but it was not that difficult once I got my head around the data_generator bit. Still donât know quite how it works, but it does.
type: custom:apexcharts-card
graph_span: 3d
span:
end: day
offset: +1d
header:
show: true
title: 'Solar Forecast: yesterday - today - tomorrow'
now:
show: true
label: now
show:
last_updated: true
series:
- entity: sensor.fc_table
data_generator: |
return entity.attributes.fchours.map((fchr, index) => {
return [new Date(fchr).getTime(), entity.attributes.fcwatts[index]];
});
curve: smooth
name: Forecast
show:
in_header: false
legend_value: false
stroke_width: 2
color: orange
- entity: sensor.fc_table
data_generator: |
return entity.attributes.fchours.map((fchr, index) => {
return [new Date(fchr).getTime(), entity.attributes.fcold[index]];
});
curve: smooth
name: History
show:
in_header: false
legend_value: false
stroke_width: 2
color: magenta
- entity: sensor.fc_table
data_generator: |
return entity.attributes.fchours.map((fchr, index) => {
return [new Date(fchr).getTime(), entity.attributes.fcactual[index]];
});
curve: smooth
name: Actual
show:
in_header: false
legend_value: false
stroke_width: 2
color: blue
- entity: sensor.fc_table
data_generator: |
return entity.attributes.fchours.map((fchr, index) => {
return [new Date(fchr).getTime(), entity.attributes.fcwh[index]];
});
curve: smooth
name: Energy
show:
in_header: false
legend_value: false
stroke_width: 2
color: green
My next objective is to get the time to start an appliance given a required power level. This is where FC energy estimate sensor comes in. The state value is my estimate of the total daily energy, and the attributes hold the power table as well as the hour of start, stop, and maximum. If there are humps in the day there will also be one or more minimums.
I have got as far as being able to provide in HA an array of events for a given power level, and even sorting this for the largest duration if there are more than one - this returns âstartâ and âstopâ as hh:mm, and duration in minutes. I am trying to extend this to working out, for a given power, earliest start time, and minimum duration, a return that says when the period starts, when to switch on the device, and how long you have left. Doing the unpicking of the power array in HA has defeated me, so I am now looking at using Node-RED to do the work, based on an API call from HA back to Node-RED with the query parameters - power, start, duration, with a return object containing switching states and times. Still work in progress, and I am looking to first fine-tune my forecast to better match my actual.
Hope you find the code of use!
All your input is very helpful! Many thanks for this!
Hi, sorry to drop in on a seemingly solved matter.
But canât we grab the already existing data from somewhere in the database?
Anyone know?
I donât know the database structure that well. But this dashed curve is âplottedâ from info that must be stored locally somewhere.
Just a thought.
My goal is to simply get the forecast energy (kWh) for the remainder of the day. (from now until sunset)
The marked solution is an alternate way to solve this by making additionnal calls to the Forecast.Solar API, so it may not be optimal.
What you are saying is exactly the original issue on this thread.
There is no known solution to directly âgrabâ that dashed ploted data.
If you find it please share for othersâŚ
Iâll grab the database from a backup file, and dissect it to try to find it there.
âŚ
[EDIT] Ok, after some digging I donât think this is stored in the HA database.
Maybe this is stored in RAM afterall.
The Energy settings file also doesnât help (.storage/energy)
"data": {
"energy_sources": [
(...)
{
"type": "solar",
"stat_energy_from": "sensor.[YOUR_ENERGY_SENSOR]",
"config_entry_solar_forecast": [
"[long identifier found in the core.config_entry file]"
]
},
(...)