Skimming through the thread, this could also be related to my issue #72181: forecast solar updates the reported value only every hour (or 30 mins with an API key). However, if you would load the integration at, say, 14:22, then the “now” for 14:00-15:00 is reported until 15:22. This could also lead to shifts (at least this is what I am expecting with influxdb)
1 Like
Not related at all.
The forecast.solar integration calls the forecast API once (per integration) per hour, every hour, based on a ‘random’ (but predictable) time during each hour. The call is then processed by code and re-plotted as a line for the entire day, not just the past hour. (The solar generation bar chart is updated and plotted for the past hour each hour after the end of the hour - based on an internal utility meter that resets each hour - quite different).
Each forecast hourly API call re-processes the entire day forecast, using the returned cumulative watt-hour figures, by subtracting each hour-figure from the next hour-figure to get an absolute forecast watt-hour value for the hour period between the two figures. At this point the entire forecast line is re-plotted.
As well as updating the forecast line, the integration picks out a few watt power and watt-hour energy values from the API return, presenting a selection including ‘this hour’ and ‘next hour’ forecast.
Your concern is that the update is called once per hour but this can be at any point up to the 59th minute, although that has little immediate impact on the forecast line. The forecast does indeed change during the day, hence the line will move, even the ‘old’ part of the line. If this moves at 14:01 or 14:59, it is still an update every hour.
My concern is that the figure for the period 14:00 to 15:00 is plotted at 15:00 (which is 15:00 to 16:00 on the bar chart) and I contest that it should be plotted at 14:00. I have been through the code -
and I believe that the iteration loop should be something along the lines of
previous_value = 0
previous_timestamp = 0
wh_hours = {}
for timestamp, energy in data["result"]["watt_hours"].items():
timestamp = datetime.fromisoformat(timestamp)
# If we get a reset and when moving to the next day
if energy < previous_value:
previous_value = 0
# At start of day (sunrise) energy is zero, minutes are not, period has only started
if energy = 0:
# previous_value = 0
timestamp = timestamp.replace(minute=0)
# otherwise period has ended so capture energy difference and post to start of period
else:
wh_hours[previous_timestamp] = energy - previous_value
previous_value = energy
previous_timestamp = timestamp
I am asking for the iteration code that calculates the array of the hourly watt-hour energy forecasts for each hour period to assign each value to the correct hour (ie at the start of the hour period not the end of the hour period) so that the array is correctly plotted as a line on the energy graph in agreement with the convention of the consumption bar chart plot.
What are you really asking for?
- For an update more than once per hour?
- For each hourly update to take place at the start of each hour?
What I hope that you will see is that this is a free service (for most of us) and therefore we have to take it as it comes. If everyone decided that they wanted their forecast update at 14:01 on the dot, or several times during the hour (to no effective additional value) the solar forecast server would probably not cope. By making the integration use the ‘random’ trigger time that is randomly distributed across the hour, the demand from HA users (for the free integration API calls) is evenly spread, and limited to less than the 12 free calls per hour so generously provided.
You have the solution to your issue in your own hands - you already know how to modify the trigger point for the update call yourself.
Alternatively, assuming that you are lifting the provided this hour forecast data, then I assume that your issues is really that, when you go for this data, if the update has not taken place yet then this hour is actually the last hour and not this hour. However, by checking the time of the update, if the update hour is the same as this hour, then the this hour value is the value for this hour and not the last hour. If the update hour is not the same as this hour, then this hour is the value for the last hour, but the next hour holds the value for the next hour from the last hour which is this hour, so you could use the next hour value for this hour value up to the point where the update takes place. If you also want the next hour during the time when the update hour for this hour is not this hour but the last hour, then the value in the next hour is this hour, so you will have to read the history for the six hour forecast from four hours ago which will give you the value of the hour ahead of the next hour but before it was updated so it will be the value for the next hour. All really very simple.
I have spent many hours on this in great depth, now probably into three figures, and my final solution - I have created my own Node-RED flow to pull the data and process it myself, so I can control the time at which I make the API calls and manage the data as I wish. However I am very mindful that I should treat this free service with some respect and not overload it. I hope you agree.
1 Like
As you raised this issue in the first place, I thought you might appreciate an update on my attempts to get the solar forecast to be realistic against my production.
The good news is that I now have a good match. This is achieved using my own code, calling and processing the forecast.solar API myself.
I have a nice graph showing the upcoming forecast for today and tomorrow, as well as a history record for yesterday, and a record for actual production (all carefully worked out to plot at the correct hour for the correct period).
Here is this morning - you will notice that the forecast (orange line) is different to the history (magenta line). As I capture the forecast for the last hour into a history line, I can see where the forecast moves after the event. Today, the actual matches history at the start, but the forecast has moved and now does not - hence we cannot rely on looking at the forecast line at the end of the day as it will not be the forecast as it was!
Getting the match was a question of shuffling the damping factor around. I have power, slope, azimuth all correct, but my situation is complicated by having two planes - east and west. Damping has a symmetrical effect for a south facing production curve, equally both morning and afternoon. My planes produce a shifted generation curve, and damping over-corrects and under-corrects at opposite ends of the day, so I had to dive into the individual planes rather than looking at the summation.
Here is a plot for 14th August (from HA history) which was a nice sunny day. I capture the power values for each plane every 20 seconds from my inverter, and against this I have plotted the forecast values for each plane for the hour, updated at the start of the hour (read the leading edge of the steps).
With a lot of trial and error I have managed to get the over/under compensation to balance out and the summation curve is now a good fit for a sunny day. I continue to monitor other days, but I believe I have, at last, found a reliable and working solution.
I’m not asking to query the forecast.solar API more than once an hour. The problem is that forecast solar gives power estimates for a precise time (this issue is less relevant for energy estimates, which, as an integral, are more continuous). My point is that reporting these values is shifted by (up to) one hour since the loop you mentioned above is also only executed once an hour.
In particular, the resulting graphs are shifted as well by (up to) one hour, since the graphing (at least when using grafana + influxdb, and I am rather certain for HA graphs as well) has a sub-hour granularity. Suppose that forecast solar reports 14:00 1000W and 15:00 2000W. Using the integration, you might see (15:59, 1000W) or (15:00, 2000W) in your graph, since the integration only “syncs” its sensor value with the data it gets from the forecast solar server every hour. There is no need to fetch data more than once an hour to fix the issue. If the sensor value is updated, say, every minute, and FC only queried once an hour, this misalignment vanishes.
I am offering to fix the issue through a PR if acknowledged / discussed with the code owners (its a bit disrespectful to just do that without communication IMO) and was merely mentioning this since it might be related. Pretty sure it indeed is, since “rounding down” the timestamp is more or less what I am also suggesting (only instead of changing the timestamp retroactively, I suggest to update at the hour). However, I think that the proposed fix is not necessarily correct, since FC sometimes reports with sub-hour granularity even on the free API (I think it reports “0” at the exact times of sunrise and sunset)
If you know how to solve this in the code, then I look forward to receiving that pull request 
1 Like
I am sorry but I have no idea about pull requests, and I have had to struggle with the Python code. However I have looked at your code reference, and after trying this out in JS (which I understand better) I came up with the following…
previous_value = 0
previous_timestamp = 0
wh_hours = {}
for timestamp, energy in data["result"]["watt_hours"].items():
timestamp = datetime.fromisoformat(timestamp)
# If we get a reset and when moving to the next day
if energy < previous_value:
previous_value = 0
# At start of day (sunrise) energy is zero, minutes are not, period has only started
if energy = 0:
# previous_value = 0
timestamp = timestamp.replace(minute=0)
# otherwise period has ended so capture energy difference and post to start of period
else:
wh_hours[previous_timestamp] = energy - previous_value
previous_value = energy
previous_timestamp = timestamp
How you manage to write this integration at all is beyond me, but hopefully this is not complete nonsense.
Testing it (at least the near equivalent in JS) produces an array for the watt-hours from the sunrise hour (at the 00 minute) and thereafter for the following hour with each hour for the following period. Clearly I have no idea how this gets into the energy dashboard solar graph, but my assumption is that it would shift everything to one hour earlier.
Best wishes
Hello,
The data from the forecast solar integration appears to be correct (very close forecast to solar array output as viewed in history graphs), it’s just being plotted with a 1h offset on the frontend (as mentioned by others in the thread). Applying a 1h correction provided great match on the displayed data.
Adjust forecast solar data time offset to match by arthurbenemann · Pull Request #13855 · home-assistant/frontend · GitHub contains the quick-fix on the frontend. But it may just be uncovering an underlying issue. Once I get another set of data (clock turned over, and forecast is not available anymore) I’ll post it here.
One extra note for a different issue:
The API being used doesn’t seem to respect the “inverter size” field, at least not for the integrated results of wh_days and wh_hours.
Diagnostics below show the results when the solar array was increased 10x to exaggerate the error/effect (2.1 kW inverter can only generate 50400 kWh per day even if it runs 24/7).
{
"home_assistant": {
"installation_type": "Home Assistant OS",
"version": "2022.9.5",
"dev": false,
"hassio": true,
"virtualenv": false,
"python_version": "3.10.5",
"docker": true,
"arch": "x86_64",
"timezone": "America/Los_Angeles",
"os_name": "Linux",
"os_version": "5.15.67",
"supervisor": "2022.09.1",
"host_os": "Home Assistant OS 9.0",
"docker_version": "20.10.17",
"chassis": "vm",
"run_as_root": true
},
"custom_components": {
"hacs": {
"version": "1.27.1",
"requirements": [
"aiogithubapi>=22.2.4"
]
},
"pfsense": {
"version": "0.1.0",
"requirements": [
"mac-vendor-lookup>=0.1.11"
]
},
"truenas": {
"version": "0.0.0",
"requirements": []
},
"frigate": {
"version": "2.3",
"requirements": []
},
"awnet_local": {
"version": "0.2.0",
"requirements": []
},
"enphase_envoy": {
"version": "0.1.2",
"requirements": [
"envoy-utils"
]
},
"powercalc": {
"version": "v0.25.4",
"requirements": [
"numpy>=1.21.1"
]
},
"rainforest_emu_2": {
"version": "1.2.0",
"requirements": [
"pyserial-asyncio==0.6"
]
}
},
"integration_manifest": {
"domain": "forecast_solar",
"name": "Forecast.Solar",
"config_flow": true,
"documentation": "https://www.home-assistant.io/integrations/forecast_solar",
"requirements": [
"forecast_solar==2.2.0"
],
"codeowners": [
"@klaasnicolaas",
"@frenck"
],
"quality_scale": "platinum",
"iot_class": "cloud_polling",
"is_built_in": true
},
"data": {
"entry": {
"title": "Solar Forecast",
"data": {
"latitude": "**REDACTED**",
"longitude": "**REDACTED**"
},
"options": {
"declination": 14,
"azimuth": 177,
"modules power": 24000,
"damping": 0.0,
"inverter_size": 2100
}
},
"data": {
"energy_production_today": 152076,
"energy_production_tomorrow": 147902,
"energy_current_hour": 0,
"power_production_now": 0,
"watts": {
"2022-09-24T06:44:00-07:00": 0,
"2022-09-24T07:00:00-07:00": 1052,
"2022-09-24T08:00:00-07:00": 2100,
"2022-09-24T09:00:00-07:00": 2100,
"2022-09-24T10:00:00-07:00": 2100,
"2022-09-24T11:00:00-07:00": 2100,
"2022-09-24T12:00:00-07:00": 2100,
"2022-09-24T13:00:00-07:00": 2100,
"2022-09-24T14:00:00-07:00": 2100,
"2022-09-24T15:00:00-07:00": 2100,
"2022-09-24T16:00:00-07:00": 2100,
"2022-09-24T17:00:00-07:00": 2100,
"2022-09-24T18:00:00-07:00": 2100,
"2022-09-24T18:51:00-07:00": 0,
"2022-09-25T06:45:00-07:00": 0,
"2022-09-25T07:00:00-07:00": 965,
"2022-09-25T08:00:00-07:00": 2100,
"2022-09-25T09:00:00-07:00": 2100,
"2022-09-25T10:00:00-07:00": 2100,
"2022-09-25T11:00:00-07:00": 2100,
"2022-09-25T12:00:00-07:00": 2100,
"2022-09-25T13:00:00-07:00": 2100,
"2022-09-25T14:00:00-07:00": 2100,
"2022-09-25T15:00:00-07:00": 2100,
"2022-09-25T16:00:00-07:00": 2100,
"2022-09-25T17:00:00-07:00": 2100,
"2022-09-25T18:00:00-07:00": 2100,
"2022-09-25T18:49:00-07:00": 0
},
"wh_days": {
"2022-09-24T00:00:00": 152076,
"2022-09-25T00:00:00": 147902
},
"wh_hours": {
"2022-09-24T06:44:00-07:00": 0,
"2022-09-24T07:00:00-07:00": 140,
"2022-09-24T08:00:00-07:00": 3189,
"2022-09-24T09:00:00-07:00": 7931,
"2022-09-24T10:00:00-07:00": 12774,
"2022-09-24T11:00:00-07:00": 16704,
"2022-09-24T12:00:00-07:00": 19413,
"2022-09-24T13:00:00-07:00": 20668,
"2022-09-24T14:00:00-07:00": 20358,
"2022-09-24T15:00:00-07:00": 18406,
"2022-09-24T16:00:00-07:00": 14961,
"2022-09-24T17:00:00-07:00": 10539,
"2022-09-24T18:00:00-07:00": 5654,
"2022-09-24T18:51:00-07:00": 1339,
"2022-09-25T06:45:00-07:00": 0,
"2022-09-25T07:00:00-07:00": 121,
"2022-09-25T08:00:00-07:00": 2990,
"2022-09-25T09:00:00-07:00": 7480,
"2022-09-25T10:00:00-07:00": 12162,
"2022-09-25T11:00:00-07:00": 16152,
"2022-09-25T12:00:00-07:00": 18947,
"2022-09-25T13:00:00-07:00": 20211,
"2022-09-25T14:00:00-07:00": 19933,
"2022-09-25T15:00:00-07:00": 18057,
"2022-09-25T16:00:00-07:00": 14719,
"2022-09-25T17:00:00-07:00": 10366,
"2022-09-25T18:00:00-07:00": 5522,
"2022-09-25T18:49:00-07:00": 1242
}
},
"account": {
"type": "public",
"rate_limit": 12,
"timezone": "America/Los_Angeles"
}
}
}
There is also another PR open to in the core integration: Update forecast.solar sensor values exactly when reported API values change by incaseoftrouble · Pull Request #77217 · home-assistant/core · GitHub
Hi @klaasnicolaas thanks for pointing out PR #77217. The updates on that are quite useful, but I think we are still going to have the 1h offset on plotted data because of the way the front end is handling it .As current data for exact hour periods appears correct, but still plotted offset in the fronted.
In PR #13855 I updated the comments with screen-capture of the change. re-posting it here for visibility. Current production version:
With PR applied (crude 1h offset in frontend):
Then I hope they review your frontend PR soon and then see to what extent the core PR fine-tunes things even more 
The time shift looks like an additional DST, Forcast.Solar deliveres, if requested local times, with DST inclusive.
Integration calls API asking for ISO timestamp.
API returns list of power values for each hour (local time in ISO format - see integration diagnostic, ie for my UK location 10:00 BST (= GMT +1 hour for DST) is “2022-07-21T10:00:00+01:00”
Integration calculates energy as difference between consecutive power values, eg P(11:00) - P(10:00) = energy for 10:00 to 11:00
Integration plots this on graph at 11:00
Graph treats this as forecast for 11:00 to 12:00
Basic problem is:
Graph is plotting values at the hour, for the hour ahead (10:00 value is production/forecast for 10:00 to 11:00)
Integration is taking value for the hour just gone, at the end of the hour, and plotting at that end hour (10:00 value is forecast for 09:00 to 10:00).
Of course I can wait until the end of next month when the clocks go back (here in the UK), but it really does seem very obvious to me that, if you take the forecast for the hour 10:00 to 11:00 and plot it on the graph at 11:00 which on the graph is for the hour 11:00 to 12:00 it will always be one hour out.
Why calculate the watt hours?
The API also sends the watt hours matching the watts …
First. Thank you for your great solar forecast webservice. As a ‘free’ service I find this a powerful tool to work alongside the reporting I get from my solar inverter. Not always accurate (that is weather for you) but on sunny days, as the ‘clear sky’ default, I now have a good match for forecast to actual production. This means I can rely on and use the solar forecast!
I have fully investigated the API and documentation, and I now use Node-RED to call the API myself, pull all the values provided in the call, and manipulate them myself into my own graph etc, giving me captured history from yesterday as well as a full display for today and tomorrow. In addition I now have access to useable and worthwhile power and energy forecast data which makes the HA integration redundant.
I believe I fully understand how your service and API works, including the difference between the hourly power values and the hour-period watt_hour energy values, and how these are calculated and presented in the API call.
Why calculate the watt hours? Why indeed.
This (see ‘created here’ link) is apparently the code used in the HA integration. As far as I understand Python, this code runs around the power values, calculating the watt_hours, and placing them into an array to be used to plot the graph forecast line.
So, it is currently 11:15 here in the UK - local time being BST which is DST for GMT.
My HA energy graph has just updated the actual production for the past hour - which is the period 10:00 to 11:00
This bar appears at 10:00 on the graph. Note - this is the START of the plotted hour period.
The forecast overlay is a line plot. To work correctly, the forecast value for the hour period 10:00 to 11:00 has to be likewise plotted at 10:00.
My conjecture is this. The looping code that calculated the energy from the power subtracts the last hour power from the current hour power and then plots this at the current hour. Or, in English, the forecast for the period hour A to hour B is Power(hour B) - Power(hour A) plotted at hour B.
My conclusion is that the solution is to plot at hour A (not B), or to use your already provided watt_hour figures but to make sure that the value for each hour is plotted on the graph at the START of the hour period, not the END of the hour period.
I have spent an inordinate amount of time on this and I believe that I am unable to add any further value to this discussion.
The power values are mapped as they are also received by the API, nothing else happens with that. As seen in the code. But the watt_hours values we get from the API are used to calculate the differences between the two time points and it is necessary because this list only adds up to the maximum of what you could possibly produce on that day.
The problem now is whether to plot that value on the beginning or the end of a time frame. Currently that is done at the end so that it counts to the next time frame, but it should actually be at the beginning so that it no longer has a shift of 1 hour.
The only problem is that I also see graphs of people where the forecast is going exactly right compared to their real product (in The Netherlands). What makes the difference? Is it the time zone? or something else?
OK.
So we agree - the (however it is calculated) forecast values are plotted at the back end of the time period.
Clearly, for me here in the UK at the moment, the actual is plotted at the front end of the time period, and the forecast at the back end of the time period. Ergo problem.
However, if you see graphs where both actual and forecast line up, my question would now be, on these graphs is the actual plotted at the end of the time period (ie matching with the forecast)?
If so, that would perhaps suggest that there is something else going on with the energy plot that makes the UK graphs plot the actual at the front when elsewhere they are plotted at the back.
I assume that HA works internally on UTC and converts to local time zone on the fly. Here in the UK we are almost uniquely with UTC and GMT being equal (except for DST). All my HA times are UTC but displayed correctly as BST. I had not considered this but perhaps the issue is that my actual is actually on the wrong hour and not the forecast?
I have done all I can to check the API call, times, plots and actuals. Someone somewhere else where the graph is correct will have to check to see if the forecast is produced to local time, and where the actual is being plotted - front or back.
Thought - just need someone in, say The Netherlands, to capture a screenshot of their energy actual at, say 12:15 after update, to see if the 11:00 to 12:00 actual is plotted against 11:00 or 12:00.
Thanks for your PR! I would like to test this out myself. How can I go about making this change manually on my Home Assistant setup? I searched the file system Docker uses for Home Assistant and I was unable to locate hui-energy-solar-graph-card.ts. Where can I locate this file to make the modification?
One possibility is to fork the frontend repo and then load it into the frontend integration. However, this is easier in a HA development environment than on a production environment with HA OS.
I’m not using HA OS. I just installed Home Assistant in Docker on an existing Debian Bullseye server I use for a number of tasks (ZFS main and backup storage, Plex etc.).