Can't get forecast.solar forecast to line up with my production

My high point throughout the day is 1pm. Tried to make my forecast more accurate by modifying the azimuth from 180 to the actual value of 227, as well as the declination of the panels (default is 25, mine are at 28, so not a big change). What I got though is completely inaccurate - I can’t get the forecast to shift to the left (earlier in the day). When I move it to 227, it says high is 3pm. I went drastic the other direction and 160 looks almost identical to 180, but is FAR more accurate than 180 is on the forecast.

I produce quite a bit in the morning and the forecasts just don’t line up. Is there any way to really fix this?

Here’s the chart for 227 degrees

Here’s the chart for 160 degrees

1 Like

Hi,
I admit that i don’t know much about how the forecast.solar service arrives at the results, but at the end of the day it’s just a “forecast” right. Like a weather forecast, or is it supposed to be more accurate than that. If it’s a forecast, or educated guess, then i would be surprised (actually bloody amazed) if it was spot on. I imagine the forecast for expected solar generation can’t really account in real-time for cloud cover etc.
I think it’s a great service, i’m using it too. I have also found the forecasted solar to be way different to the actual solar as measured by my system. Now i’m more curious about how the data is generated.
Cheers
Nick

Well, I get that it’s a forecast, but the problem I’m experiencing is related to the actual input fields for azimuth and declination. I’m ok if it shows I under/over produce, or even have the lines off a little, but if you look at the correct inputs using the chart on the top, it’s nowhere close.

I’m not discrediting the forecast, I do understand that it’s by no means an end-all, matter of fact measurement, I’m disputing the fact that its overlay is inaccurate when using actual azimuth and declination figures.

I have been spending some time recently trying to get more out of the integrated solar forecast, and finding the provided entities not quite what I want, I have resorted to fetching and manipulating the data in Node-Red myself. Now I have a complete forecast for the next two days, updates when I want, and an array of ‘power levels’ with start/end times and duration.

In doing all of this I have been working out how the integration is set up to get the figures it does, and I’m finding that my values do not always agree. I also suspect that the displayed forecast chart on the energy page is one hour out and shifted to the right from where it should be. This might explain why my actual generation does not seem to match the forecast.

Here is the logic:
solar.forecast API returns a list of power values (watts) for each hour, effective at the hour, as well as a list of watt-hours for each hour period. All times are local (confirmed since the start/end time matches with my sunrise/sunset) so being on daylight saving in the UK shouldn’t be the issue. For example, as I write this at 15:30, at 18:00 today plane 1 forecast (API values) is 367 watts, and at 19:00 247 watts. The watt-hours figures are available in the API return, but can also be calculated simply by averaging the power over the hour. So, assuming a straight line between 18:00 and 19:00, the average power is 307 watts, and assuming this is for the full hour, equates to 307 watt-hours between 18:00 and 19:00. I have two planes, so doing the same for the second gives me a total 1184 watt-hours for 18:00 to 19:00. Round this to 1.19 kWh as is plotted on the energy page, and there it is, but for 19:00 to 20:00 ! Bingo.

The more I look closely at the forecast line, the more I see that the first and the last values (start/end of day) are out by one hour, so I’m of the thought that the graph should be shifted left by one hour. I wonder if the integration code has been written to add in daylight saving, as in the UK I am currently 1 hour ahead of UTC (BST and not GMT) although solar.forecast returns results based on the local time of the location provided?

The top chart is the standard HA energy page, the bottom chart is something I have just managed to get working based on the API results from solar.forecast (today in the middle, left is ‘history’ and right is tomorrow). It does not show up at all well, but I think my orange forecast line is one hour to the left of the dotted line. Hmmm.

Well that’s super interesting and would actually cause my issue to make sense.

You’ve proven this is a far more intricate and involved topic than I previously thought.

Yes, the more I look at solar forecasting the more involved and challenging it becomes. I sat down this morning and pulled all the figures again to check, and yes the integration forecast graph is being incorrectly plotted one hour to the right and I think this is just a mistake with the coding. I don’t have a GitHub account (I don’t want to go down that route, this stuff is too complicated already) so I can’t report this feature/bug via the approved mechanism, but hereby follows the logic for anyone else who is struggling to get the forecast.solar dotted line to match their generation.

Lovely nice forecast this morning, just the sort of day to test out my graph and code! (screenshot taken at 9:30, the integration had just updated at 9:20)

Notice the very big dip, showing on the forecast graph at 12:00 - 13:00 (plotted where the 12:00 - 13:00 generation bar will later appear). This is for the time period starting at 12:00 and ending at 13:00.

The forecast.solar API call returns a list of watts and a list of watt-hours, both set as values for the hour, local time zone. I have two integrations, so I additionally manually called the API once for each plane. Adding the figures together I get

At 11:00 - 131 watts
At 12:00 - 222 watts

these figures are for power at the hour, so a simple calculation gives an average power of 177, or 177 watt-hours (177 watts for one hour) for the period 11:00 to 12:00

The watt-hour figures from the API are provided as cumulative, being

At 11:00 - 3978 watt-hours
At 12:00 - 4155 watt-hours

which means that the increase is 177 watt-hours during the time period 11:00 to 12:00.

The calculation matches as I expect, so we can work in terms of 131 watts at 11:00, 177 watts at 11:30, 222 watts at 12:00 or 177 watt-hours at 11:30 (for 11:00 to 12:00). To get the forecast for each hour period, the calculation is to subtract the last hour cumulative value from this hour cumulative value. However, and this is the tricky bit, that calculation provides the expected increase for the period starting at the last hour, not this hour. The integration plots 177 (rounded to 2dp at 0.18 kWh) at 12:00 (to 13:00) when it should plot that value at 11:00 (to 12:00).

I rest my case m’lud.

And here is a screenshot at 13:25. Clearly the forecast has changed, which is to be expected. For my graph I am directly calling the forecast.solar API and plotting the watts (the integration uses the watt-hours). I want to be able to forecast power levels for turning appliances on, so I think watts will be of more use. My graph and the integration graph will never quite match therefore, but they are closely related (as the integration graph being based on watt-hours is just a plot of the average for each pair of points on the watt graph).

Worth noting however that the forecast changes, quite regularly, and sometimes quite a lot. Also it continues to change even when it has passed ‘now’ and becomes history. I am saving and plotting the past-hour generation figure from my inverter sensor, and at the same time I am now saving the ‘current’ forecast value for that hour (one hour old). This is the magenta line. I am finding that the forecast seems to revert to a clear-sky plot. Yes the morning part is now old and no longer a forecast, however the forecast watt-hour figures also change, and the integration ‘energy production today’ figure increases making it unreliable once we are moving into the current day.

Not sure if anyone is actually going to read this stuff, but if you do I hope you have found it interesting and useful. :nerd_face:

1 Like

I am so thoroughly impressed with your reply. I think you hit the nail on the head - and I’m surprised it hasn’t come up in the past, this is a fairly popular integration. I think many probably accept the default values and don’t question since the forecast tends to line up nicely - at least for me.

I do have a github account and will report this.

Thanks for the hard work. Hopefully your comment is satisfactory enough to get the dev(s) looking at it.

Linked on github: https://github.com/home-assistant/core/issues/74375

Looking at old posts / publicity screenshots I suspect that this is the result of a recent change. Probably worked correctly at some point. And yes strange that no one else appears to have noticed it. I do wonder how many people actually use this feature - given that in practical terms you can’t do very much with it.

I had solar pv installed last October, and I bought an Odroid with HA (Blue) last year, basically because of this very forecast feature and the energy dashboard, to form the basis for a monitoring and control system for my solar pv. Frankly, whilst I like HA very much, I have found the forecast of little practical use, the energy dashboard I have replaced with another community power card, and I do most of my work in Node-Red (which I understand and can use). I can see how powerful HA is when creating one sensor and one associated automation, but to do anything more complex seems to require many inter-connected entities written in yaml, and thus the spiderweb of bits and pieces rapidly becomes a ‘write only language’, impossible to maintain!

Thank you for reporting it. I would happily have done that myself but I am drawing the line at GitHub as I think there has to be a limit to the amount of coding stuff I get involved in. Inline jason and jinja are, frankly, testing my abilities to the limit. :confused:

I’ve read the entire thread, but it’s very difficult to figure out where exactly this is going wrong. Is it in the API that provides the data, is it the python package that incorrectly processes the obtained data or in the integration that actually all runs through the list.

I am wondering if it has something to do with DST, may be on the API end.

Quite interesting. I think i got this same thing going on for as long as it can be seen in the energy dashboard:

Here you can see the exact thing going on. I more or less accepted it since the data in the end of the day is quite accurate.

Glad to know it’s happening to everyone - Once I pumped in accurate location data, the forecast became pretty worthless. But I’m a little annoyed with the dev - i opened an issue in GitHub and he said to keep it here, but has gone silent.

It’s not just snapping your fingers and it’s solved … I also do all this in my spare time and I haven’t had it very quiet lately. The problem that I mentioned earlier is that it is still unknown where this problem lies.

1 Like

Sorry, I get that. I could have phrased it better.

I appreciate the work you do on this and hope for a fix. Apologies for acting ungrateful.

2 Likes

I need some more information to rule things out. For those who have problems with a shifted production graph, see if you can download a diagnostic file from HA and an export of this directly from the API, if you access the url. In this way we can compare and see if something goes wrong in the data processing.

OK. I guess that will be me then.

Diagnostic EAST plane:

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.7.6",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.45",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.2",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH East",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 73,
        "modules power": 2150,
        "damping": 0.0
      }
    },
    "data": {
      "energy_production_today": 14783,
      "energy_production_tomorrow": 2716,
      "energy_current_hour": 277,
      "power_production_now": 215,
      "watts": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 617,
        "2022-07-21T07:00:00+01:00": 1098,
        "2022-07-21T08:00:00+01:00": 1462,
        "2022-07-21T09:00:00+01:00": 1685,
        "2022-07-21T10:00:00+01:00": 1757,
        "2022-07-21T11:00:00+01:00": 1725,
        "2022-07-21T12:00:00+01:00": 1595,
        "2022-07-21T13:00:00+01:00": 1372,
        "2022-07-21T14:00:00+01:00": 1136,
        "2022-07-21T15:00:00+01:00": 806,
        "2022-07-21T16:00:00+01:00": 523,
        "2022-07-21T17:00:00+01:00": 425,
        "2022-07-21T18:00:00+01:00": 340,
        "2022-07-21T19:00:00+01:00": 215,
        "2022-07-21T20:00:00+01:00": 93,
        "2022-07-21T21:00:00+01:00": 17,
        "2022-07-21T21:17:00+01:00": 0,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 6,
        "2022-07-22T07:00:00+01:00": 35,
        "2022-07-22T08:00:00+01:00": 49,
        "2022-07-22T09:00:00+01:00": 24,
        "2022-07-22T10:00:00+01:00": 27,
        "2022-07-22T11:00:00+01:00": 29,
        "2022-07-22T12:00:00+01:00": 188,
        "2022-07-22T13:00:00+01:00": 323,
        "2022-07-22T14:00:00+01:00": 367,
        "2022-07-22T15:00:00+01:00": 404,
        "2022-07-22T16:00:00+01:00": 323,
        "2022-07-22T17:00:00+01:00": 343,
        "2022-07-22T18:00:00+01:00": 277,
        "2022-07-22T19:00:00+01:00": 207,
        "2022-07-22T20:00:00+01:00": 102,
        "2022-07-22T21:00:00+01:00": 20,
        "2022-07-22T21:15:00+01:00": 0
      },
      "wh_days": {
        "2022-07-21T00:00:00": 14783,
        "2022-07-22T00:00:00": 2716
      },
      "wh_hours": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 231,
        "2022-07-21T07:00:00+01:00": 858,
        "2022-07-21T08:00:00+01:00": 1280,
        "2022-07-21T09:00:00+01:00": 1573,
        "2022-07-21T10:00:00+01:00": 1721,
        "2022-07-21T11:00:00+01:00": 1741,
        "2022-07-21T12:00:00+01:00": 1660,
        "2022-07-21T13:00:00+01:00": 1484,
        "2022-07-21T14:00:00+01:00": 1254,
        "2022-07-21T15:00:00+01:00": 971,
        "2022-07-21T16:00:00+01:00": 664,
        "2022-07-21T17:00:00+01:00": 474,
        "2022-07-21T18:00:00+01:00": 383,
        "2022-07-21T19:00:00+01:00": 277,
        "2022-07-21T20:00:00+01:00": 154,
        "2022-07-21T21:00:00+01:00": 55,
        "2022-07-21T21:17:00+01:00": 3,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 2,
        "2022-07-22T07:00:00+01:00": 21,
        "2022-07-22T08:00:00+01:00": 42,
        "2022-07-22T09:00:00+01:00": 36,
        "2022-07-22T10:00:00+01:00": 26,
        "2022-07-22T11:00:00+01:00": 28,
        "2022-07-22T12:00:00+01:00": 108,
        "2022-07-22T13:00:00+01:00": 256,
        "2022-07-22T14:00:00+01:00": 345,
        "2022-07-22T15:00:00+01:00": 385,
        "2022-07-22T16:00:00+01:00": 364,
        "2022-07-22T17:00:00+01:00": 333,
        "2022-07-22T18:00:00+01:00": 310,
        "2022-07-22T19:00:00+01:00": 242,
        "2022-07-22T20:00:00+01:00": 154,
        "2022-07-22T21:00:00+01:00": 61,
        "2022-07-22T21:15:00+01:00": 3
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

API direct call

{
   "watts":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":617,
      "2022-07-21 07:00:00":1098,
      "2022-07-21 08:00:00":1462,
      "2022-07-21 09:00:00":1688,
      "2022-07-21 10:00:00":1769,
      "2022-07-21 11:00:00":1735,
      "2022-07-21 12:00:00":1601,
      "2022-07-21 13:00:00":1370,
      "2022-07-21 14:00:00":1135,
      "2022-07-21 15:00:00":834,
      "2022-07-21 16:00:00":552,
      "2022-07-21 17:00:00":424,
      "2022-07-21 18:00:00":339,
      "2022-07-21 19:00:00":225,
      "2022-07-21 20:00:00":92,
      "2022-07-21 21:00:00":16,
      "2022-07-21 21:17:00":0,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":38,
      "2022-07-22 07:00:00":51,
      "2022-07-22 08:00:00":19,
      "2022-07-22 09:00:00":115,
      "2022-07-22 10:00:00":222,
      "2022-07-22 11:00:00":185,
      "2022-07-22 12:00:00":324,
      "2022-07-22 13:00:00":331,
      "2022-07-22 14:00:00":392,
      "2022-07-22 15:00:00":422,
      "2022-07-22 16:00:00":385,
      "2022-07-22 17:00:00":298,
      "2022-07-22 18:00:00":274,
      "2022-07-22 19:00:00":200,
      "2022-07-22 20:00:00":101,
      "2022-07-22 21:00:00":20,
      "2022-07-22 21:15:00":0
   },
   "watt_hours":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":231,
      "2022-07-21 07:00:00":1089,
      "2022-07-21 08:00:00":2369,
      "2022-07-21 09:00:00":3944,
      "2022-07-21 10:00:00":5672,
      "2022-07-21 11:00:00":7424,
      "2022-07-21 12:00:00":9092,
      "2022-07-21 13:00:00":10578,
      "2022-07-21 14:00:00":11830,
      "2022-07-21 15:00:00":12815,
      "2022-07-21 16:00:00":13508,
      "2022-07-21 17:00:00":13996,
      "2022-07-21 18:00:00":14377,
      "2022-07-21 19:00:00":14659,
      "2022-07-21 20:00:00":14818,
      "2022-07-21 21:00:00":14872,
      "2022-07-21 21:17:00":14874,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":14,
      "2022-07-22 07:00:00":58,
      "2022-07-22 08:00:00":93,
      "2022-07-22 09:00:00":160,
      "2022-07-22 10:00:00":329,
      "2022-07-22 11:00:00":532,
      "2022-07-22 12:00:00":787,
      "2022-07-22 13:00:00":1114,
      "2022-07-22 14:00:00":1476,
      "2022-07-22 15:00:00":1883,
      "2022-07-22 16:00:00":2286,
      "2022-07-22 17:00:00":2628,
      "2022-07-22 18:00:00":2914,
      "2022-07-22 19:00:00":3151,
      "2022-07-22 20:00:00":3301,
      "2022-07-22 21:00:00":3362,
      "2022-07-22 21:15:00":3364
   },
   "watt_hours_day":{
      "2022-07-21":14874,
      "2022-07-22":3364
   }
}

Diagnostic WEST plane

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.7.6",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.45",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.2",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH West",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 253,
        "modules power": 2150,
        "damping": 0.0
      }
    },
    "data": {
      "energy_production_today": 15465,
      "energy_production_tomorrow": 5601,
      "energy_current_hour": 780,
      "power_production_now": 565,
      "watts": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 127,
        "2022-07-21T07:00:00+01:00": 237,
        "2022-07-21T08:00:00+01:00": 343,
        "2022-07-21T09:00:00+01:00": 554,
        "2022-07-21T10:00:00+01:00": 917,
        "2022-07-21T11:00:00+01:00": 1298,
        "2022-07-21T12:00:00+01:00": 1620,
        "2022-07-21T13:00:00+01:00": 1823,
        "2022-07-21T14:00:00+01:00": 1990,
        "2022-07-21T15:00:00+01:00": 1846,
        "2022-07-21T16:00:00+01:00": 1576,
        "2022-07-21T17:00:00+01:00": 1335,
        "2022-07-21T18:00:00+01:00": 995,
        "2022-07-21T19:00:00+01:00": 565,
        "2022-07-21T20:00:00+01:00": 222,
        "2022-07-21T21:00:00+01:00": 51,
        "2022-07-21T21:17:00+01:00": 0,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 2,
        "2022-07-22T07:00:00+01:00": 12,
        "2022-07-22T08:00:00+01:00": 19,
        "2022-07-22T09:00:00+01:00": 14,
        "2022-07-22T10:00:00+01:00": 19,
        "2022-07-22T11:00:00+01:00": 24,
        "2022-07-22T12:00:00+01:00": 189,
        "2022-07-22T13:00:00+01:00": 412,
        "2022-07-22T14:00:00+01:00": 590,
        "2022-07-22T15:00:00+01:00": 849,
        "2022-07-22T16:00:00+01:00": 882,
        "2022-07-22T17:00:00+01:00": 1002,
        "2022-07-22T18:00:00+01:00": 764,
        "2022-07-22T19:00:00+01:00": 539,
        "2022-07-22T20:00:00+01:00": 247,
        "2022-07-22T21:00:00+01:00": 60,
        "2022-07-22T21:15:00+01:00": 0
      },
      "wh_days": {
        "2022-07-21T00:00:00": 15465,
        "2022-07-22T00:00:00": 5601
      },
      "wh_hours": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 48,
        "2022-07-21T07:00:00+01:00": 182,
        "2022-07-21T08:00:00+01:00": 290,
        "2022-07-21T09:00:00+01:00": 448,
        "2022-07-21T10:00:00+01:00": 736,
        "2022-07-21T11:00:00+01:00": 1107,
        "2022-07-21T12:00:00+01:00": 1459,
        "2022-07-21T13:00:00+01:00": 1722,
        "2022-07-21T14:00:00+01:00": 1906,
        "2022-07-21T15:00:00+01:00": 1918,
        "2022-07-21T16:00:00+01:00": 1711,
        "2022-07-21T17:00:00+01:00": 1456,
        "2022-07-21T18:00:00+01:00": 1165,
        "2022-07-21T19:00:00+01:00": 780,
        "2022-07-21T20:00:00+01:00": 393,
        "2022-07-21T21:00:00+01:00": 137,
        "2022-07-21T21:17:00+01:00": 7,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 1,
        "2022-07-22T07:00:00+01:00": 7,
        "2022-07-22T08:00:00+01:00": 15,
        "2022-07-22T09:00:00+01:00": 17,
        "2022-07-22T10:00:00+01:00": 16,
        "2022-07-22T11:00:00+01:00": 22,
        "2022-07-22T12:00:00+01:00": 106,
        "2022-07-22T13:00:00+01:00": 301,
        "2022-07-22T14:00:00+01:00": 501,
        "2022-07-22T15:00:00+01:00": 719,
        "2022-07-22T16:00:00+01:00": 866,
        "2022-07-22T17:00:00+01:00": 942,
        "2022-07-22T18:00:00+01:00": 883,
        "2022-07-22T19:00:00+01:00": 651,
        "2022-07-22T20:00:00+01:00": 393,
        "2022-07-22T21:00:00+01:00": 154,
        "2022-07-22T21:15:00+01:00": 7
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

API direct call (West)

{
   "watts":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":127,
      "2022-07-21 07:00:00":237,
      "2022-07-21 08:00:00":343,
      "2022-07-21 09:00:00":553,
      "2022-07-21 10:00:00":916,
      "2022-07-21 11:00:00":1301,
      "2022-07-21 12:00:00":1626,
      "2022-07-21 13:00:00":1818,
      "2022-07-21 14:00:00":1998,
      "2022-07-21 15:00:00":1934,
      "2022-07-21 16:00:00":1681,
      "2022-07-21 17:00:00":1331,
      "2022-07-21 18:00:00":995,
      "2022-07-21 19:00:00":603,
      "2022-07-21 20:00:00":220,
      "2022-07-21 21:00:00":49,
      "2022-07-21 21:17:00":0,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":10,
      "2022-07-22 07:00:00":16,
      "2022-07-22 08:00:00":10,
      "2022-07-22 09:00:00":49,
      "2022-07-22 10:00:00":128,
      "2022-07-22 11:00:00":145,
      "2022-07-22 12:00:00":326,
      "2022-07-22 13:00:00":422,
      "2022-07-22 14:00:00":633,
      "2022-07-22 15:00:00":887,
      "2022-07-22 16:00:00":1066,
      "2022-07-22 17:00:00":856,
      "2022-07-22 18:00:00":753,
      "2022-07-22 19:00:00":516,
      "2022-07-22 20:00:00":245,
      "2022-07-22 21:00:00":60,
      "2022-07-22 21:15:00":0
   },
   "watt_hours":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":48,
      "2022-07-21 07:00:00":230,
      "2022-07-21 08:00:00":520,
      "2022-07-21 09:00:00":968,
      "2022-07-21 10:00:00":1702,
      "2022-07-21 11:00:00":2811,
      "2022-07-21 12:00:00":4274,
      "2022-07-21 13:00:00":5996,
      "2022-07-21 14:00:00":7904,
      "2022-07-21 15:00:00":9870,
      "2022-07-21 16:00:00":11678,
      "2022-07-21 17:00:00":13184,
      "2022-07-21 18:00:00":14347,
      "2022-07-21 19:00:00":15146,
      "2022-07-21 20:00:00":15557,
      "2022-07-21 21:00:00":15692,
      "2022-07-21 21:17:00":15699,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":4,
      "2022-07-22 07:00:00":17,
      "2022-07-22 08:00:00":30,
      "2022-07-22 09:00:00":59,
      "2022-07-22 10:00:00":148,
      "2022-07-22 11:00:00":284,
      "2022-07-22 12:00:00":520,
      "2022-07-22 13:00:00":894,
      "2022-07-22 14:00:00":1421,
      "2022-07-22 15:00:00":2181,
      "2022-07-22 16:00:00":3158,
      "2022-07-22 17:00:00":4119,
      "2022-07-22 18:00:00":4923,
      "2022-07-22 19:00:00":5558,
      "2022-07-22 20:00:00":5938,
      "2022-07-22 21:00:00":6091,
      "2022-07-22 21:15:00":6098
   },
   "watt_hours_day":{
      "2022-07-21":15699,
      "2022-07-22":6098
   }
}

Current energy graph (screen shot)

Calculations:

API calls and diagnostics appear to align (within slight tolerance - both updated at around the same time)

Watts - East and West. Sunrise is correct at 5:15
5:15 - 0 & 0 (sunrise)
6:00 - 617 & 127
7:00 -1098 & 237
8:00 - 1462 & 343

I assume that this is the POWER at the hour.

The watt-hour will be the average of the two powers at the hours
ie for 6:00 to 7:00 = (617+1098)/2 = 858 (east) and 182 (west)

Compare to the API/diagnostic…

Watt-hours (for East & West)
API call (cumulative watt-hours)
6:00 = 231 / 48
7:00 = 1089 / 230

Diagnostic (which shows the difference not the cumulative)
7:00 = 858 / 182
which is indeed 1089-231 and 230-48

So for the hour 6:00 to 7:00 the watt-hours are clearly 858 and 182 (ie at 5:15 nothing, at 6:00 231, at 7:00 1089, increase of 858 during that hour)

The energy graph, being the sum of the two, is showing for 7:00 to 8:00 as 1.04 kWh or 1040
858+182 = 1040

Therefore the calculations are correct. API agrees with diagnostic agrees with watts to watt-hour calculation agrees with cumulative to difference.

Here is the bit that I have the problem with.

If the cumulative energy (east plane only) at 7:00 is 1089, and at 6:00 231, then the difference 858 applies between 6:00 and 7:00, but this is being shown on the graph for 7:00 to 8:00

Or have I completely lost the plot here?

The short version

Cumulative (west plane only) watt-hours from API call
at 5:15 - 0
at 6:00 - 48
at 7:00 - 230
at 8:00 - 520
at 9:00 - 968

Difference by calculation
between 5:15 - 6:00 48
between 6:00 - 7:00 182 =230-48
between 7:00 - 8:00 290

Diagnostic
at 5:15 - 0
at 6:00 - 48
at 7:00 - 182
at 8:00 - 290

Graph
at 7:00 (for 7:00 to 8:00) - 182

I have tried to interpret your information as well as possible and now I understand better where the problem could lie. For an even better comparison could you make the direct API call with a
?time=iso8601 parameter? Because the diagnostic data shows a UTC offset and that of the direct API call does not (by default the api returns data that applies to your own timezone).

The dict that Home Assistant uses with the differences are created here in the for loop.

Just to rule out one more thing, is your Home Assistant timezone set to Europe/London?

My HA device is set to GMT+00:00, London in settings.
We are currently on BST which is one hour ahead for DST. I have no idea how HA knows we are on DST, but all the timestamps internally are UTC, yet for display everything is in my current local time so it all seems to work correctly.

All data collected this morning between 10:00 and 11:00 BST (GMT+1)

East - HA diagnostic dump:

  • timezone is Europe/London
  • sunrise 05:45 (correct for current local time GMT+1 British Summer Time)
  • watts at 10:00 = 1574, 9:00 = 1531
  • watthours 10:00 = average = 1553 (agrees)
{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.8.3",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.55",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.4",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH East",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 73,
        "modules power": 2190,
        "damping": 0.0,
        "inverter_size": 4600
      }
    },
    "data": {
      "energy_production_today": 12368,
      "energy_production_tomorrow": 11061,
      "energy_current_hour": 1553,
      "power_production_now": 1574,
      "watts": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 636,
        "2022-08-10T07:00:00+01:00": 1099,
        "2022-08-10T08:00:00+01:00": 1355,
        "2022-08-10T09:00:00+01:00": 1531,
        "2022-08-10T10:00:00+01:00": 1574,
        "2022-08-10T11:00:00+01:00": 1493,
        "2022-08-10T12:00:00+01:00": 1319,
        "2022-08-10T13:00:00+01:00": 1092,
        "2022-08-10T14:00:00+01:00": 870,
        "2022-08-10T15:00:00+01:00": 595,
        "2022-08-10T16:00:00+01:00": 353,
        "2022-08-10T17:00:00+01:00": 277,
        "2022-08-10T18:00:00+01:00": 215,
        "2022-08-10T19:00:00+01:00": 141,
        "2022-08-10T20:00:00+01:00": 65,
        "2022-08-10T20:45:00+01:00": 0,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 593,
        "2022-08-11T07:00:00+01:00": 993,
        "2022-08-11T08:00:00+01:00": 1273,
        "2022-08-11T09:00:00+01:00": 1423,
        "2022-08-11T10:00:00+01:00": 1433,
        "2022-08-11T11:00:00+01:00": 1343,
        "2022-08-11T12:00:00+01:00": 1182,
        "2022-08-11T13:00:00+01:00": 979,
        "2022-08-11T14:00:00+01:00": 759,
        "2022-08-11T15:00:00+01:00": 507,
        "2022-08-11T16:00:00+01:00": 277,
        "2022-08-11T17:00:00+01:00": 211,
        "2022-08-11T18:00:00+01:00": 165,
        "2022-08-11T19:00:00+01:00": 111,
        "2022-08-11T20:00:00+01:00": 51,
        "2022-08-11T20:43:00+01:00": 0
      },
      "wh_days": {
        "2022-08-10T00:00:00": 12368,
        "2022-08-11T00:00:00": 11061
      },
      "wh_hours": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 80,
        "2022-08-10T07:00:00+01:00": 867,
        "2022-08-10T08:00:00+01:00": 1227,
        "2022-08-10T09:00:00+01:00": 1443,
        "2022-08-10T10:00:00+01:00": 1553,
        "2022-08-10T11:00:00+01:00": 1533,
        "2022-08-10T12:00:00+01:00": 1406,
        "2022-08-10T13:00:00+01:00": 1206,
        "2022-08-10T14:00:00+01:00": 981,
        "2022-08-10T15:00:00+01:00": 732,
        "2022-08-10T16:00:00+01:00": 474,
        "2022-08-10T17:00:00+01:00": 315,
        "2022-08-10T18:00:00+01:00": 246,
        "2022-08-10T19:00:00+01:00": 178,
        "2022-08-10T20:00:00+01:00": 103,
        "2022-08-10T20:45:00+01:00": 24,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 64,
        "2022-08-11T07:00:00+01:00": 793,
        "2022-08-11T08:00:00+01:00": 1133,
        "2022-08-11T09:00:00+01:00": 1348,
        "2022-08-11T10:00:00+01:00": 1428,
        "2022-08-11T11:00:00+01:00": 1388,
        "2022-08-11T12:00:00+01:00": 1263,
        "2022-08-11T13:00:00+01:00": 1080,
        "2022-08-11T14:00:00+01:00": 869,
        "2022-08-11T15:00:00+01:00": 633,
        "2022-08-11T16:00:00+01:00": 392,
        "2022-08-11T17:00:00+01:00": 244,
        "2022-08-11T18:00:00+01:00": 188,
        "2022-08-11T19:00:00+01:00": 138,
        "2022-08-11T20:00:00+01:00": 81,
        "2022-08-11T20:43:00+01:00": 19
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

West diagnostic dump

  • watts 10:00 = 770 , 9:00 = 451
  • watthours 10:00 = average = 611 (agrees)
{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.8.3",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.55",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.4",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH West",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 253,
        "modules power": 2190,
        "damping": 0.0,
        "inverter_size": 4600
      }
    },
    "data": {
      "energy_production_today": 13785,
      "energy_production_tomorrow": 12285,
      "energy_current_hour": 611,
      "power_production_now": 770,
      "watts": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 131,
        "2022-08-10T07:00:00+01:00": 227,
        "2022-08-10T08:00:00+01:00": 271,
        "2022-08-10T09:00:00+01:00": 451,
        "2022-08-10T10:00:00+01:00": 770,
        "2022-08-10T11:00:00+01:00": 1091,
        "2022-08-10T12:00:00+01:00": 1352,
        "2022-08-10T13:00:00+01:00": 1541,
        "2022-08-10T14:00:00+01:00": 1700,
        "2022-08-10T15:00:00+01:00": 1691,
        "2022-08-10T16:00:00+01:00": 1569,
        "2022-08-10T17:00:00+01:00": 1325,
        "2022-08-10T18:00:00+01:00": 977,
        "2022-08-10T19:00:00+01:00": 560,
        "2022-08-10T20:00:00+01:00": 203,
        "2022-08-10T20:45:00+01:00": 0,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 107,
        "2022-08-11T07:00:00+01:00": 166,
        "2022-08-11T08:00:00+01:00": 213,
        "2022-08-11T09:00:00+01:00": 378,
        "2022-08-11T10:00:00+01:00": 666,
        "2022-08-11T11:00:00+01:00": 963,
        "2022-08-11T12:00:00+01:00": 1217,
        "2022-08-11T13:00:00+01:00": 1411,
        "2022-08-11T14:00:00+01:00": 1536,
        "2022-08-11T15:00:00+01:00": 1534,
        "2022-08-11T16:00:00+01:00": 1413,
        "2022-08-11T17:00:00+01:00": 1186,
        "2022-08-11T18:00:00+01:00": 878,
        "2022-08-11T19:00:00+01:00": 509,
        "2022-08-11T20:00:00+01:00": 175,
        "2022-08-11T20:43:00+01:00": 0
      },
      "wh_days": {
        "2022-08-10T00:00:00": 13785,
        "2022-08-11T00:00:00": 12285
      },
      "wh_hours": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 16,
        "2022-08-10T07:00:00+01:00": 179,
        "2022-08-10T08:00:00+01:00": 249,
        "2022-08-10T09:00:00+01:00": 361,
        "2022-08-10T10:00:00+01:00": 611,
        "2022-08-10T11:00:00+01:00": 930,
        "2022-08-10T12:00:00+01:00": 1222,
        "2022-08-10T13:00:00+01:00": 1446,
        "2022-08-10T14:00:00+01:00": 1621,
        "2022-08-10T15:00:00+01:00": 1695,
        "2022-08-10T16:00:00+01:00": 1630,
        "2022-08-10T17:00:00+01:00": 1447,
        "2022-08-10T18:00:00+01:00": 1151,
        "2022-08-10T19:00:00+01:00": 769,
        "2022-08-10T20:00:00+01:00": 381,
        "2022-08-10T20:45:00+01:00": 77,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 12,
        "2022-08-11T07:00:00+01:00": 136,
        "2022-08-11T08:00:00+01:00": 190,
        "2022-08-11T09:00:00+01:00": 295,
        "2022-08-11T10:00:00+01:00": 522,
        "2022-08-11T11:00:00+01:00": 815,
        "2022-08-11T12:00:00+01:00": 1090,
        "2022-08-11T13:00:00+01:00": 1314,
        "2022-08-11T14:00:00+01:00": 1473,
        "2022-08-11T15:00:00+01:00": 1535,
        "2022-08-11T16:00:00+01:00": 1474,
        "2022-08-11T17:00:00+01:00": 1299,
        "2022-08-11T18:00:00+01:00": 1032,
        "2022-08-11T19:00:00+01:00": 694,
        "2022-08-11T20:00:00+01:00": 342,
        "2022-08-11T20:43:00+01:00": 62
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

Direct API call (with same settings, parameter ?time=iso8601)

East:

  • watts (same values as diagnostic 10:00 = 1574, 9:00 = 1531)
{"2022-08-10T05:45:00+01:00":0,"2022-08-10T06:00:00+01:00":636,"2022-08-10T07:00:00+01:00":1099,"2022-08-10T08:00:00+01:00":1355,"2022-08-10T09:00:00+01:00":1531,"2022-08-10T10:00:00+01:00":1574,"2022-08-10T11:00:00+01:00":1493,"2022-08-10T12:00:00+01:00":1319,"2022-08-10T13:00:00+01:00":1092,"2022-08-10T14:00:00+01:00":870,"2022-08-10T15:00:00+01:00":595,"2022-08-10T16:00:00+01:00":353,"2022-08-10T17:00:00+01:00":277,"2022-08-10T18:00:00+01:00":215,"2022-08-10T19:00:00+01:00":141,"2022-08-10T20:00:00+01:00":65,"2022-08-10T20:45:00+01:00":0,"2022-08-11T05:47:00+01:00":0,"2022-08-11T06:00:00+01:00":593,"2022-08-11T07:00:00+01:00":993,"2022-08-11T08:00:00+01:00":1273,"2022-08-11T09:00:00+01:00":1423,"2022-08-11T10:00:00+01:00":1433,"2022-08-11T11:00:00+01:00":1343,"2022-08-11T12:00:00+01:00":1182,"2022-08-11T13:00:00+01:00":979,"2022-08-11T14:00:00+01:00":759,"2022-08-11T15:00:00+01:00":507,"2022-08-11T16:00:00+01:00":277,"2022-08-11T17:00:00+01:00":211,"2022-08-11T18:00:00+01:00":165,"2022-08-11T19:00:00+01:00":111,"2022-08-11T20:00:00+01:00":51,"2022-08-11T20:43:00+01:00":0}type or paste code here
  • watthours (again same) - 10:00 = 5170, 9:00 = 3617, difference = 1553
{"2022-08-10T05:45:00+01:00":0,"2022-08-10T06:00:00+01:00":80,"2022-08-10T07:00:00+01:00":947,"2022-08-10T08:00:00+01:00":2174,"2022-08-10T09:00:00+01:00":3617,"2022-08-10T10:00:00+01:00":5170,"2022-08-10T11:00:00+01:00":6703,"2022-08-10T12:00:00+01:00":8109,"2022-08-10T13:00:00+01:00":9315,"2022-08-10T14:00:00+01:00":10296,"2022-08-10T15:00:00+01:00":11028,"2022-08-10T16:00:00+01:00":11502,"2022-08-10T17:00:00+01:00":11817,"2022-08-10T18:00:00+01:00":12063,"2022-08-10T19:00:00+01:00":12241,"2022-08-10T20:00:00+01:00":12344,"2022-08-10T20:45:00+01:00":12368,"2022-08-11T05:47:00+01:00":0,"2022-08-11T06:00:00+01:00":64,"2022-08-11T07:00:00+01:00":857,"2022-08-11T08:00:00+01:00":1990,"2022-08-11T09:00:00+01:00":3338,"2022-08-11T10:00:00+01:00":4766,"2022-08-11T11:00:00+01:00":6154,"2022-08-11T12:00:00+01:00":7417,"2022-08-11T13:00:00+01:00":8497,"2022-08-11T14:00:00+01:00":9366,"2022-08-11T15:00:00+01:00":9999,"2022-08-11T16:00:00+01:00":10391,"2022-08-11T17:00:00+01:00":10635,"2022-08-11T18:00:00+01:00":10823,"2022-08-11T19:00:00+01:00":10961,"2022-08-11T20:00:00+01:00":11042,"2022-08-11T20:43:00+01:00":11061}

I can confirm that the West API call is likewise identical to the west integration diagnostic.

Summary:
Watt-hours 9:00 to 10:00 (local time), reported at “2022-08-10T10:00:00+01:00”
East = 1553 (diagnostic matches API)
West = 611 (diagnostic matches API)

Total (for period 09:00 to 10:00) 2164 = 2.16 kWh

And, to finally tie this back to the energy dashboard display, showing for 10:00 to 11:00 (local time). Note that when I took this picture at 10:20, the energy generation period 9:00 to 10:00 was completed and 10:00 to 11:00 was still ‘in progress’ as I would expect.

2.16 kWh, for 10:00 to 11:00 (local time)

I will try and have a look at the code reference you have provided, although I may struggle to understand much of it. I have enough trouble getting all of this data and cross referencing everything to ensure my thinking is correct.

Currently I am trying to set the damping factor to get the forecast better aligned. Although the energy dashboard shows a good match in the morning, correcting the hour shift (which I have done on my own chart from direct API calls) has shown a poor match AM, and I am using damping around 0.5/0.6 to try a better match. This is complicated by the fact that my planes are East and West, with a good match on only part of the day, offset by a corresponding poor match on the converse part of the other plane.
I am getting there, slowly!

Power sampling on each plane, plotted against the current hour forecast from API (no damping)


for an almost perfect sunny day this weekend.

Ok. So I have looked at the code, which I assume to be Python, and I am currently really only (semi) fluent in Javascript. Thus I may be completely wrong here, however…

Whatever the code does, I believe the problems appears to be confusion between plotting at a time, compared to plotting (being) for a period of time.

When a forecast value for a timestamp of 10:00 arrives into the HA Energy Dashboard, it appears to be plotted on the graph at 10:00. The graph is a mixture of bar chart and line. The solar generation bar (a period) plotted at 10:00 is commented as being for the period 10:00 to 11:00. The line, plotted at 10:00, is in association also commented as being for the period 10:00 to 11:00.

May I suggest, that the value at 10:00 should actually be plotted at 09:00 for the period 09:00 to 10:00 which could be easily achieved by keeping the previous timestamp and using

wh_hours[previous_timestamp]= energy-previous_value
previous_timestamp = timestamp

etc.

The only problem here is that the first loop iteration, for sunrise, does not have a previous timestamp, and therefore the first iteration should only really pick up the previous_value and previous_timestamp, with the output object being constructed from the second iteration onwards. Naturally more complex at a reset when the day changes…

My assumption here - do correct me if I am wrong - is that the real issue is that the Energy Dashboard plots the solar forecast value for 10:00-11:00 at 10:00, and therefore the value for the difference between watt-hours at 09:00 and 10:00 should actually be plotted at 09:00, not at 10:00.

???