Can't get forecast.solar forecast to line up with my production

Biscuit · June 27, 2022, 2:59pm

I have been spending some time recently trying to get more out of the integrated solar forecast, and finding the provided entities not quite what I want, I have resorted to fetching and manipulating the data in Node-Red myself. Now I have a complete forecast for the next two days, updates when I want, and an array of ‘power levels’ with start/end times and duration.

In doing all of this I have been working out how the integration is set up to get the figures it does, and I’m finding that my values do not always agree. I also suspect that the displayed forecast chart on the energy page is one hour out and shifted to the right from where it should be. This might explain why my actual generation does not seem to match the forecast.

Here is the logic:
solar.forecast API returns a list of power values (watts) for each hour, effective at the hour, as well as a list of watt-hours for each hour period. All times are local (confirmed since the start/end time matches with my sunrise/sunset) so being on daylight saving in the UK shouldn’t be the issue. For example, as I write this at 15:30, at 18:00 today plane 1 forecast (API values) is 367 watts, and at 19:00 247 watts. The watt-hours figures are available in the API return, but can also be calculated simply by averaging the power over the hour. So, assuming a straight line between 18:00 and 19:00, the average power is 307 watts, and assuming this is for the full hour, equates to 307 watt-hours between 18:00 and 19:00. I have two planes, so doing the same for the second gives me a total 1184 watt-hours for 18:00 to 19:00. Round this to 1.19 kWh as is plotted on the energy page, and there it is, but for 19:00 to 20:00 ! Bingo.

The more I look closely at the forecast line, the more I see that the first and the last values (start/end of day) are out by one hour, so I’m of the thought that the graph should be shifted left by one hour. I wonder if the integration code has been written to add in daylight saving, as in the UK I am currently 1 hour ahead of UTC (BST and not GMT) although solar.forecast returns results based on the local time of the location provided?

The top chart is the standard HA energy page, the bottom chart is something I have just managed to get working based on the API results from solar.forecast (today in the middle, left is ‘history’ and right is tomorrow). It does not show up at all well, but I think my orange forecast line is one hour to the left of the dotted line. Hmmm.

Squishbutt · June 30, 2022, 12:44am

Well that’s super interesting and would actually cause my issue to make sense.

You’ve proven this is a far more intricate and involved topic than I previously thought.

Biscuit · July 2, 2022, 12:37pm

Yes, the more I look at solar forecasting the more involved and challenging it becomes. I sat down this morning and pulled all the figures again to check, and yes the integration forecast graph is being incorrectly plotted one hour to the right and I think this is just a mistake with the coding. I don’t have a GitHub account (I don’t want to go down that route, this stuff is too complicated already) so I can’t report this feature/bug via the approved mechanism, but hereby follows the logic for anyone else who is struggling to get the forecast.solar dotted line to match their generation.

Lovely nice forecast this morning, just the sort of day to test out my graph and code! (screenshot taken at 9:30, the integration had just updated at 9:20)

Notice the very big dip, showing on the forecast graph at 12:00 - 13:00 (plotted where the 12:00 - 13:00 generation bar will later appear). This is for the time period starting at 12:00 and ending at 13:00.

The forecast.solar API call returns a list of watts and a list of watt-hours, both set as values for the hour, local time zone. I have two integrations, so I additionally manually called the API once for each plane. Adding the figures together I get

At 11:00 - 131 watts
At 12:00 - 222 watts

these figures are for power at the hour, so a simple calculation gives an average power of 177, or 177 watt-hours (177 watts for one hour) for the period 11:00 to 12:00

The watt-hour figures from the API are provided as cumulative, being

At 11:00 - 3978 watt-hours
At 12:00 - 4155 watt-hours

which means that the increase is 177 watt-hours during the time period 11:00 to 12:00.

The calculation matches as I expect, so we can work in terms of 131 watts at 11:00, 177 watts at 11:30, 222 watts at 12:00 or 177 watt-hours at 11:30 (for 11:00 to 12:00). To get the forecast for each hour period, the calculation is to subtract the last hour cumulative value from this hour cumulative value. However, and this is the tricky bit, that calculation provides the expected increase for the period starting at the last hour, not this hour. The integration plots 177 (rounded to 2dp at 0.18 kWh) at 12:00 (to 13:00) when it should plot that value at 11:00 (to 12:00).

I rest my case m’lud.

And here is a screenshot at 13:25. Clearly the forecast has changed, which is to be expected. For my graph I am directly calling the forecast.solar API and plotting the watts (the integration uses the watt-hours). I want to be able to forecast power levels for turning appliances on, so I think watts will be of more use. My graph and the integration graph will never quite match therefore, but they are closely related (as the integration graph being based on watt-hours is just a plot of the average for each pair of points on the watt graph).

Worth noting however that the forecast changes, quite regularly, and sometimes quite a lot. Also it continues to change even when it has passed ‘now’ and becomes history. I am saving and plotting the past-hour generation figure from my inverter sensor, and at the same time I am now saving the ‘current’ forecast value for that hour (one hour old). This is the magenta line. I am finding that the forecast seems to revert to a clear-sky plot. Yes the morning part is now old and no longer a forecast, however the forecast watt-hour figures also change, and the integration ‘energy production today’ figure increases making it unreliable once we are moving into the current day.

Not sure if anyone is actually going to read this stuff, but if you do I hope you have found it interesting and useful.

Squishbutt · July 4, 2022, 12:30am

I am so thoroughly impressed with your reply. I think you hit the nail on the head - and I’m surprised it hasn’t come up in the past, this is a fairly popular integration. I think many probably accept the default values and don’t question since the forecast tends to line up nicely - at least for me.

I do have a github account and will report this.

Thanks for the hard work. Hopefully your comment is satisfactory enough to get the dev(s) looking at it.

Linked on github: https://github.com/home-assistant/core/issues/74375

Biscuit · July 4, 2022, 9:08am

Looking at old posts / publicity screenshots I suspect that this is the result of a recent change. Probably worked correctly at some point. And yes strange that no one else appears to have noticed it. I do wonder how many people actually use this feature - given that in practical terms you can’t do very much with it.

I had solar pv installed last October, and I bought an Odroid with HA (Blue) last year, basically because of this very forecast feature and the energy dashboard, to form the basis for a monitoring and control system for my solar pv. Frankly, whilst I like HA very much, I have found the forecast of little practical use, the energy dashboard I have replaced with another community power card, and I do most of my work in Node-Red (which I understand and can use). I can see how powerful HA is when creating one sensor and one associated automation, but to do anything more complex seems to require many inter-connected entities written in yaml, and thus the spiderweb of bits and pieces rapidly becomes a ‘write only language’, impossible to maintain!

Thank you for reporting it. I would happily have done that myself but I am drawing the line at GitHub as I think there has to be a limit to the amount of coding stuff I get involved in. Inline jason and jinja are, frankly, testing my abilities to the limit.

klaasnicolaas · July 4, 2022, 3:48pm

I’ve read the entire thread, but it’s very difficult to figure out where exactly this is going wrong. Is it in the API that provides the data, is it the python package that incorrectly processes the obtained data or in the integration that actually all runs through the list.

firstof9 · July 4, 2022, 3:51pm

I am wondering if it has something to do with DST, may be on the API end.

liquidacid · July 13, 2022, 8:59am

Quite interesting. I think i got this same thing going on for as long as it can be seen in the energy dashboard:

Here you can see the exact thing going on. I more or less accepted it since the data in the end of the day is quite accurate.

Squishbutt · July 13, 2022, 3:46pm

Glad to know it’s happening to everyone - Once I pumped in accurate location data, the forecast became pretty worthless. But I’m a little annoyed with the dev - i opened an issue in GitHub and he said to keep it here, but has gone silent.

klaasnicolaas · July 13, 2022, 4:08pm

It’s not just snapping your fingers and it’s solved … I also do all this in my spare time and I haven’t had it very quiet lately. The problem that I mentioned earlier is that it is still unknown where this problem lies.

Squishbutt · July 13, 2022, 5:02pm

Sorry, I get that. I could have phrased it better.

I appreciate the work you do on this and hope for a fix. Apologies for acting ungrateful.

klaasnicolaas · July 21, 2022, 12:46pm

I need some more information to rule things out. For those who have problems with a shifted production graph, see if you can download a diagnostic file from HA and an export of this directly from the API, if you access the url. In this way we can compare and see if something goes wrong in the data processing.

Biscuit · July 21, 2022, 7:12pm

OK. I guess that will be me then.

Diagnostic EAST plane:

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.7.6",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.45",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.2",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH East",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 73,
        "modules power": 2150,
        "damping": 0.0
      }
    },
    "data": {
      "energy_production_today": 14783,
      "energy_production_tomorrow": 2716,
      "energy_current_hour": 277,
      "power_production_now": 215,
      "watts": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 617,
        "2022-07-21T07:00:00+01:00": 1098,
        "2022-07-21T08:00:00+01:00": 1462,
        "2022-07-21T09:00:00+01:00": 1685,
        "2022-07-21T10:00:00+01:00": 1757,
        "2022-07-21T11:00:00+01:00": 1725,
        "2022-07-21T12:00:00+01:00": 1595,
        "2022-07-21T13:00:00+01:00": 1372,
        "2022-07-21T14:00:00+01:00": 1136,
        "2022-07-21T15:00:00+01:00": 806,
        "2022-07-21T16:00:00+01:00": 523,
        "2022-07-21T17:00:00+01:00": 425,
        "2022-07-21T18:00:00+01:00": 340,
        "2022-07-21T19:00:00+01:00": 215,
        "2022-07-21T20:00:00+01:00": 93,
        "2022-07-21T21:00:00+01:00": 17,
        "2022-07-21T21:17:00+01:00": 0,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 6,
        "2022-07-22T07:00:00+01:00": 35,
        "2022-07-22T08:00:00+01:00": 49,
        "2022-07-22T09:00:00+01:00": 24,
        "2022-07-22T10:00:00+01:00": 27,
        "2022-07-22T11:00:00+01:00": 29,
        "2022-07-22T12:00:00+01:00": 188,
        "2022-07-22T13:00:00+01:00": 323,
        "2022-07-22T14:00:00+01:00": 367,
        "2022-07-22T15:00:00+01:00": 404,
        "2022-07-22T16:00:00+01:00": 323,
        "2022-07-22T17:00:00+01:00": 343,
        "2022-07-22T18:00:00+01:00": 277,
        "2022-07-22T19:00:00+01:00": 207,
        "2022-07-22T20:00:00+01:00": 102,
        "2022-07-22T21:00:00+01:00": 20,
        "2022-07-22T21:15:00+01:00": 0
      },
      "wh_days": {
        "2022-07-21T00:00:00": 14783,
        "2022-07-22T00:00:00": 2716
      },
      "wh_hours": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 231,
        "2022-07-21T07:00:00+01:00": 858,
        "2022-07-21T08:00:00+01:00": 1280,
        "2022-07-21T09:00:00+01:00": 1573,
        "2022-07-21T10:00:00+01:00": 1721,
        "2022-07-21T11:00:00+01:00": 1741,
        "2022-07-21T12:00:00+01:00": 1660,
        "2022-07-21T13:00:00+01:00": 1484,
        "2022-07-21T14:00:00+01:00": 1254,
        "2022-07-21T15:00:00+01:00": 971,
        "2022-07-21T16:00:00+01:00": 664,
        "2022-07-21T17:00:00+01:00": 474,
        "2022-07-21T18:00:00+01:00": 383,
        "2022-07-21T19:00:00+01:00": 277,
        "2022-07-21T20:00:00+01:00": 154,
        "2022-07-21T21:00:00+01:00": 55,
        "2022-07-21T21:17:00+01:00": 3,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 2,
        "2022-07-22T07:00:00+01:00": 21,
        "2022-07-22T08:00:00+01:00": 42,
        "2022-07-22T09:00:00+01:00": 36,
        "2022-07-22T10:00:00+01:00": 26,
        "2022-07-22T11:00:00+01:00": 28,
        "2022-07-22T12:00:00+01:00": 108,
        "2022-07-22T13:00:00+01:00": 256,
        "2022-07-22T14:00:00+01:00": 345,
        "2022-07-22T15:00:00+01:00": 385,
        "2022-07-22T16:00:00+01:00": 364,
        "2022-07-22T17:00:00+01:00": 333,
        "2022-07-22T18:00:00+01:00": 310,
        "2022-07-22T19:00:00+01:00": 242,
        "2022-07-22T20:00:00+01:00": 154,
        "2022-07-22T21:00:00+01:00": 61,
        "2022-07-22T21:15:00+01:00": 3
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

API direct call

{
   "watts":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":617,
      "2022-07-21 07:00:00":1098,
      "2022-07-21 08:00:00":1462,
      "2022-07-21 09:00:00":1688,
      "2022-07-21 10:00:00":1769,
      "2022-07-21 11:00:00":1735,
      "2022-07-21 12:00:00":1601,
      "2022-07-21 13:00:00":1370,
      "2022-07-21 14:00:00":1135,
      "2022-07-21 15:00:00":834,
      "2022-07-21 16:00:00":552,
      "2022-07-21 17:00:00":424,
      "2022-07-21 18:00:00":339,
      "2022-07-21 19:00:00":225,
      "2022-07-21 20:00:00":92,
      "2022-07-21 21:00:00":16,
      "2022-07-21 21:17:00":0,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":38,
      "2022-07-22 07:00:00":51,
      "2022-07-22 08:00:00":19,
      "2022-07-22 09:00:00":115,
      "2022-07-22 10:00:00":222,
      "2022-07-22 11:00:00":185,
      "2022-07-22 12:00:00":324,
      "2022-07-22 13:00:00":331,
      "2022-07-22 14:00:00":392,
      "2022-07-22 15:00:00":422,
      "2022-07-22 16:00:00":385,
      "2022-07-22 17:00:00":298,
      "2022-07-22 18:00:00":274,
      "2022-07-22 19:00:00":200,
      "2022-07-22 20:00:00":101,
      "2022-07-22 21:00:00":20,
      "2022-07-22 21:15:00":0
   },
   "watt_hours":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":231,
      "2022-07-21 07:00:00":1089,
      "2022-07-21 08:00:00":2369,
      "2022-07-21 09:00:00":3944,
      "2022-07-21 10:00:00":5672,
      "2022-07-21 11:00:00":7424,
      "2022-07-21 12:00:00":9092,
      "2022-07-21 13:00:00":10578,
      "2022-07-21 14:00:00":11830,
      "2022-07-21 15:00:00":12815,
      "2022-07-21 16:00:00":13508,
      "2022-07-21 17:00:00":13996,
      "2022-07-21 18:00:00":14377,
      "2022-07-21 19:00:00":14659,
      "2022-07-21 20:00:00":14818,
      "2022-07-21 21:00:00":14872,
      "2022-07-21 21:17:00":14874,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":14,
      "2022-07-22 07:00:00":58,
      "2022-07-22 08:00:00":93,
      "2022-07-22 09:00:00":160,
      "2022-07-22 10:00:00":329,
      "2022-07-22 11:00:00":532,
      "2022-07-22 12:00:00":787,
      "2022-07-22 13:00:00":1114,
      "2022-07-22 14:00:00":1476,
      "2022-07-22 15:00:00":1883,
      "2022-07-22 16:00:00":2286,
      "2022-07-22 17:00:00":2628,
      "2022-07-22 18:00:00":2914,
      "2022-07-22 19:00:00":3151,
      "2022-07-22 20:00:00":3301,
      "2022-07-22 21:00:00":3362,
      "2022-07-22 21:15:00":3364
   },
   "watt_hours_day":{
      "2022-07-21":14874,
      "2022-07-22":3364
   }
}

Diagnostic WEST plane

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.7.6",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.45",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.2",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH West",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 253,
        "modules power": 2150,
        "damping": 0.0
      }
    },
    "data": {
      "energy_production_today": 15465,
      "energy_production_tomorrow": 5601,
      "energy_current_hour": 780,
      "power_production_now": 565,
      "watts": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 127,
        "2022-07-21T07:00:00+01:00": 237,
        "2022-07-21T08:00:00+01:00": 343,
        "2022-07-21T09:00:00+01:00": 554,
        "2022-07-21T10:00:00+01:00": 917,
        "2022-07-21T11:00:00+01:00": 1298,
        "2022-07-21T12:00:00+01:00": 1620,
        "2022-07-21T13:00:00+01:00": 1823,
        "2022-07-21T14:00:00+01:00": 1990,
        "2022-07-21T15:00:00+01:00": 1846,
        "2022-07-21T16:00:00+01:00": 1576,
        "2022-07-21T17:00:00+01:00": 1335,
        "2022-07-21T18:00:00+01:00": 995,
        "2022-07-21T19:00:00+01:00": 565,
        "2022-07-21T20:00:00+01:00": 222,
        "2022-07-21T21:00:00+01:00": 51,
        "2022-07-21T21:17:00+01:00": 0,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 2,
        "2022-07-22T07:00:00+01:00": 12,
        "2022-07-22T08:00:00+01:00": 19,
        "2022-07-22T09:00:00+01:00": 14,
        "2022-07-22T10:00:00+01:00": 19,
        "2022-07-22T11:00:00+01:00": 24,
        "2022-07-22T12:00:00+01:00": 189,
        "2022-07-22T13:00:00+01:00": 412,
        "2022-07-22T14:00:00+01:00": 590,
        "2022-07-22T15:00:00+01:00": 849,
        "2022-07-22T16:00:00+01:00": 882,
        "2022-07-22T17:00:00+01:00": 1002,
        "2022-07-22T18:00:00+01:00": 764,
        "2022-07-22T19:00:00+01:00": 539,
        "2022-07-22T20:00:00+01:00": 247,
        "2022-07-22T21:00:00+01:00": 60,
        "2022-07-22T21:15:00+01:00": 0
      },
      "wh_days": {
        "2022-07-21T00:00:00": 15465,
        "2022-07-22T00:00:00": 5601
      },
      "wh_hours": {
        "2022-07-21T05:15:00+01:00": 0,
        "2022-07-21T06:00:00+01:00": 48,
        "2022-07-21T07:00:00+01:00": 182,
        "2022-07-21T08:00:00+01:00": 290,
        "2022-07-21T09:00:00+01:00": 448,
        "2022-07-21T10:00:00+01:00": 736,
        "2022-07-21T11:00:00+01:00": 1107,
        "2022-07-21T12:00:00+01:00": 1459,
        "2022-07-21T13:00:00+01:00": 1722,
        "2022-07-21T14:00:00+01:00": 1906,
        "2022-07-21T15:00:00+01:00": 1918,
        "2022-07-21T16:00:00+01:00": 1711,
        "2022-07-21T17:00:00+01:00": 1456,
        "2022-07-21T18:00:00+01:00": 1165,
        "2022-07-21T19:00:00+01:00": 780,
        "2022-07-21T20:00:00+01:00": 393,
        "2022-07-21T21:00:00+01:00": 137,
        "2022-07-21T21:17:00+01:00": 7,
        "2022-07-22T05:17:00+01:00": 0,
        "2022-07-22T06:00:00+01:00": 1,
        "2022-07-22T07:00:00+01:00": 7,
        "2022-07-22T08:00:00+01:00": 15,
        "2022-07-22T09:00:00+01:00": 17,
        "2022-07-22T10:00:00+01:00": 16,
        "2022-07-22T11:00:00+01:00": 22,
        "2022-07-22T12:00:00+01:00": 106,
        "2022-07-22T13:00:00+01:00": 301,
        "2022-07-22T14:00:00+01:00": 501,
        "2022-07-22T15:00:00+01:00": 719,
        "2022-07-22T16:00:00+01:00": 866,
        "2022-07-22T17:00:00+01:00": 942,
        "2022-07-22T18:00:00+01:00": 883,
        "2022-07-22T19:00:00+01:00": 651,
        "2022-07-22T20:00:00+01:00": 393,
        "2022-07-22T21:00:00+01:00": 154,
        "2022-07-22T21:15:00+01:00": 7
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

API direct call (West)

{
   "watts":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":127,
      "2022-07-21 07:00:00":237,
      "2022-07-21 08:00:00":343,
      "2022-07-21 09:00:00":553,
      "2022-07-21 10:00:00":916,
      "2022-07-21 11:00:00":1301,
      "2022-07-21 12:00:00":1626,
      "2022-07-21 13:00:00":1818,
      "2022-07-21 14:00:00":1998,
      "2022-07-21 15:00:00":1934,
      "2022-07-21 16:00:00":1681,
      "2022-07-21 17:00:00":1331,
      "2022-07-21 18:00:00":995,
      "2022-07-21 19:00:00":603,
      "2022-07-21 20:00:00":220,
      "2022-07-21 21:00:00":49,
      "2022-07-21 21:17:00":0,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":10,
      "2022-07-22 07:00:00":16,
      "2022-07-22 08:00:00":10,
      "2022-07-22 09:00:00":49,
      "2022-07-22 10:00:00":128,
      "2022-07-22 11:00:00":145,
      "2022-07-22 12:00:00":326,
      "2022-07-22 13:00:00":422,
      "2022-07-22 14:00:00":633,
      "2022-07-22 15:00:00":887,
      "2022-07-22 16:00:00":1066,
      "2022-07-22 17:00:00":856,
      "2022-07-22 18:00:00":753,
      "2022-07-22 19:00:00":516,
      "2022-07-22 20:00:00":245,
      "2022-07-22 21:00:00":60,
      "2022-07-22 21:15:00":0
   },
   "watt_hours":{
      "2022-07-21 05:15:00":0,
      "2022-07-21 06:00:00":48,
      "2022-07-21 07:00:00":230,
      "2022-07-21 08:00:00":520,
      "2022-07-21 09:00:00":968,
      "2022-07-21 10:00:00":1702,
      "2022-07-21 11:00:00":2811,
      "2022-07-21 12:00:00":4274,
      "2022-07-21 13:00:00":5996,
      "2022-07-21 14:00:00":7904,
      "2022-07-21 15:00:00":9870,
      "2022-07-21 16:00:00":11678,
      "2022-07-21 17:00:00":13184,
      "2022-07-21 18:00:00":14347,
      "2022-07-21 19:00:00":15146,
      "2022-07-21 20:00:00":15557,
      "2022-07-21 21:00:00":15692,
      "2022-07-21 21:17:00":15699,
      "2022-07-22 05:17:00":0,
      "2022-07-22 06:00:00":4,
      "2022-07-22 07:00:00":17,
      "2022-07-22 08:00:00":30,
      "2022-07-22 09:00:00":59,
      "2022-07-22 10:00:00":148,
      "2022-07-22 11:00:00":284,
      "2022-07-22 12:00:00":520,
      "2022-07-22 13:00:00":894,
      "2022-07-22 14:00:00":1421,
      "2022-07-22 15:00:00":2181,
      "2022-07-22 16:00:00":3158,
      "2022-07-22 17:00:00":4119,
      "2022-07-22 18:00:00":4923,
      "2022-07-22 19:00:00":5558,
      "2022-07-22 20:00:00":5938,
      "2022-07-22 21:00:00":6091,
      "2022-07-22 21:15:00":6098
   },
   "watt_hours_day":{
      "2022-07-21":15699,
      "2022-07-22":6098
   }
}

Current energy graph (screen shot)

Calculations:

API calls and diagnostics appear to align (within slight tolerance - both updated at around the same time)

Watts - East and West. Sunrise is correct at 5:15
5:15 - 0 & 0 (sunrise)
6:00 - 617 & 127
7:00 -1098 & 237
8:00 - 1462 & 343

I assume that this is the POWER at the hour.

The watt-hour will be the average of the two powers at the hours
ie for 6:00 to 7:00 = (617+1098)/2 = 858 (east) and 182 (west)

Compare to the API/diagnostic…

Watt-hours (for East & West)
API call (cumulative watt-hours)
6:00 = 231 / 48
7:00 = 1089 / 230

Diagnostic (which shows the difference not the cumulative)
7:00 = 858 / 182
which is indeed 1089-231 and 230-48

So for the hour 6:00 to 7:00 the watt-hours are clearly 858 and 182 (ie at 5:15 nothing, at 6:00 231, at 7:00 1089, increase of 858 during that hour)

The energy graph, being the sum of the two, is showing for 7:00 to 8:00 as 1.04 kWh or 1040
858+182 = 1040

Therefore the calculations are correct. API agrees with diagnostic agrees with watts to watt-hour calculation agrees with cumulative to difference.

Here is the bit that I have the problem with.

If the cumulative energy (east plane only) at 7:00 is 1089, and at 6:00 231, then the difference 858 applies between 6:00 and 7:00, but this is being shown on the graph for 7:00 to 8:00

Or have I completely lost the plot here?

Biscuit · July 21, 2022, 7:30pm

The short version

Cumulative (west plane only) watt-hours from API call
at 5:15 - 0
at 6:00 - 48
at 7:00 - 230
at 8:00 - 520
at 9:00 - 968

Difference by calculation
between 5:15 - 6:00 48
between 6:00 - 7:00 182 =230-48
between 7:00 - 8:00 290

Diagnostic
at 5:15 - 0
at 6:00 - 48
at 7:00 - 182
at 8:00 - 290

Graph
at 7:00 (for 7:00 to 8:00) - 182

klaasnicolaas · August 9, 2022, 11:47pm

I have tried to interpret your information as well as possible and now I understand better where the problem could lie. For an even better comparison could you make the direct API call with a
?time=iso8601 parameter? Because the diagnostic data shows a UTC offset and that of the direct API call does not (by default the api returns data that applies to your own timezone).

The dict that Home Assistant uses with the differences are created here in the for loop.

Just to rule out one more thing, is your Home Assistant timezone set to Europe/London?

Biscuit · August 10, 2022, 10:31am

My HA device is set to GMT+00:00, London in settings.
We are currently on BST which is one hour ahead for DST. I have no idea how HA knows we are on DST, but all the timestamps internally are UTC, yet for display everything is in my current local time so it all seems to work correctly.

All data collected this morning between 10:00 and 11:00 BST (GMT+1)

East - HA diagnostic dump:

timezone is Europe/London
sunrise 05:45 (correct for current local time GMT+1 British Summer Time)
watts at 10:00 = 1574, 9:00 = 1531
watthours 10:00 = average = 1553 (agrees)

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.8.3",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.55",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.4",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH East",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 73,
        "modules power": 2190,
        "damping": 0.0,
        "inverter_size": 4600
      }
    },
    "data": {
      "energy_production_today": 12368,
      "energy_production_tomorrow": 11061,
      "energy_current_hour": 1553,
      "power_production_now": 1574,
      "watts": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 636,
        "2022-08-10T07:00:00+01:00": 1099,
        "2022-08-10T08:00:00+01:00": 1355,
        "2022-08-10T09:00:00+01:00": 1531,
        "2022-08-10T10:00:00+01:00": 1574,
        "2022-08-10T11:00:00+01:00": 1493,
        "2022-08-10T12:00:00+01:00": 1319,
        "2022-08-10T13:00:00+01:00": 1092,
        "2022-08-10T14:00:00+01:00": 870,
        "2022-08-10T15:00:00+01:00": 595,
        "2022-08-10T16:00:00+01:00": 353,
        "2022-08-10T17:00:00+01:00": 277,
        "2022-08-10T18:00:00+01:00": 215,
        "2022-08-10T19:00:00+01:00": 141,
        "2022-08-10T20:00:00+01:00": 65,
        "2022-08-10T20:45:00+01:00": 0,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 593,
        "2022-08-11T07:00:00+01:00": 993,
        "2022-08-11T08:00:00+01:00": 1273,
        "2022-08-11T09:00:00+01:00": 1423,
        "2022-08-11T10:00:00+01:00": 1433,
        "2022-08-11T11:00:00+01:00": 1343,
        "2022-08-11T12:00:00+01:00": 1182,
        "2022-08-11T13:00:00+01:00": 979,
        "2022-08-11T14:00:00+01:00": 759,
        "2022-08-11T15:00:00+01:00": 507,
        "2022-08-11T16:00:00+01:00": 277,
        "2022-08-11T17:00:00+01:00": 211,
        "2022-08-11T18:00:00+01:00": 165,
        "2022-08-11T19:00:00+01:00": 111,
        "2022-08-11T20:00:00+01:00": 51,
        "2022-08-11T20:43:00+01:00": 0
      },
      "wh_days": {
        "2022-08-10T00:00:00": 12368,
        "2022-08-11T00:00:00": 11061
      },
      "wh_hours": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 80,
        "2022-08-10T07:00:00+01:00": 867,
        "2022-08-10T08:00:00+01:00": 1227,
        "2022-08-10T09:00:00+01:00": 1443,
        "2022-08-10T10:00:00+01:00": 1553,
        "2022-08-10T11:00:00+01:00": 1533,
        "2022-08-10T12:00:00+01:00": 1406,
        "2022-08-10T13:00:00+01:00": 1206,
        "2022-08-10T14:00:00+01:00": 981,
        "2022-08-10T15:00:00+01:00": 732,
        "2022-08-10T16:00:00+01:00": 474,
        "2022-08-10T17:00:00+01:00": 315,
        "2022-08-10T18:00:00+01:00": 246,
        "2022-08-10T19:00:00+01:00": 178,
        "2022-08-10T20:00:00+01:00": 103,
        "2022-08-10T20:45:00+01:00": 24,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 64,
        "2022-08-11T07:00:00+01:00": 793,
        "2022-08-11T08:00:00+01:00": 1133,
        "2022-08-11T09:00:00+01:00": 1348,
        "2022-08-11T10:00:00+01:00": 1428,
        "2022-08-11T11:00:00+01:00": 1388,
        "2022-08-11T12:00:00+01:00": 1263,
        "2022-08-11T13:00:00+01:00": 1080,
        "2022-08-11T14:00:00+01:00": 869,
        "2022-08-11T15:00:00+01:00": 633,
        "2022-08-11T16:00:00+01:00": 392,
        "2022-08-11T17:00:00+01:00": 244,
        "2022-08-11T18:00:00+01:00": 188,
        "2022-08-11T19:00:00+01:00": 138,
        "2022-08-11T20:00:00+01:00": 81,
        "2022-08-11T20:43:00+01:00": 19
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

West diagnostic dump

watts 10:00 = 770 , 9:00 = 451
watthours 10:00 = average = 611 (agrees)

{
  "home_assistant": {
    "installation_type": "Home Assistant OS",
    "version": "2022.8.3",
    "dev": false,
    "hassio": true,
    "virtualenv": false,
    "python_version": "3.10.5",
    "docker": true,
    "arch": "aarch64",
    "timezone": "Europe/London",
    "os_name": "Linux",
    "os_version": "5.15.55",
    "supervisor": "2022.07.0",
    "host_os": "Home Assistant OS 8.4",
    "docker_version": "20.10.14",
    "chassis": "embedded",
    "run_as_root": true
  },
  "custom_components": {
    "nodered": {
      "version": "1.0.4",
      "requirements": []
    },
    "octopus_energy": {
      "version": "4.1.3",
      "requirements": []
    }
  },
  "integration_manifest": {
    "domain": "forecast_solar",
    "name": "Forecast.Solar",
    "config_flow": true,
    "documentation": "https://www.home-assistant.io/integrations/forecast_solar",
    "requirements": [
      "forecast_solar==2.2.0"
    ],
    "codeowners": [
      "@klaasnicolaas",
      "@frenck"
    ],
    "quality_scale": "platinum",
    "iot_class": "cloud_polling",
    "is_built_in": true
  },
  "data": {
    "entry": {
      "title": "WH West",
      "data": {
        "latitude": "**REDACTED**",
        "longitude": "**REDACTED**"
      },
      "options": {
        "declination": 35,
        "azimuth": 253,
        "modules power": 2190,
        "damping": 0.0,
        "inverter_size": 4600
      }
    },
    "data": {
      "energy_production_today": 13785,
      "energy_production_tomorrow": 12285,
      "energy_current_hour": 611,
      "power_production_now": 770,
      "watts": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 131,
        "2022-08-10T07:00:00+01:00": 227,
        "2022-08-10T08:00:00+01:00": 271,
        "2022-08-10T09:00:00+01:00": 451,
        "2022-08-10T10:00:00+01:00": 770,
        "2022-08-10T11:00:00+01:00": 1091,
        "2022-08-10T12:00:00+01:00": 1352,
        "2022-08-10T13:00:00+01:00": 1541,
        "2022-08-10T14:00:00+01:00": 1700,
        "2022-08-10T15:00:00+01:00": 1691,
        "2022-08-10T16:00:00+01:00": 1569,
        "2022-08-10T17:00:00+01:00": 1325,
        "2022-08-10T18:00:00+01:00": 977,
        "2022-08-10T19:00:00+01:00": 560,
        "2022-08-10T20:00:00+01:00": 203,
        "2022-08-10T20:45:00+01:00": 0,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 107,
        "2022-08-11T07:00:00+01:00": 166,
        "2022-08-11T08:00:00+01:00": 213,
        "2022-08-11T09:00:00+01:00": 378,
        "2022-08-11T10:00:00+01:00": 666,
        "2022-08-11T11:00:00+01:00": 963,
        "2022-08-11T12:00:00+01:00": 1217,
        "2022-08-11T13:00:00+01:00": 1411,
        "2022-08-11T14:00:00+01:00": 1536,
        "2022-08-11T15:00:00+01:00": 1534,
        "2022-08-11T16:00:00+01:00": 1413,
        "2022-08-11T17:00:00+01:00": 1186,
        "2022-08-11T18:00:00+01:00": 878,
        "2022-08-11T19:00:00+01:00": 509,
        "2022-08-11T20:00:00+01:00": 175,
        "2022-08-11T20:43:00+01:00": 0
      },
      "wh_days": {
        "2022-08-10T00:00:00": 13785,
        "2022-08-11T00:00:00": 12285
      },
      "wh_hours": {
        "2022-08-10T05:45:00+01:00": 0,
        "2022-08-10T06:00:00+01:00": 16,
        "2022-08-10T07:00:00+01:00": 179,
        "2022-08-10T08:00:00+01:00": 249,
        "2022-08-10T09:00:00+01:00": 361,
        "2022-08-10T10:00:00+01:00": 611,
        "2022-08-10T11:00:00+01:00": 930,
        "2022-08-10T12:00:00+01:00": 1222,
        "2022-08-10T13:00:00+01:00": 1446,
        "2022-08-10T14:00:00+01:00": 1621,
        "2022-08-10T15:00:00+01:00": 1695,
        "2022-08-10T16:00:00+01:00": 1630,
        "2022-08-10T17:00:00+01:00": 1447,
        "2022-08-10T18:00:00+01:00": 1151,
        "2022-08-10T19:00:00+01:00": 769,
        "2022-08-10T20:00:00+01:00": 381,
        "2022-08-10T20:45:00+01:00": 77,
        "2022-08-11T05:47:00+01:00": 0,
        "2022-08-11T06:00:00+01:00": 12,
        "2022-08-11T07:00:00+01:00": 136,
        "2022-08-11T08:00:00+01:00": 190,
        "2022-08-11T09:00:00+01:00": 295,
        "2022-08-11T10:00:00+01:00": 522,
        "2022-08-11T11:00:00+01:00": 815,
        "2022-08-11T12:00:00+01:00": 1090,
        "2022-08-11T13:00:00+01:00": 1314,
        "2022-08-11T14:00:00+01:00": 1473,
        "2022-08-11T15:00:00+01:00": 1535,
        "2022-08-11T16:00:00+01:00": 1474,
        "2022-08-11T17:00:00+01:00": 1299,
        "2022-08-11T18:00:00+01:00": 1032,
        "2022-08-11T19:00:00+01:00": 694,
        "2022-08-11T20:00:00+01:00": 342,
        "2022-08-11T20:43:00+01:00": 62
      }
    },
    "account": {
      "type": "public",
      "rate_limit": 12,
      "timezone": "Europe/London"
    }
  }
}

Direct API call (with same settings, parameter ?time=iso8601)

East:

watts (same values as diagnostic 10:00 = 1574, 9:00 = 1531)

{"2022-08-10T05:45:00+01:00":0,"2022-08-10T06:00:00+01:00":636,"2022-08-10T07:00:00+01:00":1099,"2022-08-10T08:00:00+01:00":1355,"2022-08-10T09:00:00+01:00":1531,"2022-08-10T10:00:00+01:00":1574,"2022-08-10T11:00:00+01:00":1493,"2022-08-10T12:00:00+01:00":1319,"2022-08-10T13:00:00+01:00":1092,"2022-08-10T14:00:00+01:00":870,"2022-08-10T15:00:00+01:00":595,"2022-08-10T16:00:00+01:00":353,"2022-08-10T17:00:00+01:00":277,"2022-08-10T18:00:00+01:00":215,"2022-08-10T19:00:00+01:00":141,"2022-08-10T20:00:00+01:00":65,"2022-08-10T20:45:00+01:00":0,"2022-08-11T05:47:00+01:00":0,"2022-08-11T06:00:00+01:00":593,"2022-08-11T07:00:00+01:00":993,"2022-08-11T08:00:00+01:00":1273,"2022-08-11T09:00:00+01:00":1423,"2022-08-11T10:00:00+01:00":1433,"2022-08-11T11:00:00+01:00":1343,"2022-08-11T12:00:00+01:00":1182,"2022-08-11T13:00:00+01:00":979,"2022-08-11T14:00:00+01:00":759,"2022-08-11T15:00:00+01:00":507,"2022-08-11T16:00:00+01:00":277,"2022-08-11T17:00:00+01:00":211,"2022-08-11T18:00:00+01:00":165,"2022-08-11T19:00:00+01:00":111,"2022-08-11T20:00:00+01:00":51,"2022-08-11T20:43:00+01:00":0}type or paste code here

watthours (again same) - 10:00 = 5170, 9:00 = 3617, difference = 1553

{"2022-08-10T05:45:00+01:00":0,"2022-08-10T06:00:00+01:00":80,"2022-08-10T07:00:00+01:00":947,"2022-08-10T08:00:00+01:00":2174,"2022-08-10T09:00:00+01:00":3617,"2022-08-10T10:00:00+01:00":5170,"2022-08-10T11:00:00+01:00":6703,"2022-08-10T12:00:00+01:00":8109,"2022-08-10T13:00:00+01:00":9315,"2022-08-10T14:00:00+01:00":10296,"2022-08-10T15:00:00+01:00":11028,"2022-08-10T16:00:00+01:00":11502,"2022-08-10T17:00:00+01:00":11817,"2022-08-10T18:00:00+01:00":12063,"2022-08-10T19:00:00+01:00":12241,"2022-08-10T20:00:00+01:00":12344,"2022-08-10T20:45:00+01:00":12368,"2022-08-11T05:47:00+01:00":0,"2022-08-11T06:00:00+01:00":64,"2022-08-11T07:00:00+01:00":857,"2022-08-11T08:00:00+01:00":1990,"2022-08-11T09:00:00+01:00":3338,"2022-08-11T10:00:00+01:00":4766,"2022-08-11T11:00:00+01:00":6154,"2022-08-11T12:00:00+01:00":7417,"2022-08-11T13:00:00+01:00":8497,"2022-08-11T14:00:00+01:00":9366,"2022-08-11T15:00:00+01:00":9999,"2022-08-11T16:00:00+01:00":10391,"2022-08-11T17:00:00+01:00":10635,"2022-08-11T18:00:00+01:00":10823,"2022-08-11T19:00:00+01:00":10961,"2022-08-11T20:00:00+01:00":11042,"2022-08-11T20:43:00+01:00":11061}

I can confirm that the West API call is likewise identical to the west integration diagnostic.

Summary:
Watt-hours 9:00 to 10:00 (local time), reported at “2022-08-10T10:00:00+01:00”
East = 1553 (diagnostic matches API)
West = 611 (diagnostic matches API)

Total (for period 09:00 to 10:00) 2164 = 2.16 kWh

And, to finally tie this back to the energy dashboard display, showing for 10:00 to 11:00 (local time). Note that when I took this picture at 10:20, the energy generation period 9:00 to 10:00 was completed and 10:00 to 11:00 was still ‘in progress’ as I would expect.

2.16 kWh, for 10:00 to 11:00 (local time)

I will try and have a look at the code reference you have provided, although I may struggle to understand much of it. I have enough trouble getting all of this data and cross referencing everything to ensure my thinking is correct.

Currently I am trying to set the damping factor to get the forecast better aligned. Although the energy dashboard shows a good match in the morning, correcting the hour shift (which I have done on my own chart from direct API calls) has shown a poor match AM, and I am using damping around 0.5/0.6 to try a better match. This is complicated by the fact that my planes are East and West, with a good match on only part of the day, offset by a corresponding poor match on the converse part of the other plane.
I am getting there, slowly!

Power sampling on each plane, plotted against the current hour forecast from API (no damping)

for an almost perfect sunny day this weekend.

Biscuit · August 10, 2022, 4:20pm

Ok. So I have looked at the code, which I assume to be Python, and I am currently really only (semi) fluent in Javascript. Thus I may be completely wrong here, however…

Whatever the code does, I believe the problems appears to be confusion between plotting at a time, compared to plotting (being) for a period of time.

When a forecast value for a timestamp of 10:00 arrives into the HA Energy Dashboard, it appears to be plotted on the graph at 10:00. The graph is a mixture of bar chart and line. The solar generation bar (a period) plotted at 10:00 is commented as being for the period 10:00 to 11:00. The line, plotted at 10:00, is in association also commented as being for the period 10:00 to 11:00.

May I suggest, that the value at 10:00 should actually be plotted at 09:00 for the period 09:00 to 10:00 which could be easily achieved by keeping the previous timestamp and using

wh_hours[previous_timestamp]= energy-previous_value
previous_timestamp = timestamp

etc.

The only problem here is that the first loop iteration, for sunrise, does not have a previous timestamp, and therefore the first iteration should only really pick up the previous_value and previous_timestamp, with the output object being constructed from the second iteration onwards. Naturally more complex at a reset when the day changes…

My assumption here - do correct me if I am wrong - is that the real issue is that the Energy Dashboard plots the solar forecast value for 10:00-11:00 at 10:00, and therefore the value for the difference between watt-hours at 09:00 and 10:00 should actually be plotted at 09:00, not at 10:00.

???

incaseoftrouble · August 22, 2022, 9:24pm

Skimming through the thread, this could also be related to my issue #72181: forecast solar updates the reported value only every hour (or 30 mins with an API key). However, if you would load the integration at, say, 14:22, then the “now” for 14:00-15:00 is reported until 15:22. This could also lead to shifts (at least this is what I am expecting with influxdb)

Biscuit · August 23, 2022, 9:34am

Not related at all.

The forecast.solar integration calls the forecast API once (per integration) per hour, every hour, based on a ‘random’ (but predictable) time during each hour. The call is then processed by code and re-plotted as a line for the entire day, not just the past hour. (The solar generation bar chart is updated and plotted for the past hour each hour after the end of the hour - based on an internal utility meter that resets each hour - quite different).

Each forecast hourly API call re-processes the entire day forecast, using the returned cumulative watt-hour figures, by subtracting each hour-figure from the next hour-figure to get an absolute forecast watt-hour value for the hour period between the two figures. At this point the entire forecast line is re-plotted.

As well as updating the forecast line, the integration picks out a few watt power and watt-hour energy values from the API return, presenting a selection including ‘this hour’ and ‘next hour’ forecast.

Your concern is that the update is called once per hour but this can be at any point up to the 59th minute, although that has little immediate impact on the forecast line. The forecast does indeed change during the day, hence the line will move, even the ‘old’ part of the line. If this moves at 14:01 or 14:59, it is still an update every hour.

My concern is that the figure for the period 14:00 to 15:00 is plotted at 15:00 (which is 15:00 to 16:00 on the bar chart) and I contest that it should be plotted at 14:00. I have been through the code -

and I believe that the iteration loop should be something along the lines of

previous_value = 0
previous_timestamp = 0
wh_hours = {}

for timestamp, energy in data["result"]["watt_hours"].items():
    timestamp = datetime.fromisoformat(timestamp)

    # If we get a reset and when moving to the next day
    if energy < previous_value:
        previous_value = 0

    # At start of day (sunrise) energy is zero, minutes are not, period has only started
    if energy = 0:
        # previous_value = 0
        timestamp = timestamp.replace(minute=0)
    # otherwise period has ended so capture energy difference and post to start of period
    else:
        wh_hours[previous_timestamp] = energy - previous_value

    previous_value = energy
    previous_timestamp = timestamp

I am asking for the iteration code that calculates the array of the hourly watt-hour energy forecasts for each hour period to assign each value to the correct hour (ie at the start of the hour period not the end of the hour period) so that the array is correctly plotted as a line on the energy graph in agreement with the convention of the consumption bar chart plot.

What are you really asking for?

For an update more than once per hour?
For each hourly update to take place at the start of each hour?

What I hope that you will see is that this is a free service (for most of us) and therefore we have to take it as it comes. If everyone decided that they wanted their forecast update at 14:01 on the dot, or several times during the hour (to no effective additional value) the solar forecast server would probably not cope. By making the integration use the ‘random’ trigger time that is randomly distributed across the hour, the demand from HA users (for the free integration API calls) is evenly spread, and limited to less than the 12 free calls per hour so generously provided.

You have the solution to your issue in your own hands - you already know how to modify the trigger point for the update call yourself.

Alternatively, assuming that you are lifting the provided this hour forecast data, then I assume that your issues is really that, when you go for this data, if the update has not taken place yet then this hour is actually the last hour and not this hour. However, by checking the time of the update, if the update hour is the same as this hour, then the this hour value is the value for this hour and not the last hour. If the update hour is not the same as this hour, then this hour is the value for the last hour, but the next hour holds the value for the next hour from the last hour which is this hour, so you could use the next hour value for this hour value up to the point where the update takes place. If you also want the next hour during the time when the update hour for this hour is not this hour but the last hour, then the value in the next hour is this hour, so you will have to read the history for the six hour forecast from four hours ago which will give you the value of the hour ahead of the next hour but before it was updated so it will be the value for the next hour. All really very simple.

I have spent many hours on this in great depth, now probably into three figures, and my final solution - I have created my own Node-RED flow to pull the data and process it myself, so I can control the time at which I make the API calls and manage the data as I wish. However I am very mindful that I should treat this free service with some respect and not overload it. I hope you agree.

Biscuit · August 23, 2022, 10:05am

As you raised this issue in the first place, I thought you might appreciate an update on my attempts to get the solar forecast to be realistic against my production.

The good news is that I now have a good match. This is achieved using my own code, calling and processing the forecast.solar API myself.

I have a nice graph showing the upcoming forecast for today and tomorrow, as well as a history record for yesterday, and a record for actual production (all carefully worked out to plot at the correct hour for the correct period).

Here is this morning - you will notice that the forecast (orange line) is different to the history (magenta line). As I capture the forecast for the last hour into a history line, I can see where the forecast moves after the event. Today, the actual matches history at the start, but the forecast has moved and now does not - hence we cannot rely on looking at the forecast line at the end of the day as it will not be the forecast as it was!

solar forecast plot

Getting the match was a question of shuffling the damping factor around. I have power, slope, azimuth all correct, but my situation is complicated by having two planes - east and west. Damping has a symmetrical effect for a south facing production curve, equally both morning and afternoon. My planes produce a shifted generation curve, and damping over-corrects and under-corrects at opposite ends of the day, so I had to dive into the individual planes rather than looking at the summation.

Here is a plot for 14th August (from HA history) which was a nice sunny day. I capture the power values for each plane every 20 seconds from my inverter, and against this I have plotted the forecast values for each plane for the hour, updated at the start of the hour (read the leading edge of the steps).

With a lot of trial and error I have managed to get the over/under compensation to balance out and the summation curve is now a good fit for a sunny day. I continue to monitor other days, but I believe I have, at last, found a reliable and working solution.