Brace yourselves, DST is coming!
While I did tried to make the code to survive DST as much as I can its still not 100% guaranteed.
I suggest to closely keep and eye on errors or switch to your classic automations without optimization for a day or two where typically everything goes back to normal. We’ll see
Finally some people reported some issues and errors. Still some fixing needed for complete DST survival.
However in my case using only dayahead-optim I found no issue whatsoever, 24h optimization ran smoothly this time.
This is the output, looks fine to me?
rest_command:
dayahead_optim_rest:
url: http://localhost:5000/action/dayahead-optim
method: POST
content_type: "application/json"
payload: >-
{
"prediction_horizon": 48,
"pv_power_forecast": [3030, 4049, 4635, 4763, 4858, 4657, 4138, 3570, 2963, 2362, 1788, 1388, 1109, 805, 508, 248, 96, 5, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 58, 198, 391, 650, 918, 1234, 1607, 1956, 2299, 2657, 2994, 3334, 3659, 3815, 3804, 3735, 3603, 3313, 2934, 2525, 2119, 1636, 1133, 605, 191, 11, 0, 0, 0, 0, 0, 0, 0],
"load_cost_forecast": [0.1601, 0.1601, 0.14119, 0.14119, 0.1389, 0.1389, 0.1529, 0.1529, 0.17687, 0.17687, 0.19969, 0.19969, 0.21037, 0.21037, 0.24432, 0.24432, 0.21895, 0.21895, 0.19164, 0.19164, 0.18784, 0.18784, 0.19539, 0.19539, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874, 0.19874],
"num_def_loads": 1,
"def_total_hours": [4],
"P_deferrable_nom": [2500],
"treat_def_as_semi_cont": [0],
"set_def_constant": [1]
}
I have this issue also and an other one.
I thought it was my fault(issue started a few days ago) and was waiting for it to clear on it’s own.
ERROR - web_server - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 104, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 93, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 626, in get_load_forecast
data = pd.DataFrame.from_dict(data_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 1760, in from_dict
return cls(data, index=index, columns=columns, dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 709, in __init__
mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 481, in dict_to_mgr
return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 115, in arrays_to_mgr
index = _extract_index(arrays)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 655, in _extract_index
raise ValueError("All arrays must be of the same length")
ValueError: All arrays must be of the same length
ERROR - web_server - Exception on /action/perfect-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 2423, in _infer_tz_from_endpoints
inferred_tz = timezones.infer_tzinfo(start, end)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "pandas/_libs/tslibs/timezones.pyx", line 368, in pandas._libs.tslibs.timezones.infer_tzinfo
AssertionError: Inputs must both have the same timezone, UTC+01:00 != UTC+02:00
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 119, in action_call
opt_res = perfect_forecast_optim(input_data_dict, app.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 207, in perfect_forecast_optim
opt_res = input_data_dict['opt'].perform_perfect_forecast_optim(df_input_data, input_data_dict['days_list'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/optimization.py", line 538, in perform_perfect_forecast_optim
data_tp = df_input_data.copy().loc[pd.date_range(start=day_start, end=day_end, freq=self.freq)]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/datetimes.py", line 945, in date_range
dtarr = DatetimeArray._generate_range(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 422, in _generate_range
tz = _infer_tz_from_endpoints(start, end, tz)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/arrays/datetimes.py", line 2426, in _infer_tz_from_endpoints
raise TypeError(
TypeError: Start and end cannot both be tz-aware with different timezones
The first error is the error related to DST. Needs to be worked out. I just applied a new fix.
The second error seems like a bug when dealing with TZ. Will look into it.
Dear community I have created a new webapp to try to ease setting up the scrape + PVLib method for PV power forecast. I still believe that this is the more accurate option for PV power forecast. Unfortunately the PV modules and inverter database from PVLib is quite outdated. So I decided to update those databases and to add them directly to EMHASS, I will keep them updated regularly.
So this webapp can be used to search the names of your own module and inverter model in the database.
The EMHASS will be updated in the upcoming days and the following release will include this database directly in the source code.
Here is the link to the webapp: https://emhass-pvlib-database.streamlit.app/
1 Like
Will this be a brand new DB or will devices available in current PVLib be included as well?
Asking as in the webapp I can’t find an inverter I’m using: SMA_America__SB3000TL_US_22__240V_
It is an updated database but where you should be able to find the inverter that you were using before, so no change needed if you already found your correct inverter. But you are right, for the inverters I just forgot to merge the old database with the new, so there are some missing keys that were there before. I will fix it, no problem.
Done, the new DB are updated 
1 Like
@davidusb , from 0.8.5 release notes
:
- Added new tests for forecast longer than 24h by changing parameter `delta_forecast
Does that mean we can have a longer forecast now?
You can now and you could before. Just need to change the delta_forecast parameter in the configuration to the number of days that you want to forecast. I just added some unit tests to test this parameter functionality and check that is working as intended, and it is.
1 Like
Ok. Probably misunderstood. I thought I can make MPC optimization now for example 36h but this is not it.
Use case is that around 2pm each day tomorrow’s Noordpool hourly prices will be public and it would be nice to make optimization until the end of next day.
Yes you can do MPC for as long as you want. Modify the said delta_forecast parameter and of course the prediction_horizon parameter
Interesting, I tried it about a month ago and turned out it can be 24h max.
If I put both prediction horizon and delta for example to 30 (I have hourly data) it should work?
I actually use mlforecast, so the delta forecast is not applicable?
Edit: same issue here EMHASS: An Energy Management for Home Assistant - #1658 by michaelpiron
The MLForecaster will predict to the num_lags that you fixed
Is there a way to add data from your own inverter? Mine isn’t listed.
I’m using an huawei sun2000-4ktl-m1, all huawei’s that are listed are US versions.
And how do you calculate some of those values that are needed?
@davidusb
When i did try again to forecast next 31 hours i got an ValueError: Shape of passed values is (14, 1), indices imply (7, 1):
2024-04-02 17:30:00,479 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.07, 0.109, 0.146, 0.173, 0.172, 0.091, 0.071, 0.042, 0.071, 0.065, 0.061, 0.056, 0.073, 0.124, 0.167, 0.199, 0.198, 0.168, 0.158, 0.154, 0.149, 0.146, 0.143, 0.146, 0.15, 0.169, 0.191, 0.206, 0.18, 0.131, 0.115], 'prod_price_forecast': [0.009, 0.041, 0.071, 0.093, 0.093, 0.042, 0.026, 0.002, 0.025, 0.021, 0.017, 0.013, 0.027, 0.069, 0.088, 0.115, 0.114, 0.089, 0.081, 0.077, 0.074, 0.071, 0.069, 0.071, 0.075, 0.09, 0.108, 0.12, 0.099, 0.075, 0.061], 'prediction_horizon': 31, 'pv_power_forecast': [376, 232, 72, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 151, 459, 907, 1427, 1844, 2111, 2195, 2108, 1958, 1461, 835, 410, 121, 0, 0, 0, 0], 'var_model': 'sensor.ss_load_power', 'var_load:': 'sensor.ss_load_power', 'delta_forecast': 20}
2024-04-02 17:30:00,480 - web_server - INFO - >> Setting input data dict
2024-04-02 17:30:00,480 - web_server - INFO - Setting up needed data
2024-04-02 17:30:00,483 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-02 17:30:00,799 - web_server - INFO - Retrieving weather forecast data using method = list
2024-04-02 17:30:00,801 - web_server - INFO - Retrieving data from hass for load forecast using method = mlforecaster
2024-04-02 17:30:00,801 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-02 17:30:01,626 - web_server - INFO - >> Performing naive MPC optimization...
2024-04-02 17:30:01,626 - web_server - INFO - Performing naive MPC optimization
2024-04-02 17:30:01,633 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 145, in action_call
opt_res = naive_mpc_optim(input_data_dict, app.logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 273, in naive_mpc_optim
df_input_data_dayahead = input_data_dict['fcst'].get_load_cost_forecast(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 715, in get_load_cost_forecast
forecast_out = self.get_forecast_out_from_csv(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 530, in get_forecast_out_from_csv
forecast_out = pd.DataFrame(
^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 758, in __init__
mgr = ndarray_to_mgr(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 337, in ndarray_to_mgr
_check_values_indices_shape_match(values, index, columns)
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 408, in _check_values_indices_shape_match
raise ValueError(f"Shape of passed values is {passed}, indices imply {implied}")
ValueError: Shape of passed values is (14, 1), indices imply (7, 1)
There is data for each input for 31 hours.
Also MLforecast fit gave me error:
2024-04-02 17:35:51,930 - web_server - INFO - Passed runtime parameters: {'days_to_retrieve': 20, 'model_type': 'load_forecast', 'var_model': 'sensor.ss_load_power', 'num_lags': 24, 'split_date_delta': '48h', 'perform_backtest': 'True'}
2024-04-02 17:35:51,930 - web_server - INFO - >> Setting input data dict
2024-04-02 17:35:51,930 - web_server - INFO - Setting up needed data
2024-04-02 17:35:51,933 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-02 17:35:51,937 - web_server - ERROR - The retrieved JSON is empty, A sensor:sensor.ss_load_power may have 0 days of history or passed sensor may not be correct
Could it be because of daylight saving time? Error occured after updating to 0.8.5
i do have data in sensor.ss_load_power
EDIT: Also dayahead is not working:
2024-04-02 17:40:07,717 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.07, 0.109, 0.146, 0.173, 0.172, 0.091, 0.071, 0.042, 0.071, 0.065, 0.061, 0.056, 0.073, 0.124, 0.167, 0.199, 0.198, 0.168, 0.158, 0.154, 0.149, 0.146, 0.143, 0.146], 'prod_price_forecast': [0.009, 0.041, 0.071, 0.093, 0.093, 0.042, 0.026, 0.002, 0.025, 0.021, 0.017, 0.013, 0.027, 0.069, 0.088, 0.115, 0.114, 0.089, 0.081, 0.077, 0.074, 0.071, 0.069, 0.071], 'prediction_horizon': 24, 'pv_power_forecast': [376, 232, 72, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3, 151, 459, 907, 1427, 1844, 2111, 2195, 2108, 1958, 1461, 835, 410, 121, 0, 0, 0, 0], 'var_model': 'sensor.ss_load_power', 'var_load:': 'sensor.ss_load_power', 'delta_forecast': 20}
2024-04-02 17:40:07,717 - web_server - INFO - >> Setting input data dict
2024-04-02 17:40:07,718 - web_server - INFO - Setting up needed data
2024-04-02 17:40:07,722 - web_server - ERROR - ERROR: The passed data is either not a list or the length is not correct, length should be 48
2024-04-02 17:40:07,723 - web_server - ERROR - Passed type is <class 'list'> and length is 31
2024-04-02 17:40:07,723 - web_server - ERROR - ERROR: The passed data is either not a list or the length is not correct, length should be 48
2024-04-02 17:40:07,724 - web_server - ERROR - Passed type is <class 'list'> and length is 24
2024-04-02 17:40:07,724 - web_server - ERROR - ERROR: The passed data is either not a list or the length is not correct, length should be 48
2024-04-02 17:40:07,724 - web_server - ERROR - Passed type is <class 'list'> and length is 24
2024-04-02 17:40:07,735 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2024-04-02 17:40:08,717 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 108, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
df_weather = fcst.get_weather_forecast(method=optim_conf['weather_forecast_method'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 210, in get_weather_forecast
raw_data.set_index(forecast_dates_scrap, inplace=True)
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 5910, in set_index
raise ValueError(
ValueError: Length mismatch: Expected 24 rows, received array of length 48
No we can’t add new items to the list. The parameters are fitted by the NREL and other labs from experimental data.
But the DB is pretty big, for example in your case this one will do the trick I guess: Huawei_Technologies_Co___Ltd___SUN2000_11_4KTL_USL0__240V_
The delta_forecast parameter is in days, you are fixing it to 20?!!
Like I said you need to change your num_lags parameter in the MLForecaster.
Example: to forecast 2 days using the MLForecaster you’ll need to set delta_forecast=2 and num_lags=96