Again that address seems not valid and you don’t have the latest emhass version there. Update.
Oh yes, there is a bug. It can’t possibly work as it right now when using method=mlforecaster and dayahead-optim. It will be fixed. For now you can just test using forecast-model-fit, forecast-model-predict and forecast-model-tune.
It is, look in my previus posst to curl http://10.10.10.80:5001 that works.
EDit: updated the docker again and seems like I was not on the newest version.
David,
All recent updates applied. 0.3.5
Still nothing?
Following a restart I just click on an optimization - day ahead.
Then I click on publish results. This is the log following those two actions.
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
2023-03-11 08:21:25,971 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2023-03-11 08:21:25,971 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2023-03-11 08:21:25,972 - web_server - INFO - The data path is: /share
2023-03-11 08:21:25,976 - web_server - INFO - Using core emhass version: 0.4.5
waitress INFO Serving on http://0.0.0.0:5000
2023-03-11 08:26:21,381 - web_server - INFO - EMHASS server online, serving index.html...
2023-03-11 08:26:21,398 - web_server - WARNING - The data container dictionary is empty... Please launch an optimization task
2023-03-11 08:26:24,376 - web_server - INFO - Setting up needed data
2023-03-11 08:26:24,528 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2023-03-11 08:26:26,322 - web_server - INFO - Setting up needed data
2023-03-11 08:26:26,327 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2023-03-11 08:26:29,737 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-03-11 08:26:29,745 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 08:26:29,818 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 08:26:29,824 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 08:26:29,830 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 585, in get_load_forecast
rh.get_data(days_list, var_list)
File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
2023-03-11 08:26:31,453 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-03-11 08:26:31,455 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 08:26:31,514 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 08:26:31,514 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 08:26:31,515 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 585, in get_load_forecast
rh.get_data(days_list, var_list)
File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
2023-03-11 08:27:01,078 - web_server - INFO - Setting up needed data
2023-03-11 08:27:01,100 - web_server - INFO - >> Publishing data...
2023-03-11 08:27:01,101 - web_server - INFO - Publishing data to HASS instance
2023-03-11 08:27:01,102 - web_server - ERROR - File not found error, run an optimization task first.
This is fixed!
1 Like
Double check this variable of your Home Assistant instance: sensor.power_load_no_var_loads
It has correct history? The data is being saved by the Home Assistant recorder? Check that the name is correct.
A procedure to use the new machine learning feature for load power forecasting:
1.- First fit the default model with the default parameters by going to the webui and using the new buttons there:
2.- After clicking on “ML forecast model fit” you can check the logs to see the fit metrics results. In my case the logs show:
And refreshing the webui now shows the graph with the results:
Zooming in:
This used the default parameters which are these:
runtimeparams = {
"days_to_retrieve": 30,
"model_type": "load_forecast",
"var_model": "sensor.power_load_no_var_loads",
"sklearn_model": "KNeighborsRegressor",
"num_lags": 48,
"split_date_delta": '48h',
"perform_backtest": False
}
But you can change any of these parameters by using a data disctionary during the curl call. Like this:
curl -i -H "Content-Type:application/json" -X POST -d '{"days_to_retrieve": 150}' http://localhost:5000/action/forecast-model-fit
Try to provide as much data as you can.
3.- The ML forecast model tune button can be used to launch a hyperparameter optimization using bayesian optimization. Again check the logs, in my case it shows:
We have better metric and we used backtest to validate this.
Refreshing the webui will give us the graphic results after this model tuning:
4.- If we are satisfied with these results the last step is to switch to this model in the add-on configuration pane by changing the load forecast method to mlforecaster:
5.- You can automate all this by defining the curl commands using the shell_command service on Home Assistant. Check the complete documentation of this feature here: The machine learning forecaster — emhass 0.4.4 documentation
Refit the model often (once a week?) to update to any changes on your consumption dynamic. Careful with the tuning routine that can be computation intensive for RPi devices. Limit the days_to_retrieve parameter if the optimization takes too long.
3 Likes
David,
that sensor doesn’t show up in entities?
Pat
Ok so there is your problem. This sensor need to exists and it should have at least two days of recorded data otherwise you will have the error shown.
Define a new template sensor and give it that name. The data for the sensor should be your load power consumption minus your deferrable/controllable loads that you want to optimize.
Read this: Intro / Quick start — emhass 0.4.4 documentation
I seem to be failing at the first step.
‘ML forecast model fit’ button fails:
2023-03-11 17:57:43,144 - web_server - INFO - Setting up needed data
2023-03-11 17:57:43,154 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 17:57:43,190 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 17:57:43,191 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 17:57:43,191 - web_server - ERROR - Exception on /action/forecast-model-fit [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 146, in set_input_data_dict
rh.get_data(days_list, var_list)
File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
If you retry you have the same error? Is it possible that this is conflicting with other data retrieve that you are doing at the same time? As you are doing high frequency MPC if I recall well.
Your sensor name is the same as the default name? Do you have at least 30 days of data for that sensor?
Thanks David,
I tried reading the docs but soon got lost. I guess I will have to spend a bit more time in there.
thanks
Pat
Ah, standard recorder only has 7 days data.
Looks like I’ll need to extend that.
Hi again
reinstalled every thing on a Generic x86-64
Managed to see webpage with some graphs.
Changed to 60 min.
but i still get errors when trying day a head and publishing the results.
final try if somebody could help me or i´ll ditch this project.
- name: "power_photovoltaics"
device_class: power
state_class: measurement
unit_of_measurement: "W"
state: "{{ states('sensor.accumulated_production_siden')|float(0) * 1000 }}"
- name: power_load_no_var_loads
unit_of_measurement: W
device_class: "power"
value_template: >-
{{states('sensor.accumulated_consumption_siden')|float(0)*1000}}
#- states('sensor.pool_heatpump_power')|int(0)
#- states('sensor.pool_pumps_power')|int(0)
#- states('sensor.hvac_power')|int(0)
#- states('sensor.hws_power_2')|int(0)
#- states('sensor.hpwc_power')|int(0)
- binary_sensor:
- name: “Luna2000”
state: >-
{{(states(‘sensor.power_photovoltaics’)|float(0)
- states(‘sensor.power_load_no_var_loads’)|float(0))
>= 5000 }}
shell_command:
publish_data: “curl -i -H “Content-Type:application/json” -X POST -d ‘{}’ http://localhost:5000/action/publish-data”
trigger_nordpool_forecast: “curl -i -H “Content-Type: application/json” -X POST -d ‘{“load_cost_forecast”:{{((state_attr(‘sensor.nordpool_kwh_se4_sek_3_10_025’, ‘raw_today’) | map(attribute=‘value’) | list + state_attr(‘sensor.nordpool_kwh_se4_sek_3_10_025’, ‘raw_tomorrow’) | map(attribute=‘value’) | list))[now().hour:][:24] }},“prod_price_forecast”:{{((state_attr(‘sensor.nordpool_kwh_se4_sek_3_10_025’, ‘raw_today’) | map(attribute=‘value’) | list + state_attr(‘sensor.nordpool_kwh_se4_sek_3_10_025’, ‘raw_tomorrow’) | map(attribute=‘value’) | list))[now().hour:][:24]}}}’ http://localhost:5000/action/dayahead-optim”
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
2023-03-11 16:47:20,194 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2023-03-11 16:47:20,195 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2023-03-11 16:47:20,197 - web_server - INFO - The data path is: /share
2023-03-11 16:47:20,245 - web_server - INFO - Using core emhass version: 0.4.5
waitress INFO Serving on http://0.0.0.0:5000
2023-03-11 16:47:27,290 - web_server - INFO - EMHASS server online, serving index.html…
2023-03-11 16:47:41,429 - web_server - INFO - Setting up needed data
2023-03-11 16:47:41,984 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2023-03-11 16:47:51,631 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-03-11 16:47:51,895 - web_server - INFO - Retrieve hass get data method initiated…
2023-03-11 16:48:25,153 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 2528, in wsgi_app
response = self.full_dispatch_request()
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1823, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File “/usr/local/lib/python3.9/dist-packages/emhass/web_server.py”, line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File “/usr/local/lib/python3.9/dist-packages/emhass/command_line.py”, line 91, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf[‘load_forecast_method’])
File “/usr/local/lib/python3.9/dist-packages/emhass/forecast.py”, line 585, in get_load_forecast
rh.get_data(days_list, var_list)
File “/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py”, line 140, in get_data
df_tp = df_raw.copy()[[‘state’]].replace(
File “/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py”, line 5920, in astype
new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py”, line 419, in astype
return self.apply(“astype”, dtype=dtype, copy=copy, errors=errors)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py”, line 304, in apply
applied = getattr(b, f)(**kwargs)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/internals/blocks.py”, line 580, in astype
new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py”, line 1292, in astype_array_safe
new_values = astype_array(values, dtype, copy=copy)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py”, line 1237, in astype_array
values = astype_nansafe(values, dtype, copy=copy)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py”, line 1098, in astype_nansafe
result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py”, line 1181, in astype_nansafe
return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: “16512.0 - states(‘sensor.battery_charge_discharge_power’)|float(0)}}”
2023-03-11 16:48:51,392 - web_server - INFO - Setting up needed data
2023-03-11 16:48:51,409 - web_server - INFO - >> Publishing data…
2023-03-11 16:48:51,410 - web_server - INFO - Publishing data to HASS instance
2023-03-11 16:48:51,466 - web_server - ERROR - Exception on /action/publish-data [POST]
Traceback (most recent call last):
File “/usr/local/lib/python3.9/dist-packages/pandas/core/arrays/datetimelike.py”, line 962, in _validate_frequency
raise ValueError
ValueError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 2528, in wsgi_app
response = self.full_dispatch_request()
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1823, in full_dispatch_request
rv = self.dispatch_request()
File “/usr/local/lib/python3.9/dist-packages/flask/app.py”, line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File “/usr/local/lib/python3.9/dist-packages/emhass/web_server.py”, line 174, in action_call
_ = publish_data(input_data_dict, app.logger)
File “/usr/local/lib/python3.9/dist-packages/emhass/command_line.py”, line 453, in publish_data
opt_res_latest.index.freq = input_data_dict[‘retrieve_hass_conf’][‘freq’]
File “/usr/local/lib/python3.9/dist-packages/pandas/core/indexes/extension.py”, line 78, in fset
setattr(self._data, name, value)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/arrays/datetimelike.py”, line 891, in freq
self._validate_frequency(self, value)
File “/usr/local/lib/python3.9/dist-packages/pandas/core/arrays/datetimelike.py”, line 973, in _validate_frequency
raise ValueError(
ValueError: Inferred frequency 30T from passed values does not conform to passed frequency H
@nixtagangsta, I do not know if this helps but I want to share my emhass config which works for me:
I use this template for the “Power load no var loads” sensor. I see from your post you maybe have wrong template for your sensor.
- name: "Power load no var loads"
unique_id: fbfeef21-1aa3-4a54-b781-426f46fef597
unit_of_measurement: "W"
device_class: "power"
state: >
{% set powerload = states('sensor.power_adresse_xx') | float(default=0) %}
{% set vvb = states('sensor.bryter_varmvannsbereder_electric_consumption_w') | float(default=0) %}
{% set varmekabler = states('sensor.power_varmekabler') | float(default=0) %}
{% set value = ( powerload - vvb - varmekabler) | round(1,default=0) %}
{{ value }}
The shell_command I use for Nordpool is the same as the emhass documentation has:
shell_command:
publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"
trigger_nordpool_forecast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}}' http://localhost:5000/action/dayahead-optim"
trigger_nordpool_mpc: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.nordpool', 'raw_today') | map(attribute='value') | list + state_attr('sensor.nordpool', 'raw_tomorrow') | map(attribute='value') | list))[now().hour:][:24]}}, \"prediction_horizon\":6, \"def_total_hours\":[2,3,3,3,3,3,3,3]}' http://localhost:5000/action/naive-mpc-optim"
trigger_entsoe_mpc: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{((state_attr('sensor.entsoe_average_electricity_price_today', 'prices_today') | map(attribute='price') | list + state_attr('sensor.entsoe_average_electricity_price_today', 'prices_tomorrow') | map(attribute='price') | list))[now().hour:][:24] }},\"prod_price_forecast\":{{((state_attr('sensor.entsoe_average_electricity_price_today', 'prices_today') | map(attribute='price') | list + state_attr('sensor.entsoe_average_electricity_price_today', 'prices_tomorrow') | map(attribute='price') | list))[now().hour:][:24]}}, \"prediction_horizon\":6, \"def_total_hours\":[2,3,3,3,3,3,3,3]}' http://localhost:5000/action/naive-mpc-optim"
“Power load no var loads” must have more than two days with data. When importing the Nordpool prices it is best to do that before midnight. Nordpool has 24 hours with prices so the time to resample retrieved data from hass must be 60 in the config.
David helped my a lot with an error a few weeks ago. I documented my setup here in this github issue: Timed out running command when passing forecast data · Issue #37 · davidusb-geek/emhass-add-on · GitHub
Maybe it helps.
You get an error in the emhass log about a battery sensor sensor which report wrong data format. Maybe try without battery setting in emhass:
Thx. Yes i noticed the battery sensor but I did remove it from my configuration file but still shows up? Any idea why?
Sorry I do not now. If you uninstall emhass addon and install it again you should have a new fresh config.
The default emhass addon config is here: emhass-add-on/config.yml at e59fcc140bc8deb1d4e90bfbb685dfbf676b1767 · davidusb-geek/emhass-add-on · GitHub and here: emhass-add-on/config_emhass.yaml at e59fcc140bc8deb1d4e90bfbb685dfbf676b1767 · davidusb-geek/emhass-add-on · GitHub
I’ll reinstall once again.
Where do I find emhass config.yaml and config_emhass.yaml in home assistant?
The files are in the emhass docker. It is used when emhass addon is started up for the first time. It is overwritten when you use your own setting in the webinterface. I mention it because you maybe want to see the default emhass setting.