Love it Mark… great perspective!
1 Like
not work:
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
2023-06-10 12:54:21,365 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2023-06-10 12:54:21,366 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2023-06-10 12:54:21,367 - web_server - INFO - The data path is: /share
2023-06-10 12:54:21,371 - web_server - INFO - Using core emhass version: 0.4.12
waitress INFO Serving on http://0.0.0.0:5000
2023-06-10 12:54:59,043 - web_server - INFO - EMHASS server online, serving index.html...
2023-06-10 12:54:59,056 - web_server - WARNING - The data container dictionary is empty... Please launch an optimization task
2023-06-10 12:55:42,583 - web_server - INFO - Setting up needed data
2023-06-10 12:55:42,665 - web_server - INFO - Retrieve hass get data method initiated...
2023-06-10 12:55:42,702 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-06-10 12:55:42,702 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-06-10 12:55:42,703 - web_server - ERROR - Exception on /action/perfect-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 174, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 78, in set_input_data_dict
rh.get_data(days_list, var_list,
File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
Also python is on either 3.10 or 3.11 currently in the releases. My question is why has yours not updated? The RED line error.
That error has absolutely nothing to do with a Python version. Python 3.9 is still completely supported but yes we should update to 3.10. Just have to find the time to test that 
My curl command is getting quite complex to manage:
I have now switched to a dedicated rest_command, which is still complex, but less so I also get errors reported in the logs directly, which is helpful:
rest_command:
naive_mpc_optim:
url: http://localhost:5000/action/naive-mpc-optim
method: POST
content_type: 'application/json'
payload: >-
{
"prod_price_forecast": {{
([states('sensor.amber_feed_in_price')|float(0)] +
(state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list))
| tojson
}},
"load_cost_forecast": {{
([states('sensor.amber_general_price')|float(0)] +
state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
| tojson
}},
"load_power_forecast": {{
([states('sensor.power_load_no_var_loads')|int] +
(states('input_text.fi_fo_buffer').split(', ')|map('multiply',1000)|map('int')|list)[1:]
)| tojson
}},
"pv_power_forecast": {{
([states('sensor.APF_Generation_Entity')|int(0)] +
state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list +
state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list
)| tojson
}},
"prediction_horizon": {{
min(48, (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
}},
"alpha": 1,
"beta": 0,
"num_def_loads": 6,
"def_total_hours": {{[states('sensor.def_total_hours_pool_filter')|int(0),
states('sensor.def_total_hours_pool_heatpump')|int(0),
states('sensor.def_total_hours_ev')|int(0),
states('sensor.def_total_hours_hvac')|int(0),
states('sensor.def_total_hours_hws')|int(0),
states('input_number.car2_def_total_hours')|int(0)]}},
"P_deferrable_nom": [1300, 5000, 11500, 4000, 600, 3500],
"treat_def_as_semi_cont": [1, 1, 0, 0, 1, 0],
"set_def_constant": [0, 0, 0, 0, 0, 0],
"soc_init": {{ (states('sensor.filtered_powerwall_soc')|int(0))/100 }},
"soc_final": 0.0
}
publish_data:
url: http://localhost:5000/action/publish-data
method: POST
content_type: 'application/json'
payload: '{}'
3 Likes
Done! Python 3.10 is now supported.
Possible Python versions are 3.8, 3.9 and 3.10
Hi
Trying to set up EMHASS but finding the instructions difficult. Would be very appreciative of some help.
- I have 5kW PV.
- A Fronius Primo 6.0-1 inverter.
- I’m using solcast to predict PV production.
- A 10 kWh (9 kWh usable) sonnen eco 9.43 battery.
- I use the Sonnenbatterie HACS addon by weltmeyer to obtain all the battery metrics.
- I can use POST to switch the battery between charge, discharge, self consumption and TOU modes and probably others that I haven’t discovered yet.
- I’m on the Australian Amber Electric service (They don’t support Sonnen).
- I’m using the Amber Electric integration in HA to get the pricing data etc.
- Deferable loads might be pool pump, dish washer, dryer, washing machine but I currently don’t monitor power consumption for any of these loads.
- I have one EV, Tesla model Y.
- I’ve been using Charge HQ to divert feed-in to the car but suspect I can replace that with EMHASS.
I’ve set up the following in HA:
- Automation
- alias: EMHASS day-ahead optimization
trigger:
platform: time
at: 05:30:00
action:
- service: shell_command.dayahead_optim
- alias: EMHASS publish data
trigger:
- minutes: /5
platform: time_pattern
action:
- service: shell_command.publish_data
- configuration.yaml
shell_command:
dayahead_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim"
publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"
post_amber_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":{{(
state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
}},\"load_cost_forecast\":{{(
state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
}},\"prediction_horizon\":33}' http://localhost:5000/action/dayahead-optim"
post_emhass_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":{{(
state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
}},{{states('sensor.solcast_24hrs_forecast')}},\"load_cost_forecast\":{{(
state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
}}}' http://localhost:5000/action/dayahead-optim"
post_mpc_optim_solcast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{(
([states('sensor.amber_general_price')|float(0)] +
state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:48])
}}, \"prod_price_forecast\":{{(
([states('sensor.cecil_st_feed_in_price')|float(0)] +
state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)[:48])
}}, \"pv_power_forecast\":{{states('sensor.solcast_24hrs_forecast')
}}, \"prediction_horizon\":48,\"soc_init\":{{(states('sensor.sonnenbatterie_84324_state_charge_user')|float(0))/100
}},\"soc_final\":0.05,\"def_total_hours\":[2,0,0,0]}' http://localhost:5000/action/naive-mpc-optim"
and
# Solar forecast for EMHASS
- platform: rest
name: "Solcast Forecast Data"
json_attributes:
- forecasts
resource: https://api.solcast.com.au/rooftop_sites/YYYYYYYYYYYYY/forecasts?format=json&api_key=XXXXXXXXXXXXXXXXXXXXXXXXXX&hours=24
method: GET
value_template: "{{ (value_json.forecasts[0].pv_estimate)|round(2) }}"
unit_of_measurement: "kW"
device_class: power
scan_interval: 8000
force_update: true
- platform: template
sensors:
solcast_24hrs_forecast :
value_template: >-
{%- set power = state_attr('sensor.solcast_forecast_data', 'forecasts') | map(attribute='pv_estimate') | list %}
{%- set values_all = namespace(all=[]) %}
{% for i in range(power | length) %}
{%- set v = (power[i] | float |multiply(1000) ) | int(0) %}
{%- set values_all.all = values_all.all + [ v ] %}
{%- endfor %} {{ (values_all.all)[:48] }}
I assume I have to change the automation to also call the amber, EMHASS and solcast curl shell commands? But not sure if this replaces the default dayahead_optim and publish_data curl commands.
Haven’t even looked into what I do with the returned results from EMHASS. Just wanted to get this first stage set up correctly.
Do I need to measure deferable loads so I can subtract them from house consumption figure I get from battery? If so, I’ll have to go out and gets some power points that monitor power.
Am I going about this in the right way. What changes do I need to make and what next.
Thanks in advance.
Looks like you have a lot of components.
I would start with a basic system.
If you can control your battery charge/ discharge via a post command, then you should be able to get the same level of functionality as SmartShift.
Your EV and pool pump will also integrate well and you should be able to get excellent optimisation results. You will need a smart socket for the pool but the EV should be able to integrate directly.
You only need to call one optimisation and one plublish content, the others are just spending that at in additional functionality.
As it is setup it will call the simple dayahead once a day at 0530 optimisation which is good and then publish these results every 5 minutes.
You can find my post here on how you can connect other devices:
Running Devices When Energy is Cheaper and Greener
@davidusb did you have a chance to look at the errors I posted above in EMHASS: An Energy Management for Home Assistant - #728 by Octofinger?
Seems emhass is choking on microseconds in HA sensor timestamps.
You have a problem retrieving your data from Home Assistant but with the given information I’m not able to reproduce that. I’m able to perfectly retrieve data from HA using the legacy Python code on a virtual environment, that’s proven to work so there must be a problem with your configuration.
You can maybe test using the standalone docker container, it is more practical to find the issue.
Otherwise open a new issue on github providing a way to reproduce the error.
1 Like
Yes, I do have a basic system in place. I migrated to Node-Red about 4 years ago for automations.
- Discharge battery when FiT goes above 51c and return to self-consumption when FiT goes below 49c
- Charge the battery at 05:30 for a half hour if the amber general price is low and the battery is below 15% SoC
- Charge the battery at 15:30 for an hour if the amber general price is low and the battery is below 80% SoC
- Pool pump only comes on if the battery is 100% SoC and there is at least 1.5 kW PV for limited time depending on season
- Turn on aircon and pool pump when FiT goes negative and battery is 100% SoC or PV Feed-in is positive
I don’t usually charge the car at home as there are too many free chargers located around me to bother, although that’s getting difficult with more and more EVs on the road.
Difficult limiting use of other loads (washing machines etc.) as people will use them whenever they want despite my focus on energy consumption, but I’m willing to try.
So, my main aim is to see what EMHASS will deliver to the logic. But it’s difficult to configure without reading through pages of hard to understand explanations of control systems. I don’t really need to know the details of MPC etc, just the different configuration steps to achieve the different modes of operation.
Just to make sure, you also get microseconds from HA in last_changed and last_updated timestamps? Or is it just me?
I’ll give the docker version a try.
Thanks for helping out!
No, I’ve never had these problems with microseconds. Your problem is with your configuration.
In the --config argument you need to pass the path with the yaml file included like this:
emhass --action 'dayahead-optim' --config './config_emhass.yaml' --costfun 'profit'
That’s what I find so hard with the documentation. The documentation is clear, but the application doesn’t behave as the documentation states. If I run the command above, I get the below. Apparently, the application takes the --config argument as a path to the folder where a config.yaml is located.
> emhass --action 'dayahead-optim' --config './config_emhass.yaml' --costfun 'profit'
C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\pvlib\forecast.py:20: UserWarning: The forecast module algorithms and features are highly experimental. The API may change, the functionality may be consolidated into an io module, or the module may be separated into its own package.
warnings.warn(
2023-06-13 11:07:35,394 - emhass.command_line - INFO - Setting up needed data
INFO:emhass.command_line:Setting up needed data
Traceback (most recent call last):
File "<frozen runpy>", line 198, in _run_module_as_main
File "<frozen runpy>", line 88, in _run_code
File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Scripts\emhass.exe\__main__.py", line 7, in <module>
File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\command_line.py", line 182, in main
input_data_dict = setUp(config_path, costfun, logger)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\command_line.py", line 30, in setUp
retrieve_hass_conf, optim_conf, plant_conf = get_yaml_parse(config_path)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\utils.py", line 70, in get_yaml_parse
with open(config_path + '/config.yaml', 'r') as file:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './config_emhass.yaml/config.yaml'
Do you have the same result if you provide and absolute path instead of a relative path like this?
emhass --action 'dayahead-optim' --config '/home/user/emhass/config_emhass.yaml' --costfun 'profit'
Hi David,
After a HASS restart, I am seeing the following error when a run the predict and publish for the new ML Forcaster.
2023-06-14 08:45:33,181 - web_server - INFO - Setting up needed data
2023-06-14 08:45:33,182 - web_server - INFO - Retrieve hass get data method initiated...
2023-06-14 08:45:34,437 - web_server - INFO - >> Performing a machine learning forecast model predict...
2023-06-14 08:45:34,442 - web_server - ERROR - Exception on /action/forecast-model-predict [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 216, in action_call
df_pred = forecast_model_predict(input_data_dict, app.logger)
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 361, in forecast_model_predict
predictions = mlf.predict(data_last_window)
File "/usr/local/lib/python3.9/dist-packages/emhass/machine_learning_forecaster.py", line 210, in predict
predictions = self.forecaster.predict(steps=self.num_lags,
File "/usr/local/lib/python3.9/dist-packages/skforecast/ForecasterAutoreg/ForecasterAutoreg.py", line 637, in predict
check_predict_input(
File "/usr/local/lib/python3.9/dist-packages/skforecast/utils/utils.py", line 620, in check_predict_input
raise ValueError(
ValueError: `last_window` has missing values.
I have run the “fit” and “tune” steps. Any thoughts? I am running 0.3.13 of the add on.
Hi. I will need to test this but I see why the fit and tune methods worked and not the prediction. This is because there are NaN’s in your data probably caused by the HA restart and they are not treated for the prediction and the last window of data needed for that prediction. They are only treated for fit and tune. This should be fixed and a linear interpolation is a good solution for small windows of missing values. However if you have too many missing values it can lead to unexpected results. But I guess is the best we can do in that case.
It should be fixed if you wait for correct non missing data for at least the number of lags used by your predictor, ex. 48h, this value is shown in the logs when launching the fit and tune methods.
Could you please open an issue on github to fix this?
Just a question about the forecast, almost every day is spot on exacly correct forecast. But if there are som sky or rain it miss a lot like this.
The solarpanel that is configured in emhass is:
Phono_Solar_Technology_Co__Ltd__PS360MH_24_T, Dont understand where to find these list in the documentation for other 360W mono crystal panels. if we could try replacing with another panels see if results will be more close on all days?
thanks David.
1 Like