EMHASS: An Energy Management for Home Assistant

Thanks, I tried your curl as well as several variations.
Always either getting a 500 or 400 error.

400:
request (your sample):

curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":[0.01, -0.04, -0.05, -0.05, -0.06, -0.1, -0.16, -0.16, -0.14, -0.08, -0.06, -0.05, -0.05, -0.01, 0.01, 0.03, 0.07, 0.17, 0.3, 0.3, 0.3, 0.3, 0.25, 0.28, 0.23, 0.22, 0.21, 0.22, 0.21, 0.22, 0.22, 0.22, 0.22, 0.23, 0.21, 0.18, 0.16, 0.16, 0.16],\"load_cost_forecast\":[0.09, 0.04, 0.03, 0.02, 0.01, -0.03, -0.09, -0.09, -0.07, -0.0, 0.02, 0.02, 0.02, 0.07, 0.09, 0.12, 0.16, 0.27, 0.41, 0.41, 0.41, 0.41, 0.36, 0.39, 0.34, 0.33, 0.31, 0.33, 0.32, 0.33, 0.32, 0.33, 0.32, 0.34, 0.32, 0.28, 0.26, 0.26, 0.26]}' http://192.168.0.102:5000/action/dayahead-optim

reply:

26 Sep 08:16:30 - [info] [debug:debug 2] 
{
  _msgid: 'b5234528097a1aa3',
  topic: '',
  payload: 'HTTP/1.1 400 BAD REQUEST\r\n' +
    'Content-Length: 167\r\n' +
    'Content-Type: text/html; charset=utf-8\r\n' +
    'Date: Mon, 26 Sep 2022 06:16:30 GMT\r\n' +
    'Server: waitress\r\n' +
    '\r\n' +
    '<!doctype html>\n' +
    '<html lang=en>\n' +
    '<title>400 Bad Request</title>\n' +
    '<h1>Bad Request</h1>\n' +
    '<p>The browser (or proxy) sent a request that this server could not understand.</p>\n',
  rc: { code: 0 }
}
26 Sep 08:16:30 - [info] [debug:debug 2] 
{
  _msgid: 'b5234528097a1aa3',
  topic: '',
  payload: '  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n' +
    '                                 Dload  Upload   Total   Spent    Left  Speed\n' +
    '\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   697  100   167  100   530  25955  82374 --:--:-- --:--:-- --:--:--  136k\n',
  rc: { code: 0 }
}
26 Sep 08:16:30 - [info] [debug:debug 2] 
{ _msgid: 'b5234528097a1aa3', topic: '', payload: { code: 0 } }

500:
request:

curl -i -H 'Content-Type: application/json' -X POST -d '{"prod_price_forecast":[0.01, -0.04, -0.05, -0.05, -0.06, -0.1, -0.16, -0.16, -0.14, -0.08, -0.06, -0.05, -0.05, -0.01, 0.01, 0.03, 0.07, 0.17, 0.3, 0.3, 0.3, 0.3, 0.25, 0.28, 0.23, 0.22, 0.21, 0.22, 0.21, 0.22, 0.22, 0.22, 0.22, 0.23, 0.21, 0.18, 0.16, 0.16, 0.16]}' http://localhost:5000/action/dayahead-optim

reply:

26 Sep 08:18:05 - [info] [debug:debug 2] 
{
  _msgid: '8c1603e0cdeb32c9',
  topic: '',
  payload: 'HTTP/1.1 500 INTERNAL SERVER ERROR\r\n' +
    'Content-Length: 265\r\n' +
    'Content-Type: text/html; charset=utf-8\r\n' +
    'Date: Mon, 26 Sep 2022 06:18:03 GMT\r\n' +
    'Server: waitress\r\n' +
    '\r\n' +
    '<!doctype html>\n' +
    '<html lang=en>\n' +
    '<title>500 Internal Server Error</title>\n' +
    '<h1>Internal Server Error</h1>\n' +
    '<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>\n',
  rc: { code: 0 }
}
26 Sep 08:18:05 - [info] [debug:debug 2] 
{
  _msgid: '8c1603e0cdeb32c9',
  topic: '',
  payload: '  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n' +
    '                                 Dload  Upload   Total   Spent    Left  Speed\n' +
    '\r  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0\r100   266    0     0  100   266      0    221  0:00:01  0:00:01 --:--:--   221\r100   266    0     0  100   266      0    120  0:00:02  0:00:02 --:--:--   120\r 50   531    0     0  100   266      0    107  0:00:02  0:00:02 --:--:--   107\r100   531  100   265  100   266    106    107  0:00:02  0:00:02 --:--:--   213\n',
  rc: { code: 0 }
}
26 Sep 08:18:05 - [info] [debug:debug 2] 
{ _msgid: '8c1603e0cdeb32c9', topic: '', payload: { code: 0 } }

I trigger these via Node-Red and tried localhost as well as the local IP of the raspberry running Home Assistant (and EMHASS add-on).

Any idea @markpucell or @davidusb?

Thanks

Please post the EMHASS logs

When I type this in my command line I get the following error, which is expected as I have 30 minute time slots in my logs:

[2022-09-26 16:23:43,653] INFO in command_line: Setting up needed data
[2022-09-26 16:23:43,655] ERROR in utils: ERROR: The passed data is either not a list or the length is not correct, length should be 48
[2022-09-26 16:23:43,655] ERROR in utils: Passed type is <class 'list'> and length is 48

What do you see in the logs, under the add-on, EMHASS section?

You seem to have 39 elements in your list.

If you are posting on the command line, I find I need to enclose with " rather than ā€™

"Content-Type: application/json"

mark@odroid:~$ curl -i -H "Content-Type: application/json" -X POST -d '{"prod_price_forecast":[0.13, 0.15, 0.19, 0.24, 0.29, 0.22, 0.18, 0.18, 0.17, 0.17, 0.16, 0.17, 0.17, 0.17, 0.16, 0.17, 0.16, 0.16, 0.13, 0.15, 0.13, 0.13, 0.13, 0.12, 0.13, 0.14, 0.13, 0.13, 0.13, 0.11, 0.1, 0.1, 0.1, 0.08, 0.07, 0.07, 0.04, 0.05, 0.04, 0.04, 0.06, 0.04, 0.07, 0.07, 0.09, 0.08, 0.09, 0.1]}' http://localhost:5000/action/dayahead-optim
HTTP/1.1 201 CREATED
Content-Length: 45
Content-Type: text/html; charset=utf-8
Date: Mon, 26 Sep 2022 06:53:13 GMT
Server: waitress

EMHASS >> Action dayahead-optim executed... 
mark@odroid:~$ 

Node-Red request (now holding 24 values), done via exec node:

26 Sep 08:52:47 - [info] [debug:debug 1] 
{
  payload: `curl -i -H "Content-Type: application/json" -X POST -d '{"prod_price_forecast":[0.24397000000000002,0.17504999999999998,0.15308,0.13158999999999998,0.13334,0.20021,0.36932,0.40281,0.40253999999999995,0.37013999999999997,0.37012999999999996,0.37415999999999994,0.37412,0.33176999999999995,0.28028999999999993,0.28026999999999996,0.36515,0.37512,0.40264,0.43901999999999997,0.39177999999999996,0.32489999999999997,0.31744999999999995,0.25911]}' http://192.168.0.102:5000/action/dayahead-optim`,
  _msgid: '3cb4f3e86362c00d'
}

EMHASS log:

[2022-09-26 08:56:02,943] INFO in command_line: Setting up needed data
[2022-09-26 08:56:02,952] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2022-09-26 08:56:05,491] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2022-09-26 08:56:05,493] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-09-26 08:56:05,551] ERROR in retrieve_hass: The retrieved JSON is empty, check that correct day or variable names are passed
[2022-09-26 08:56:05,551] ERROR in retrieve_hass: Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
[2022-09-26 08:56:05,553] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 78, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 495, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 130, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment

I see ā€œThe retrieved JSON is empty, check that correct day or variable names are passedā€.
Does it require the space between each value in the json array?

I also tried it with ".
Pls find the log in my other reply.

Thanks

No it doesnā€™t require spaces in the JSON array, perhaps try pasting into a command shell directly, and once that is working they try node-red later.

Your data works here for me with the naive-mpc-optim.

mark@odroid:~$ curl -i -H "Content-Type: application/json" -X POST -d '{"prod_price_forecast":[0.24397000000000002,0.17504999999999998,0.15308,0.13158999999999998,0.13334,0.20021,0.36932,0.40281,0.40253999999999995,0.37013999999999997,0.37012999999999996,0.37415999999999994,0.37412,0.33176999999999995,0.28028999999999993,0.28026999999999996,0.36515,0.37512,0.40264,0.43901999999999997,0.39177999999999996,0.32489999999999997,0.31744999999999995,0.25911], "prediction_horizon":24}' http://localhost:5000/action/naive-mpc-optim
HTTP/1.1 201 CREATED
Content-Length: 46
Content-Type: text/html; charset=utf-8
Date: Mon, 26 Sep 2022 07:00:36 GMT
Server: waitress

EMHASS >> Action naive-mpc-optim executed... 
mark@odroid:~$

When did you setup your noloads sensor? by default it tries to go back 2 days, you can change this default days_to_retrieve in the configuration.

I set it up a few days ago but now reinstalled everything to see if this fixes it - which it doesnā€™t.

Minimum for days_to_retrieve is 2 days.

I tried the command via the terminal add-on but this gives me "curl. (7) Failed to connect to localhost port 5000 after 0 ms: Connection refused.

EMHASS seems to be running but I canā€™t access the interface at the moment because Iā€™m in the office and only port-forward some ports (not 5000).

[2022-09-26 09:24:22,352] INFO in web_server: Launching the emhass webserver at: http://0.0.0.0:5000
[2022-09-26 09:24:22,353] INFO in web_server: Home Assistant data fetch will be performed using url: http://supervisor/core/api
[2022-09-26 09:24:22,353] INFO in web_server: The base path is: /usr/src
[2022-09-26 09:24:22,358] INFO in web_server: Using core emhass version: 0.3.19

Yes, I had that challenge trying to login to a command line on my home assistant box from work.

I found the Terminal & SSH server add-on, will allow you to login to a local command line, but I havenā€™t had much joy cutting and pasting in that terminal session.

Sounds like you may need to wait the two days for the sensor to fill up with meaningful data.

Iā€™m a bit more concerned about ā€œERROR in retrieve_hass: The retrieved JSON is empty, check that correct day or variable names are passedā€.

Sounds to me that this is the underlying reason why I get 500 errors when trying to import prices for the next day.

Yes the minimum days_to_retrieve is 2 and this is just how the HA API was designed, we can olny retrieve entire history days in the recorder.
Donā€™t worry too much about the empty json error, this just most probably means in your case that you donā€™t have enough data in your sensor.

@davidusb @markpurcell

SOLUTION:
After I created a port forwarding (5000 to 5000) and used the external HA-address (without https) to send the command from Node-Red to EMHASS, the dayahead optimization worked.
If relevant: Iā€™m using the duckdns add-on with the built-in option to generate SSL certificates.
The Home Assistant Instance URL is still set to empty as settings the https URL did not work.

@davidusb Is it possible to set a minimum runtime and a timeframe for a deferrable load? Iā€™d like to plan my domestic hot water heat pump with EMHASS but turning it on/off frequently isnā€™t healthy for it. Itā€™d be better to set it for a minimum runtime of three hours for example. At the moment it might split 3 hours to 12hrs and 11hr.

Thanks

I have just replaced my instant gas hot water with a heat pump hot water system so am in the process of setting this up for control by EMHASS.

My heatpump is the Stiebel Electron WWK 302 which has an impressed current anode to protect the tank from corrosion. By default it has one mains lead which can be plugged into a 230V/ 10A socket. The until contains a factory installed battery to protect the anode during power cuts but it is recommended if you are going to switch the unit regularly that you either connect the anode circuit separately to a non switched circuit or replace the battery every three years. So I got my installer to connect a second 230V lead for the anode and control electronics to run continuously (drawing about 4 W) whilst the first circuit switches the compressor (drawing between 400W - 600W) when operating. The unit also has a SmartGrid signal line that can temporarily raise the set point from 61Ā° C to 65Ā° C. I have installed a Shelly 1 to switch this signal line, but havenā€™t yet integrated that into EMHASS.

I recommend you check your manual to see if you are able to protect your anode in this fashion.

Using EMHASS I have setup a 500 W deferrable load with a runtime of 3 hours to control the compressor circuit.

There is an EMHASS option you can set for continuous block operation, which I have considered but havenā€™t used for the compressor circuit:

set_def_constant: Define if we should set each deferrable load as a constant fixed value variable with just one startup for each optimization task.

I have only had it running this week but the results have been pretty reliable with most scheduling occurring during the midday solar soak window/ or my cheap buy signal from my energy provider.

Iā€™ve been using emhass to control my water heater reliably for more than two years now. As long as I have a 30min time step the water heater will not switch on/off at high frequencies. When operating manually with only the thermostat it actually switch more often. However in my use case I donā€™t have a lot of variance in my costs inputs.
As Mark noted there is an option to force fixed values in the optimization. This will add more constraints to the linear programming optimization, so if you are close to the solver problem size limit this could not be a good idea. Otherwise the set_def_constant can be set on the add-on during runtime inside the shell command.

@davidusb
Iā€™m using a 60 minutes step currently as I have pv production prices for one hours also (might consider changing that to 30 minutes and just doubling each value in the array).

The heatpump is only for hot water production and therefore does not need to run several times a day. I will check the option ā€œset_def_constantā€.
Regarding my question on timeframes: It would be cool if itā€™d be possible to set timeframes in which specific def loads are allowed to run. Not only to for example move this timeframe close to before itā€™s usually needed (like heating up the water in the afternoon so itā€™s hot when showering in the evening), but also as things like non-inverter (and therefore loud) heatpumps, especially for pools, might run in the mid of the night, disturbing neighbors and ourselfs. By saying ā€œdef_load_1 can only run between X and Yā€, this issue could be addressed.
Also, a minimum time between each run of a def load would be nice, to for example stop the device from running from 22:00 to 23:00 and then again from like 01:00 to 02:00 and not the rest of the day.

@markpurcell
My electric anode is separated from the heat pumps electrical wiring and therefore runs 24/7, only consuming a few watts.
Unfortunately the water heat pump is dump. You can set a temperature and trigger it on/off, but no temperature increase based on a relay.

A second method uses the SolCast solar forecast service. Go to https://solcast.com/ and configure your system. You will need to set method=solcast and basically use two parameters solcast_rooftop_id and solcast_api_key that should be passed as parameters at runtime.

I can see how to set the two parameters at runtime, but how do I set method=solcast if Iā€™m using emhass-addon? Do I put this into the configuration file?

If I am using the mpc optimisation and calling frequently will each optimisation result in another call to the SolCast API, noting that service has a limited number of calls allowed in each 24 hour period?

When providing the Solcast parameters at runtime the method will be automatically changed internally, so no need to do anything else.
Yes for each optimization call the Solcast API will be used. So I understand that this should be considered and adapt the optimization time steps accordingly

1 Like

Thanks for the input above about the configuration. Iā€™m still in the process of configuration.

Iā€™m loading the solar forecast from Solcast manually using an own script and adding storing that in a csv file. I have two sites setup that need to be combined and I could only add one Solcast site id. I have confirmed that there is a CSV file in the /data folder with the forecast data.

There is also a scipt running that downloads the electricity prices from my electricity provider and stores that in a CSV file.

The configuration I have now is this:

optim_conf:
  - load_cost_forecast_method: 'csv' # options are 'hp_hc_periods' for peak and non-peak hours contracts and 'csv' to load custom cost from CSV file
  - list_hp_periods: # list of different tariff periods (only needed if load_cost_forecast_method='hp_hc_periods')
    - period_hp_1:
      - start: '02:54'
      - end: '15:24'
    - period_hp_2:
      - start: '17:24'
      - end: '20:24'

When I trigger a optimization Iā€™m getting the following error in the docker console:

[2022-10-01 11:25:58,714] INFO in command_line: Setting up needed data
[2022-10-01 11:25:58,715] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2525, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1822, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1820, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1796, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "src/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.8/site-packages/emhass-0.3.19-py3.8.egg/emhass/command_line.py", line 42, in set_input_data_dict
    retrieve_hass_conf, optim_conf, plant_conf = utils.get_yaml_parse(config_path, params=params)
  File "/usr/local/lib/python3.8/site-packages/emhass-0.3.19-py3.8.egg/emhass/utils.py", line 239, in get_yaml_parse
    optim_conf['list_hp_periods'] = dict((key,d[key]) for d in optim_conf['list_hp_periods'] for key in d)
KeyError: 'list_hp_periods'

In the /data folder I have these CSV files:

  • data_load_cost_forecast.csv
  • data_load_forecast.csv
  • data_prod_price_forecast.csv
  • data_pv_power_forecast.csv
  • data_weather_forecast.csv

Some content from data_pv_power_forecast.csv

2022-10-02 06:00:00+02:00,0
2022-10-02 06:30:00+02:00,0
2022-10-02 07:00:00+02:00,0
2022-10-02 07:30:00+02:00,0.048100000000000004
2022-10-02 08:00:00+02:00,0.31379999999999997
2022-10-02 08:30:00+02:00,0.8818999999999999
2022-10-02 09:00:00+02:00,1.6953
2022-10-02 09:30:00+02:00,2.6088
2022-10-02 10:00:00+02:00,3.5980999999999996

Some content from data_prod_price_forecast.csv

2022-10-01 20:00:00+02:00,1.0199
2022-10-01 21:00:00+02:00,0.9753
2022-10-01 22:00:00+02:00,1.0066
2022-10-01 23:00:00+02:00,0.998
2022-10-02 00:00:00+02:00,0.9714

Based on the comment # list of different tariff periods (only needed if load_cost_forecast_method='hp_hc_periods') after list_hp_periods this config option should not be needed. Iā€™ve tried commenting that line and still I get the KeyError about list_hp_periods.

I havenā€™t changed whatā€™s in list_hp_periods, itā€™s still the same from the default config that came with the download of Emhass.

Why is it throwing an error on this key and how can I fix that?

Please open a new issue here: Issues Ā· davidusb-geek/emhass Ā· GitHub
And provide some information about your setup. Are you using the add-on or the standalone docker install?