EMHASS: An Energy Management for Home Assistant

[Patraff (Patrick Raftery)]

David
EMHASS
Options
The Home Assistant instance URL
empty
Enter the URL of your Home Assistant instance. This defaults to empty. If using the supervisor you can leave this to the default empty value.
A Long-Lived Access Token
empty
This can be created from the Lovelace profile page. This defaults to empty. If using the supervisor you can leave this to the default empty value.
Define the type of cost function

profit

cost

self-consumption
This is optional and the options are profit (default), cost and self-consumption
The time step to resample retrieved data from hass
30
This parameter is given in minutes. It should not be defined too low or you will run into memory problems when defining the Linear Programming optimization. Defaults to 30.
The number of history days to retrieve for perfect-optim
2
We will retrieve data from now and up to historic_days_to_retrieve days. Defaults to 2. This will be used for the perfect-optim task.
Set the method for timestamp rounding
nearest
options are first, last and nearest.
Set the total PV production to sell
Set this parameter to true to consider that all the PV power produced is injected to the grid. No direct self-consumption. The default is false, for as system with direct self-consumption.
Linear programming solver
COIN_CMD
Set the name of the linear programming solver that will be used. Defaults to COIN_CMD. The options are PULP_CBC_CMD, GLPK_CMD and COIN_CMD.
The path to the linear programming solver
/usr/bin/cbc
Set the path to the LP solver. Defaults to /usr/bin/cbc.
Set no charge from grid condition
Set this to true if you want to forbidden to charge the battery from the grid. The battery will only be charged from excess PV.
The Home Assistant sensor for PV power production
sensor.power_photovoltaics
This is the name of the photovoltaic produced power sensor in Watts from Home Assistant. For example sensor.power_photovoltaics.
The Home Assistant sensor for the load power without the deferrable loads
sensor.power_load_no_var_loads
The name of the household power consumption sensor in Watts from Home Assistant. The deferrable loads that we will want to include in the optimization problem should be substracted from this sensor in HASS. For example sensor.power_load_no_var_loads.
The number of deferrable loads
2
Define the number of deferrable loads to consider. Defaults to 2.
The nominal power for each deferrable load in Watts
1
2
3

• nominal_power_of_deferrable_loads: 3000
• nominal_power_of_deferrable_loads: 750

This is a list of elements, the number of elements should be consistent with the number of deferrable loads defined before.
The total number of hours that each deferrable load should operate
1
2
3

• operating_hours_of_each_deferrable_load: 5
• operating_hours_of_each_deferrable_load: 8

A list of elements.
This is a list of start hours of peak hours periods
1
2
3

• peak_hours_periods_start_hours: “02:54”
• peak_hours_periods_start_hours: “17:24”

A list of hours in 24h HH:MM format.
This is a list of end hours of peak hours periods
1
2
3

• peak_hours_periods_end_hours: “15:24”
• peak_hours_periods_end_hours: “20:24”

A list of hours in 24h HH:MM format.
A list to define if we should treat each deferrable load as a semi-continuous variable
1
2
3

• treat_deferrable_load_as_semi_cont: true
• treat_deferrable_load_as_semi_cont: true

Semi-continuous variables are variables that can take either their nominal value or zero.
The cost of the electrical energy during peak hours
0.0032
The cost of the electrical energy from the grid during peak hours in €/kWh. Defaults to 0.1907.
The cost of the electrical energy during non-peak hours
0.0027
The cost of the electrical energy from the grid during non-peak hours in €/kWh. Defaults to 0.1419.
The paid price for energy injected to the grid
0.065
The paid price for energy injected to the grid from excedent PV production in €/kWh. Defaults to 0.065.
The maximum power from the grid
9000
The maximum power that can be supplied by the utility grid in Watts. Defaults to 9000.
A list of the PV module model
1
2

• pv_module_model: CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M

For example ‘CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M’. This parameter can be a list of strings to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270). When finding the correct model for your installation remember to replace all the special characters in the model name by ‘_’.
A list of the PV inverter model
1
2

• pv_inverter_model: Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_

For example ‘Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_’. This parameter can be a list of strings to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270). When finding the correct model for your installation remember to replace all the special characters in the model name by ‘_’.
The tilt angle of your solar panels
1
2

• surface_tilt: 30

This is a value between 0 and 90. Defaults to 30. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270).
The azimuth of your PV installation
1
2

• surface_azimuth: 205

This is a value between 0 and 360. Defaults to 205. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270).
The number of modules per string
1
2

• modules_per_string: 16

Defaults to 16. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array and one west-facing array.
The number of used strings per inverter
1
2

• strings_per_inverter: 1

Defaults to 1. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array and one west-facing array.
Set if a battery is present
Set to True if we should consider an energy storage device such as a Li-Ion battery. Defaults to False.
The maximum discharge power in Watts
6000
Defaults to 1000.
The maximum charge power in Watts
13000
Defaults to 1000.
The discharge efficiency
0
Defaults to 0.95.
The charge efficiency
0.95
Defaults to 0.95.
The total capacity of the battery stack in Wh
13500
Defaults to 5000.
The minimun allowable battery state of charge
0.3
Defaults to 0.3.
The maximum allowable battery state of charge
0.9
Defaults to 0.9.
The desired battery state of charge at the end of each optimization cycle
0.6
Defaults to 0.6.
Network
Change the ports on your host that are exposed by the add-on

5000
5000/tcp
5000/tcp

and first run log

s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
web_server INFO  Launching the emhass webserver at: http://0.0.0.0:5000
web_server INFO  Home Assistant data fetch will be performed using url: http://supervisor/core/api
web_server INFO  The data path is: /share
web_server INFO  Using core emhass version: 0.4.3
waitress   INFO  Serving on http://0.0.0.0:5000```

Pat

[Patraff (Patrick Raftery)]

David, Sorry I’ll repost the config

Options
The Home Assistant instance URL
empty
Enter the URL of your Home Assistant instance. This defaults to empty. If using the supervisor you can leave this to the default empty value.
A Long-Lived Access Token
empty
This can be created from the Lovelace profile page. This defaults to empty. If using the supervisor you can leave this to the default empty value.
Define the type of cost function

profit

cost

self-consumption
This is optional and the options are profit (default), cost and self-consumption
The time step to resample retrieved data from hass
30
This parameter is given in minutes. It should not be defined too low or you will run into memory problems when defining the Linear Programming optimization. Defaults to 30.
The number of history days to retrieve for perfect-optim
2
We will retrieve data from now and up to historic_days_to_retrieve days. Defaults to 2. This will be used for the perfect-optim task.
Set the method for timestamp rounding
nearest
options are first, last and nearest.
Set the total PV production to sell
Set this parameter to true to consider that all the PV power produced is injected to the grid. No direct self-consumption. The default is false, for as system with direct self-consumption.
Linear programming solver
COIN_CMD
Set the name of the linear programming solver that will be used. Defaults to COIN_CMD. The options are PULP_CBC_CMD, GLPK_CMD and COIN_CMD.
The path to the linear programming solver
/usr/bin/cbc
Set the path to the LP solver. Defaults to /usr/bin/cbc.
Set no charge from grid condition
Set this to true if you want to forbidden to charge the battery from the grid. The battery will only be charged from excess PV.
The Home Assistant sensor for PV power production
sensor.power_photovoltaics
This is the name of the photovoltaic produced power sensor in Watts from Home Assistant. For example sensor.power_photovoltaics.
The Home Assistant sensor for the load power without the deferrable loads
sensor.power_load_no_var_loads
The name of the household power consumption sensor in Watts from Home Assistant. The deferrable loads that we will want to include in the optimization problem should be substracted from this sensor in HASS. For example sensor.power_load_no_var_loads.
The number of deferrable loads
2
Define the number of deferrable loads to consider. Defaults to 2.
The nominal power for each deferrable load in Watts
1
2
3
- nominal_power_of_deferrable_loads: 3000
- nominal_power_of_deferrable_loads: 750

This is a list of elements, the number of elements should be consistent with the number of deferrable loads defined before.
The total number of hours that each deferrable load should operate
1
2
3
- operating_hours_of_each_deferrable_load: 5
- operating_hours_of_each_deferrable_load: 8

A list of elements.
This is a list of start hours of peak hours periods
1
2
3
- peak_hours_periods_start_hours: "02:54"
- peak_hours_periods_start_hours: "17:24"

A list of hours in 24h HH:MM format.
This is a list of end hours of peak hours periods
1
2
3
- peak_hours_periods_end_hours: "15:24"
- peak_hours_periods_end_hours: "20:24"

A list of hours in 24h HH:MM format.
A list to define if we should treat each deferrable load as a semi-continuous variable
1
2
3
- treat_deferrable_load_as_semi_cont: true
- treat_deferrable_load_as_semi_cont: true

Semi-continuous variables are variables that can take either their nominal value or zero.
The cost of the electrical energy during peak hours
0.0032
The cost of the electrical energy from the grid during peak hours in €/kWh. Defaults to 0.1907.
The cost of the electrical energy during non-peak hours
0.0027
The cost of the electrical energy from the grid during non-peak hours in €/kWh. Defaults to 0.1419.
The paid price for energy injected to the grid
0.065
The paid price for energy injected to the grid from excedent PV production in €/kWh. Defaults to 0.065.
The maximum power from the grid
9000
The maximum power that can be supplied by the utility grid in Watts. Defaults to 9000.
A list of the PV module model
1
2
- pv_module_model: CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M

For example 'CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M'. This parameter can be a list of strings to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270). When finding the correct model for your installation remember to replace all the special characters in the model name by '_'.
A list of the PV inverter model
1
2
- pv_inverter_model: Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_

For example 'Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_'. This parameter can be a list of strings to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270). When finding the correct model for your installation remember to replace all the special characters in the model name by '_'.
The tilt angle of your solar panels
1
2
- surface_tilt: 30

This is a value between 0 and 90. Defaults to 30. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270).
The azimuth of your PV installation
1
2
- surface_azimuth: 205

This is a value between 0 and 360. Defaults to 205. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array (azimuth=90) and one west-facing array (azimuth=270).
The number of modules per string
1
2
- modules_per_string: 16

Defaults to 16. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array and one west-facing array.
The number of used strings per inverter
1
2
- strings_per_inverter: 1

Defaults to 1. This parameter can be a list of integers to enable the simulation of mixed orientation systems, for example one east-facing array and one west-facing array.
Set if a battery is present
Set to True if we should consider an energy storage device such as a Li-Ion battery. Defaults to False.
The maximum discharge power in Watts
6000
Defaults to 1000.
The maximum charge power in Watts
13000
Defaults to 1000.
The discharge efficiency
0
Defaults to 0.95.
The charge efficiency
0.95
Defaults to 0.95.
The total capacity of the battery stack in Wh
13500
Defaults to 5000.
The minimun allowable battery state of charge
0.3
Defaults to 0.3.
The maximum allowable battery state of charge
0.9
Defaults to 0.9.
The desired battery state of charge at the end of each optimization cycle
0.6
Defaults to 0.6.
Network
Change the ports on your host that are exposed by the add-on

5000
5000/tcp
5000/tcp```

Pat

[davidusb]

You can just share your yaml so that it can be readable.
I don’t see anything wrong. So the error arises everytime that you try to launch the day-ahead optimization task? or it was just one time. Try it now to see if you have the error.

[torstein]

root@portainer:/docker/emhass# python3 test.py

HTTP Status Code: 500
Response Text: <!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>

root@portainer:/docker/emhass# cat test.py

import requests

# Set the URL of the API endpoint
url = "http://10.10.10.80:5001/action/forecast-model-fit"

# Set the data payload to an empty JSON object
runtimeparams = {
    "days_to_retrieve": 1,
    "model_type": "load_forecast",
    "var_model": "sensor.power_load_no_var_loads",
    "sklearn_model": "KNeighborsRegressor",
    "num_lags": 48,
    "split_date_delta": '48h',
    "perform_backtest": False
}

# Set the content type header to application/json
headers = {"Content-Type": "application/json"}

# Send the HTTP POST request to the API endpoint
response = requests.post(url, json=runtimeparams, headers=headers)

# Print the HTTP status code and response text
print("HTTP Status Code:", response.status_code)
print("Response Text:", response.text)

[davidusb]

@torstein

http://10.10.10.80:5001/

If you go to this address on your browser just click on the fit button. I added these in the latests versions. It allows to test this new feature and visualize the quality of the fitted machine learning model.

I guess that the problem with your test.py is that address. Try to execute that outside the container and check the validity of the address first, by pinging it or by checking it on your browser.

[torstein]

oot@portainer:/docker/emhass# curl -i -H "Content-Type:application/json" -X POST -d '{"model_type": "load_forecast"}' http://10.10.10.80:5001/action/forecast-model-predict
HTTP/1.1 500 INTERNAL SERVER ERROR
Content-Length: 265
Content-Type: text/html; charset=utf-8
Date: Fri, 10 Mar 2023 09:20:38 GMT
Server: waitress

<!doctype html>
<html lang=en>
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
root@portainer:/docker/emhass# curl  http://10.10.10.80:5001
<!DOCTYPE html>
<html>
    <head>
        <script src="http://ajax.googleapis.com/ajax/libs/jquery/2.0.0/jquery.min.js"></script>
        <title>EMHASS: Energy Management Optimization for Home Assistant</title>
        <link rel="stylesheet" type="text/css" href="../static/style.css">
    </head>
    <body>
        <center>
            <h2 style="font-family: Helvetica;color: darkblue;">EMHASS: Energy Management Optimization for Home Assistant</h2>
        </center>
        <div class="center">
            <div align="center" class="form">
                <form action="" method="get">
                    <h4 style="font-family: Helvetica;color: darkblue;">Use the buttons below to manually  launch different optimization tasks</h4>
                    <button type="button" id="perfect-optimization" class="button button1" >Perfect Optimization</button>
                    <button type="button" id="dayahead-optimization" class="button button2" >Day-ahead Optimization</button>
                    <!-- <button type="button" id="naive-mpc-optimization" class="button button3" >Naive MPC Optimization</button> -->
                    <h4 style="font-family: Helvetica;color: darkblue;">Use the button below to publish the optimized variables at the current timestamp</h4>
                    <button type="button" id="publish-data" class="button button1" >Publish Optimization Results</button>
                    <br>
                </form>
                <br><br>
            </div>
        </div>

        <footer class="footer">
            <center>
                <p style="font-family: Helvetica;color: darkblue;">&copy; MIT License | Copyright (c) 2021-2022 David HERNANDEZ</p>
            </center>
        </footer>
    </body>
    <script>
        document.getElementById('perfect-optimization').addEventListener('click', function() {
            var url = "/action/perfect-optim"
            var settings = {
            "async": true,
            "crossDomain": true,
            "url": url,
            "method": "POST",
            "headers": {
                'Content-Type':'application/json'
            },
            "dataType": 'json',
            "data": '{}'
            }
            $.ajax(settings).done(function (response) {
            });
        });
        document.getElementById('dayahead-optimization').addEventListener('click', function() {
            var url = "/action/dayahead-optim"
            var settings = {
            "async": true,
            "crossDomain": true,
            "url": url,
            "method": "POST",
            "headers": {
                'Content-Type':'application/json'
            },
            "dataType": 'json',
            "data": '{}'
            }
            $.ajax(settings).done(function (response) {
            });
        });
        // document.getElementById('naive-mpc-optimization').addEventListener('click', function() {
        //     var url = "/action/naive-mpc-optim"
        //     var settings = {
        //     "async": true,
        //     "crossDomain": true,
        //     "url": url,
        //     "method": "POST",
        //     "headers": {
        //         'Content-Type':'application/json'
        //     },
        //     "dataType": 'json',
        //     "data": '{}'
        //     }
        //     $.ajax(settings).done(function (response) {
        //     });
        // });
        document.getElementById('publish-data').addEventListener('click', function() {
            var url = "/action/publish-data"
            var settings = {
            "async": true,
            "crossDomain": true,
            "url": url,
            "method": "POST",
            "headers": {
                'Content-Type':'application/json'
            },
            "dataType": 'json',
            "data": '{}'
            }
            $.ajax(settings).done(function (response) {
            });
        });
    </script>
</html>

[torstein]

If I try with day ahead optimization in the gui:

eb_server INFO  Setting up needed data
web_server INFO  Retrieving weather forecast data using method = scrapper
web_server INFO  Retrieving data from hass for load forecast using method = mlforecaster
web_server INFO  Retrieve hass get data method initiated...
web_server ERROR Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "src/emhass/web_server.py", line 150, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.8/site-packages/emhass-0.4.3-py3.8.egg/emhass/command_line.py", line 91, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.8/site-packages/emhass-0.4.3-py3.8.egg/emhass/forecast.py", line 600, in get_load_forecast
    model_type = self.params['passed_data']['model_type']
KeyError: 'model_type'

[davidusb]

@torstein

http://10.10.10.80:5001

Again that address seems not valid and you don’t have the latest emhass version there. Update.

[davidusb]

@torstein

    model_type = self.params['passed_data']['model_type']
KeyError: 'model_type'

Oh yes, there is a bug. It can’t possibly work as it right now when using method=mlforecaster and dayahead-optim. It will be fixed. For now you can just test using forecast-model-fit, forecast-model-predict and forecast-model-tune.

[torstein]

@torstein

http://10.10.10.80:5001

It is, look in my previus posst to curl http://10.10.10.80:5001 that works.

EDit: updated the docker again and seems like I was not on the newest version.

[Patraff (Patrick Raftery)]

David,
All recent updates applied. 0.3.5
Still nothing?
Following a restart I just click on an optimization - day ahead.
Then I click on publish results. This is the log following those two actions.

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
2023-03-11 08:21:25,971 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2023-03-11 08:21:25,971 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2023-03-11 08:21:25,972 - web_server - INFO - The data path is: /share
2023-03-11 08:21:25,976 - web_server - INFO - Using core emhass version: 0.4.5
waitress   INFO  Serving on http://0.0.0.0:5000
2023-03-11 08:26:21,381 - web_server - INFO - EMHASS server online, serving index.html...
2023-03-11 08:26:21,398 - web_server - WARNING - The data container dictionary is empty... Please launch an optimization task
2023-03-11 08:26:24,376 - web_server - INFO - Setting up needed data
2023-03-11 08:26:24,528 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2023-03-11 08:26:26,322 - web_server - INFO - Setting up needed data
2023-03-11 08:26:26,327 - web_server - INFO - Retrieving weather forecast data using method = scrapper
2023-03-11 08:26:29,737 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-03-11 08:26:29,745 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 08:26:29,818 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 08:26:29,824 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 08:26:29,830 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 585, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
2023-03-11 08:26:31,453 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-03-11 08:26:31,455 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 08:26:31,514 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 08:26:31,514 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 08:26:31,515 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 585, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
2023-03-11 08:27:01,078 - web_server - INFO - Setting up needed data
2023-03-11 08:27:01,100 - web_server - INFO -  >> Publishing data...
2023-03-11 08:27:01,101 - web_server - INFO - Publishing data to HASS instance
2023-03-11 08:27:01,102 - web_server - ERROR - File not found error, run an optimization task first.

[davidusb]

@davidusb

@torstein

>     model_type = self.params['passed_data']['model_type']
KeyError: 'model_type'
> ```

Oh yes, there is a bug. It can’t possibly work as it right now when using method=mlforecaster and dayahead-optim. It will be fixed. For now you can just test using forecast-model-fit, forecast-model-predict and forecast-model-tune.

This is fixed!

[davidusb]

@Patraff

David,
All recent updates applied. 0.3.5
Still nothing?

Double check this variable of your Home Assistant instance: sensor.power_load_no_var_loads
It has correct history? The data is being saved by the Home Assistant recorder? Check that the name is correct.

[davidusb]

A procedure to use the new machine learning feature for load power forecasting:

1.- First fit the default model with the default parameters by going to the webui and using the new buttons there:

2.- After clicking on “ML forecast model fit” you can check the logs to see the fit metrics results. In my case the logs show:

And refreshing the webui now shows the graph with the results:

Zooming in:

This used the default parameters which are these:

runtimeparams = {
    "days_to_retrieve": 30,
    "model_type": "load_forecast",
    "var_model": "sensor.power_load_no_var_loads",
    "sklearn_model": "KNeighborsRegressor",
    "num_lags": 48,
    "split_date_delta": '48h',
    "perform_backtest": False
}

But you can change any of these parameters by using a data disctionary during the curl call. Like this:

curl -i -H "Content-Type:application/json" -X POST -d '{"days_to_retrieve": 150}' http://localhost:5000/action/forecast-model-fit

Try to provide as much data as you can.

3.- The ML forecast model tune button can be used to launch a hyperparameter optimization using bayesian optimization. Again check the logs, in my case it shows:

We have better metric and we used backtest to validate this.
Refreshing the webui will give us the graphic results after this model tuning:

4.- If we are satisfied with these results the last step is to switch to this model in the add-on configuration pane by changing the load forecast method to mlforecaster:

5.- You can automate all this by defining the curl commands using the shell_command service on Home Assistant. Check the complete documentation of this feature here: The machine learning forecaster — emhass 0.4.4 documentation
Refit the model often (once a week?) to update to any changes on your consumption dynamic. Careful with the tuning routine that can be computation intensive for RPi devices. Limit the days_to_retrieve parameter if the optimization takes too long.

[Patraff (Patrick Raftery)]

David,
that sensor doesn’t show up in entities?
Pat

[davidusb]

Ok so there is your problem. This sensor need to exists and it should have at least two days of recorded data otherwise you will have the error shown.
Define a new template sensor and give it that name. The data for the sensor should be your load power consumption minus your deferrable/controllable loads that you want to optimize.
Read this: Intro / Quick start — emhass 0.4.4 documentation

[markpurcell]

I seem to be failing at the first step.

‘ML forecast model fit’ button fails:

2023-03-11 17:57:43,144 - web_server - INFO - Setting up needed data
2023-03-11 17:57:43,154 - web_server - INFO - Retrieve hass get data method initiated...
2023-03-11 17:57:43,190 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-03-11 17:57:43,191 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-03-11 17:57:43,191 - web_server - ERROR - Exception on /action/forecast-model-fit [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 146, in set_input_data_dict
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 147, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment

[davidusb]

If you retry you have the same error? Is it possible that this is conflicting with other data retrieve that you are doing at the same time? As you are doing high frequency MPC if I recall well.
Your sensor name is the same as the default name? Do you have at least 30 days of data for that sensor?

[Patraff (Patrick Raftery)]

Thanks David,
I tried reading the docs but soon got lost. I guess I will have to spend a bit more time in there.
thanks
Pat

[markpurcell]

Ah, standard recorder only has 7 days data.

Looks like I’ll need to extend that.