EMHASS: An Energy Management for Home Assistant

markpurcell · May 20, 2024, 3:40am

I have an input helper for the datetime.

Then use a template sensor to continuously calculate the number of 30 minute time slots until the input date time.

{% set current_time = now() %}
{% set sensor_time_str = states('input_datetime.emhass_p_deferable_2_end_time') %}
{% set sensor_time = current_time.replace(hour=0, minute=0, second=0, microsecond=0) + timedelta(hours=(sensor_time_str.split(':')[0]|int), minutes=(sensor_time_str.split(':')[1]|int)) %}
{% if sensor_time <= current_time %}
  {% set sensor_time = sensor_time + timedelta(days=1) %}
{% endif %}
{% set time_difference = sensor_time - current_time %}
{{ time_difference.total_seconds() / 3600 * 2 }}

I then include these timeslots as the def_end_timestep parameter in my MPC payload:

        "def_end_timestep":[0, 0, 
                            {{max(0,states('sensor.emhass_deferrable2_end_timeslots')|int(0))}}, 
                            0, 0, 
                            {{max(0,states('sensor.emhass_p_deferable_5_end_timeslots')|int(0))}}],

Johan71 · May 20, 2024, 4:27am

I’ve tried your variation, but can’t get it to work. I have 60 min timesteps, how should I think then?

markpurcell · May 20, 2024, 4:30am

What do you get when you enter your template into the developer tools?

Johan71 · May 20, 2024, 4:32am

Johan71 · May 20, 2024, 4:44am

Not only that, my timesteps will be fewer until about 13:30 when new prices are released for the coming day. Doesn’t it affect the hours negatively

markpurcell · May 20, 2024, 4:45am

Great, so that states that the end time is 10 timesteps away, is that what you desire, or do you actually want the endtime to be 5 timesteps away?

If the latter. remove the *2 in the last line and it should report 5.

Johan71 · May 20, 2024, 3:43pm

Something is fishy, every time I connect the car and try according to your examples, I get that the system is Infeasible

2024-05-20 17:25:00,248 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.5, 0.58, 0.61, 0.61, 0.56, 0.44, 0.26, 0.24, 0.23, 0.21, 0.2, 0.22, 0.3, 0.59, 0.83, 0.91, 0.63, 0.56, 0.48, 0.42, 0.38, 0.33, 0.42, 0.48], 'prod_price_forecast': [0.5, 0.58, 0.61, 0.61, 0.56, 0.44, 0.26, 0.24, 0.23, 0.21, 0.2, 0.22, 0.3, 0.59, 0.83, 0.91, 0.63, 0.56, 0.48, 0.42, 0.38, 0.33, 0.42, 0.48], 'prediction_horizon': 24, 'pv_power_forecast': [5138, 4006, 2333, 645, 14, 0, 0, 0, 0, 0, 0, 34, 129, 163, 1088, 2604, 4294, 5691, 6835, 7585, 7962, 7974, 7569, 6788, 5634, 4184, 2511, 815, 17, 0, 0], 'load_power_forecast': [806, 700, 700, 700, 800, 800, 600, 600, 700, 600, 500, 500, 400, 600, 1700, 600, 600, 600, 600, 600, 600, 600, 600, 600, 600, 700, 700, 400, 400, 400, 600, 600, 600, 600, 600, 600, 600, 700, 600, 600, 700, 600, 600, 600, 600, 700, 600, 600], 'soc_init': 1.0, 'soc_final': 1.0, 'delta_forecast': 2, 'def_total_hours': [0.0, 0, 10], 'def_end_timestep': [0, 0, 12]}
2024-05-20 17:25:00,248 - web_server - INFO -  >> Setting input data dict
2024-05-20 17:25:00,248 - web_server - INFO - Setting up needed data
2024-05-20 17:25:00,250 - web_server - INFO - Retrieve hass get data method initiated...
2024-05-20 17:25:00,719 - web_server - INFO - Retrieving weather forecast data using method = list
2024-05-20 17:25:00,723 - web_server - INFO -  >> Performing naive MPC optimization...
2024-05-20 17:25:00,723 - web_server - INFO - Performing naive MPC optimization
2024-05-20 17:25:00,733 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-05-20 17:25:00,740 - web_server - WARNING - Deferrable load 2 : Available timeframe is shorter than the specified number of hours to operate. Optimization will fail.
2024-05-20 17:25:00,780 - web_server - INFO - Status: Infeasible
2024-05-20 17:25:00,780 - web_server - INFO - Total value of the Cost function = 27.15
2024-05-20 17:25:06,028 - web_server - INFO - Passed runtime parameters: {}
2024-05-20 17:25:06,029 - web_server - INFO -  >> Setting input data dict
2024-05-20 17:25:06,029 - web_server - INFO - Setting up needed data
2024-05-20 17:25:06,030 - web_server - INFO -  >> Publishing data...
2024-05-20 17:25:06,030 - web_server - INFO - Publishing data to HASS instance
2024-05-20 17:25:06,042 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = 5401.74
2024-05-20 17:25:06,050 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 782.46
2024-05-20 17:25:06,058 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2024-05-20 17:25:06,065 - web_server - INFO - Successfully posted to sensor.p_deferrable1 = 0.0
2024-05-20 17:25:06,072 - web_server - INFO - Successfully posted to sensor.p_deferrable2 = 0.0
2024-05-20 17:25:06,079 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 0.0
2024-05-20 17:25:06,086 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 100.0
2024-05-20 17:25:06,094 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = -4619.29
2024-05-20 17:25:06,101 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = 30.13
2024-05-20 17:25:06,107 - web_server - INFO - Successfully posted to sensor.optim_status = Infeasible
2024-05-20 17:25:06,114 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.5
2024-05-20 17:25:06,121 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.5``

And when the car is disconnected.

024-05-20 17:35:00,258 - web_server - INFO - Passed runtime parameters: {'load_cost_forecast': [0.5, 0.58, 0.61, 0.61, 0.56, 0.44, 0.26, 0.24, 0.23, 0.21, 0.2, 0.22, 0.3, 0.59, 0.83, 0.91, 0.63, 0.56, 0.48, 0.42, 0.38, 0.33, 0.42, 0.48], 'prod_price_forecast': [0.5, 0.58, 0.61, 0.61, 0.56, 0.44, 0.26, 0.24, 0.23, 0.21, 0.2, 0.22, 0.3, 0.59, 0.83, 0.91, 0.63, 0.56, 0.48, 0.42, 0.38, 0.33, 0.42, 0.48], 'prediction_horizon': 24, 'pv_power_forecast': [5138, 4006, 2333, 645, 14, 0, 0, 0, 0, 0, 0, 34, 129, 163, 1088, 2604, 4294, 5691, 6835, 7585, 7962, 7974, 7569, 6788, 5634, 4184, 2511, 815, 17, 0, 0], 'load_power_forecast': [1711, 700, 700, 800, 800, 600, 600, 700, 600, 500, 500, 400, 600, 1700, 600, 600, 600, 600, 600, 600, 600, 600, 600, 600, 700, 700, 400, 400, 400, 600, 600, 600, 600, 600, 600, 600, 700, 600, 600, 700, 600, 600, 600, 600, 700, 600, 600, 800], 'soc_init': 1.0, 'soc_final': 1.0, 'delta_forecast': 2, 'def_total_hours': [0.0, 0, 10], 'def_end_timestep': [0, 0, 0]}
2024-05-20 17:35:00,259 - web_server - INFO -  >> Setting input data dict
2024-05-20 17:35:00,259 - web_server - INFO - Setting up needed data
2024-05-20 17:35:00,265 - web_server - INFO - Retrieve hass get data method initiated...
2024-05-20 17:35:00,750 - web_server - INFO - Retrieving weather forecast data using method = list
2024-05-20 17:35:00,753 - web_server - INFO -  >> Performing naive MPC optimization...
2024-05-20 17:35:00,753 - web_server - INFO - Performing naive MPC optimization
2024-05-20 17:35:00,764 - web_server - INFO - Perform an iteration of a naive MPC controller
2024-05-20 17:35:00,844 - web_server - INFO - Status: Optimal
2024-05-20 17:35:00,844 - web_server - INFO - Total value of the Cost function = 24.15
2024-05-20 17:35:06,065 - web_server - INFO - Passed runtime parameters: {}
2024-05-20 17:35:06,065 - web_server - INFO -  >> Setting input data dict
2024-05-20 17:35:06,065 - web_server - INFO - Setting up needed data
2024-05-20 17:35:06,067 - web_server - INFO -  >> Publishing data...
2024-05-20 17:35:06,067 - web_server - INFO - Publishing data to HASS instance
2024-05-20 17:35:06,081 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = 5343.18
2024-05-20 17:35:06,089 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 1253.3
2024-05-20 17:35:06,101 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2024-05-20 17:35:06,109 - web_server - INFO - Successfully posted to sensor.p_deferrable1 = 0.0
2024-05-20 17:35:06,118 - web_server - INFO - Successfully posted to sensor.p_deferrable2 = 0.0
2024-05-20 17:35:06,126 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 0.0
2024-05-20 17:35:06,135 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 100.0
2024-05-20 17:35:06,143 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = -4089.88
2024-05-20 17:35:06,151 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = 27.13
2024-05-20 17:35:06,160 - web_server - INFO - Successfully posted to sensor.optim_status = Optimal
2024-05-20 17:35:06,169 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.5
2024-05-20 17:35:06,176 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.5

JoshC1994 · May 21, 2024, 11:26am

Hi gents, are you able to give me some guidance. I’ve pushed some updates and broken my code, I cant for the life of me get my syntax right, all the data is showing in HA template tool , but i get a log error when running the MPC.

Here’s my code,

shell_command:
  ##EMHASS related##
  ####
  post_mpc_optim: >
    curl -i -H "Content-Type: application/json" -X POST -d '{
      "load_cost_forecast":{{(([states('sensor.amber_general_price')|float(0)] +state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:48])}},
      "prod_price_forecast":{{(([states('sensor.amber_feed_in_price')|float(0)] +state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)[:48]) }},
      "pv_power_forecast": {{([states('sensor.combined_total_pv')|int(0)] +state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list +state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list)| tojson}},
      "prediction_horizon":{{min(48,(state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)}},
      "soc_init":{{states('sensor.inv001_battery_state_of_charge')|float(0)/100}},
      "def_total_hours":[6, 0]
    }' http://localhost:5000/action/naive-mpc-optim
  ###
  publish_data: >
    curl -i -H "Content-Type: application/json" -X POST -d '{
    }' http://localhost:5000/action/publish-data
  ###
  post_emhass_forecast: >
    curl -i -H "Content-Type: application/json" -X POST -d '{
      "prod_price_forecast":{{(state_attr("sensor.amber_feed_in_forecast", "forecasts")|map(attribute="per_kwh")|list)}},
      "pv_power_forecast":{{states("sensor.solcast_pv_forecast_forecast_remaining_today")}},
      "load_cost_forecast":{{(state_attr("sensor.amber_general_forecast", "forecasts") |map(attribute="per_kwh")|list)}}
    }' http://localhost:5000/action/dayahead-optim
########################################################################################################

Heres my Template return data

And here’s my error log.

Also my EMHASS is not making any sense, if it wasn’t already implied

bert1111 · May 22, 2024, 9:28am

Hi,

I made a new sensor for sensor.power_load_no_var_loads called sensor.house_power_consumption_less_deferrables trying to reduce errors.
When I change it I get this error. I already tried removing the files in the share folder but that doesn’t help.

INFO - web_server - Retrieve hass get data method initiated...
ERROR - web_server - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/base.py", line 3653, in get_loc
    return self._engine.get_loc(casted_key)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pandas/_libs/index.pyx", line 147, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/index.pyx", line 176, in pandas._libs.index.IndexEngine.get_loc
  File "pandas/_libs/hashtable_class_helper.pxi", line 7080, in pandas._libs.hashtable.PyObjectHashTable.get_item
  File "pandas/_libs/hashtable_class_helper.pxi", line 7088, in pandas._libs.hashtable.PyObjectHashTable.get_item
KeyError: 'sensor.power_load_no_var_loads'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 108, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 93, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 659, in get_load_forecast
    forecast_out = mlf.predict(data_last_window)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/machine_learning_forecaster.py", line 207, in predict
    last_window=data_last_window[self.var_model],
                ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 3761, in __getitem__
    indexer = self.columns.get_loc(key)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/base.py", line 3655, in get_loc
    raise KeyError(key) from err
KeyError: 'sensor.power_load_no_var_loads'

kingphil123 · May 22, 2024, 2:07pm

Hi All! Ive been keeping my eye on and off EMHASS and looks like people are not getting loads of value out of the system and I want to get a slice of that cake! I have installed and set things up, I have got the standalone docker installed and the configuration in place. I have setup the Shell Commands in Home assistant to publish data for my Octopus Flux Import / Export Tariff and i can see charts being generated in the EMHASS Webpage…

What do i do now!? I dont really understand how i interpolate the graphs and and how i turn the output into actionable automations in Home Assistant. Im assuming that the EMHASS sensors are meant to then drive adjustments in my Inverters / BMS integration? If so - what have other people done?

For reference I have a SolaX X1 G4 Hybrid (3kW) with 12kWh’s of Battery storage. 3kW PV Array.

Appreciate any support you can give, I have read the Docs multiple times and all the detail seem to end when you get your graphs and there is no detail about what to do next

davidusb · May 22, 2024, 3:08pm

If you look here: https://emhass.readthedocs.io/en/latest/intro.html#configuration-and-installation

You are at step number 5. What it says there is that now you need to define some automations to effectively link the sensor published by EMHASS to the switches controlling real loads on your home. There are examples there on how to do that.

Just for information, hybrid inverters are not supported but they will really really soon on the next release (< 1 week)

kingphil123 · May 22, 2024, 3:13pm

Thanks David! But for the moment, I haven’t set up and Deferrable loads, what I’m focused on now is my battery system - charge from the grid when its cheap and im low, discharge to the grid when the sell value is high and i dont need the energy for the rest of the day.

davidusb · May 22, 2024, 3:25pm

Got it. The principle is the same, you need to write automations to link the sensors related to to battery management to the control of your battery inverter. This is probably the hardest part and not possible to provide a unique automation blueprint as it will greatly vary from one inverter manufacturer to another.
There have been examples on this thread for specific inverter manufacturers.

markpurcell · May 22, 2024, 9:58pm

Here is my battery automation, it calls a series of scripts depending on what the state of p_batt_forecast is.

If you show us what controls your inverter has we maybe able to assist with which controls would work.

bert1111 · May 23, 2024, 5:14am

Does anyone know what the problem is?

Johan71 · May 23, 2024, 8:25am

Hi @markpurcell
When you use charge and discharge the battery, you change something on power or run on P Batt forecast power in your automations.

bert1111 · May 23, 2024, 9:34am

@davidusb I just updated emhass to the latest version and now I get this error.
Completely broken now.
Does anybody have the same problem?

2024-05-23 11:28:38,053 - web_server - INFO - Retrieving data from hass for load forecast using method = mlforecaster
2024-05-23 11:28:38,055 - web_server - INFO - Retrieve hass get data method initiated...
/usr/local/lib/python3.11/dist-packages/sklearn/base.py:376: InconsistentVersionWarning:

Trying to unpickle estimator KNeighborsRegressor from version 1.3.2 when using version 1.4.2. This might lead to breaking code or invalid results. Use at your own risk. For more info please refer to:
https://scikit-learn.org/stable/model_persistence.html#security-maintainability-limitations

2024-05-23 11:28:39,880 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1473, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 882, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 880, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 865, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 109, in action_call
    input_data_dict = set_input_data_dict(emhass_conf, costfun,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 100, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(
                      ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 673, in get_load_forecast
    forecast_out = mlf.predict(data_last_window)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/machine_learning_forecaster.py", line 206, in predict
    predictions = self.forecaster.predict(steps=self.lags_opt,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/skforecast/ForecasterAutoreg/ForecasterAutoreg.py", line 774, in predict
    window_size      = self.window_size_diff,
                       ^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'ForecasterAutoreg' object has no attribute 'window_size_diff'

markpurcell · May 23, 2024, 9:41am

What controls does your battery provide to Home Assistant?

For some batteries you can give the battery the desired SOC and it will charge/ discharge to that point. In this case use soc_forecast

Other batteries you can provide a charge/ discharge amount in Watt. In this case use p_batt_forecast.

For other batteries you can only specify charge or discharge, so check is p_batt_forecast is negative or positive and switch your battery into that mode.

This is my p_batt_forecast sensor, what does yours look like?

kingphil123 · May 23, 2024, 10:14am

Hey Mark! Thanks so much for helping with this! I have a Solax Inverter and Im using the SolaX Modbus custom integration. Not sure the best way to share what options i have available as there are 237 entities!! Best place to look would be the docs

RikBast · May 23, 2024, 1:24pm

Hi, I am trying to use forecast.solar (Public version) and to 1 call/hour. The pv_forecast at night cannot be correct (value of about 500W) so my battery wants to charge at night. See enclosed pictures, is that a forecast.solar issue you guys think?

Schermafbeelding 2024-05-23 151857

Schermafbeelding 2024-05-23 152404