EMHASS: An Energy Management for Home Assistant

markpurcell · April 23, 2023, 2:42am

David,

I have had some time to experiment with the ML module over the weekend.

I now have ML models for my buy/ sell prices, which is quite exciting.

Thanks for this extension.

2023-04-23 12:40:41,646 - web_server - INFO -  >> Performing a machine learning forecast model fit...
2023-04-23 12:40:41,649 - web_server - INFO - Performing a forecast model fit for price_forecast
2023-04-23 12:40:41,669 - web_server - INFO - Training a KNeighborsRegressor model
2023-04-23 12:40:41,720 - web_server - INFO - Elapsed time for model fit: 0.05129742622375488
2023-04-23 12:40:42,042 - web_server - INFO - Prediction R2 score of fitted model on test data: 0.4393118941478219
2023-04-23 12:40:44,704 - web_server - INFO - Setting up needed data
2023-04-23 12:40:44,711 - web_server - INFO - Retrieve hass get data method initiated...
2023-04-23 12:40:45,562 - web_server - INFO -  >> Performing a machine learning forecast model fit...
2023-04-23 12:40:45,565 - web_server - INFO - Performing a forecast model fit for price_forecast
2023-04-23 12:40:45,589 - web_server - INFO - Training a KNeighborsRegressor model
2023-04-23 12:40:45,610 - web_server - INFO - Elapsed time for model fit: 0.02150583267211914
2023-04-23 12:40:45,865 - web_server - INFO - Prediction R2 score of fitted model on test data: 0.4393118941478219
2023-04-23 12:40:45,872 - web_server - INFO - Performing simple backtesting of fitted model
2023-04-23 12:40:46,799 - web_server - INFO - Elapsed backtesting time: 0.9260625839233398
2023-04-23 12:40:46,799 - web_server - INFO - Backtest R2 score: 0.9048172716630719

davidusb · April 23, 2023, 8:25am

That’s really nice!
That was the goal of the custom ML model, you can actually build a forecast of whatever sensor you have in HA, provided that it have enough data and that it is stationary.
Prod price and load cost are good examples. What it would be nice now is that you compare those results with the real forecast obtained from your provider. I don’t expect that we can obtain much better results for prices or weather forecasts, because typically forecasts provider will be using very complex models. In the case of weather forecasts they are a combination of satellite and/or radar data along with deep neural networks and the such. Here we are using more simple (but efficient) ML models.
The real deal with this ML module was the need for a better load power forecast model. And it works really well for that.
One thing that I will add in the future is the possibility to pass exogenous data for training and prediction. For example train a model to forecast the load power based on the historical values of the load power itself and other additional data as the outdoor temperature, humidity, house occupancy, etc. The main issue to this is that you will have to provide future estimates of those exogenous data. For weather data those estimates are easy to obtain. But for other types of data such as occupancy you will have to build them yourself and in that case we can use the emhass custom ML model approach. That can be interesting to evaluate with the issue of error propagating along those different forecast models.

davidusb · April 23, 2023, 8:30am

Thanks for these inputs. I’m still evaluating if it worths the effort.

davidusb · April 23, 2023, 8:35am

This is very interesting and it is a great idea to try to standardize interfaces and communication protocols.
There is a similar effort in the multi physics simulation world using FMI/FMU models.

Unfortunately for this EFI effort it seems quite dead. The last update to the github was 6 years ago and I couldn’t find a lot of information and documentation.

Joel_DE · April 30, 2023, 9:21pm

This seems like a very powerful tool thanks for sharing.

I am on amber, but don’t yet have solar or a battery. I am wanting to setup rules (either in node red or this tool) to look at the forecast of power and do the following for example:

if someone turns on the dryer (I have smart power switch on it) and price is expensive - and will be much cheaper in the next 5 hours to turn it off and remind us to turn it on later.

Pretty much that. I also have ecebee turn off etc if it’s expensive but this app seems like it can be used for this even if i don’t have solar or battery is that true?

thank you

markpurcell · April 30, 2023, 11:26pm

Hi,

EMHASS works well even without a battery or solar system, as it can provide an optimized schedule based on price alone and then switch your devices according to that plan.

I also use Amber and have set up automations that trigger based on their dynamic pricing. For instance, I have rules in place for my dryer to switch on when Amber prices are low (green) and switch off when prices are high (yellow/red). It’s an efficient way to manage energy usage for a few devices.

However, if you’re looking to manage multiple devices, EMHASS is the way to go, as it can optimize the energy usage for all your devices simultaneously. This helps maximize savings and maintain a more sustainable household. I’ve found this combination of EMHASS and Amber to be incredibly useful in managing my home’s energy consumption effectively.

Joel_DE · April 30, 2023, 11:41pm

thanks so much. ill take a good look.

how do you turn your dryer on - i assume it doesn’t have touch panels? ours has on off and start touch panels but i can’t turn it on with the smart switch which sucks. but i can turn it off and then announce when to turn it on.

markpurcell · April 30, 2023, 11:55pm

My old dryer and dishwasher, if I start the cycle then turn off the smart switch, it will resume the cycle when the power comes back on.

My new heat pump dryer uses a lot less power 600W down from 2kW and has a home assistant remote start option:

smi · May 6, 2023, 6:55pm

Unfortunately I suddenly got some problems with EMHASS.
I installed the latest update for Home Assistant and since then, I get the error of an empty JSON.

My json:

{"pv_power_forecast":[0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,14.6,65.35000000000001,133.8,302.75,544.5,832.75,1142.45,1447.05,1686.75,1875.85,2040.05,2084.55], "prod_price_forecast":[0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457,0.14457], "load_cost_forecast":[0.234036,0.230366,0.230366,0.223456,0.223456,0.210182,0.210182,0.214396,0.214396,0.202506,0.202506,0.198168,0.198168,0.199218,0.199218,0.198514,0.198514,0.199911,0.199911,0.198761,0.198761,0.197352,0.197352,0.189775,0.189775,0.179554,0.179554,0.149791,0.149791,0.121894], "prediction_horizon":30, "def_total_hours":[0.00], "alpha":0.5, "beta":0.5}

log:

2023-05-06T18:52:08.139145771Z	stdout	UnboundLocalError: local variable 'df_day' referenced before assignment
2023-05-06T18:52:08.139074920Z	stdout	    self.df_final = pd.concat([self.df_final, df_day], axis=0)
2023-05-06T18:52:08.139009681Z	stdout	  File "/usr/local/lib/python3.8/site-packages/emhass-0.4.8-py3.8.egg/emhass/retrieve_hass.py", line 147, in get_data
2023-05-06T18:52:08.138952880Z	stdout	    rh.get_data(days_list, var_list)
2023-05-06T18:52:08.138724339Z	stdout	  File "/usr/local/lib/python3.8/site-packages/emhass-0.4.8-py3.8.egg/emhass/forecast.py", line 585, in get_load_forecast
2023-05-06T18:52:08.138650588Z	stdout	    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'], set_mix_forecast=True, df_now=df_input_data)
2023-05-06T18:52:08.138584837Z	stdout	  File "/usr/local/lib/python3.8/site-packages/emhass-0.4.8-py3.8.egg/emhass/command_line.py", line 120, in set_input_data_dict
2023-05-06T18:52:08.138515323Z	stdout	    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
2023-05-06T18:52:08.138458610Z	stdout	  File "src/emhass/web_server.py", line 170, in action_call
2023-05-06T18:52:08.138400534Z	stdout	    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
2023-05-06T18:52:08.138341945Z	stdout	  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1799, in dispatch_request
2023-05-06T18:52:08.138287669Z	stdout	    rv = self.dispatch_request()
2023-05-06T18:52:08.138228431Z	stdout	  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1823, in full_dispatch_request
2023-05-06T18:52:08.138175105Z	stdout	    rv = self.handle_user_exception(e)
2023-05-06T18:52:08.138112429Z	stdout	  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1825, in full_dispatch_request
2023-05-06T18:52:08.138054291Z	stdout	    response = self.full_dispatch_request()
2023-05-06T18:52:08.137979027Z	stdout	  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2528, in wsgi_app
2023-05-06T18:52:08.137922238Z	stdout	Traceback (most recent call last):
2023-05-06T18:52:08.137832824Z	stdout	2023-05-06 20:52:08,137 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
2023-05-06T18:52:08.137516519Z	stdout	2023-05-06 20:52:08,136 - web_server - ERROR - Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
2023-05-06T18:52:08.137313341Z	stdout	2023-05-06 20:52:08,136 - web_server - ERROR - The retrieved JSON is empty, check that correct day or variable names are passed
2023-05-06T18:52:08.120239184Z	stdout	2023-05-06 20:52:08,119 - web_server - INFO - Retrieve hass get data method initiated...
2023-05-06T18:52:08.118733146Z	stdout	2023-05-06 20:52:08,118 - web_server - INFO - Retrieving data from hass for load forecast using method = naive
2023-05-06T18:52:08.114915583Z	stdout	2023-05-06 20:52:08,114 - web_server - INFO - Retrieving weather forecast data using method = list
2023-05-06T18:52:02.659197136Z	stdout	2023-05-06 20:52:02,658 - web_server - INFO - Retrieve hass get data method initiated...
2023-05-06T18:52:02.650789122Z	stdout	2023-05-06 20:52:02,650 - web_server - INFO - Setting up needed data

The json itself wasn’t changed and I also created a new long-lived token and set it in the secrets-yaml.

Any idea?

Thanks!

vdiogo · May 7, 2023, 11:43am

My use case:

Have PV generation (no batteries)
Want to control my boiler heat resistance so it turns one when PV is generating more power than what the house consumes
Have flat energy rate

I configured EMHASS so that energy cost is higher during non-peak hours (as an attempt to influence the model believe it is NOT a good timing to turn my heat resistance) but since some weeks of gathering stats, it’s still saying it’s OK to connect the resistance in non peak hours:

ps:

Shelly–> PV generation > house consumption for >1500w (basically I’m injecting energy in the grid over 1500w)
EMHASS → EMHASS cost function model results

Question: setting a higher cost during non peak hours makes sense for my use case?

davidusb · May 7, 2023, 5:21pm

No idea about this, I haven’t seen these issues on my install. But this seems related to the long lived access token. Did you try restarting your system?

davidusb · May 7, 2023, 5:28pm

I don’t understand what are you trying to achieve, but if you want this kind of action then just setup an automation to do that.

EMHASS is an optimization, so the results are the reflect of the maximum possible profit for your system (if you choose that as the objective function).

Peak and non peak hours are related to power consumption not to PV production, so I really don’t understand how could that makes sense. But may be as I said I don’t understand what are you trying to achieve.

smi · May 7, 2023, 6:38pm

Yes I restarted home assistant and EMHASS.

I activated MFA a few days before it stopped working.
Might this be related?

EDIT: I don’t know why but without any further changes, it started working again over night

adammcdonagh · May 8, 2023, 9:22pm

Hello,
I’m just in the process of getting this all working… It seems I need to wait some time for there to be enough data in HA for the new ML function to work.

In the meantime, setting up the entity for var_load… sensor.power_load_no_var_loads, just want to clarify if I’ve understanding this right…

I’ve got PV and a 12KWh battery. Is the battery counted as a deferrable load? Does that mean I need to include it in the power_load_no_var_loads template to be subtracted when it’s charging? There’s no differentiation between when it’s being charged by the grid or by the PV is there? Is this OK?

This is how I’ve defined the template sensor. Does this look correct to you?

- sensor:
    - name: "Power load no var loads"
      unit_of_measurement: "W"
      device_class: "power"
      state: >
        {% set powerload = states('sensor.home_consumption') | float(default=0) %}
        {% set ac = states('sensor.power_aircon') | float(default=0) %}
        {% set ev = states('sensor.power_ev_charger') | float(default=0) %}
        {% set utility = states('sensor.power_utility_room') | float(default=0) %}
        {% set battery = states('sensor.battery_charge_rate') | float(default=0) %}
        {% set value = ( powerload - ac - ev - utility - battery ) | round(1,default=0) %}
        {{ value }}

sensor.home_consumption is my total power draw for the home (including EV charger, AC, battery charge from grid, etc)

Thanks

davidusb · May 9, 2023, 4:58am

No it is not. Don’t include in the sensor template.

This will be handled inside the optimization code. You don’t need to do anything about this.

There is an option to constrain to only charge your battery from PV. Set set_nocharge_from_grid: true

adammcdonagh · May 9, 2023, 9:35pm

Great, thanks a lot.

Will you be updating the add-on at some point? I notice in the main emhass repo, you changed days_to_retrieve = 30 to 9. But haven’t released this as a new version yet.

Thanks

davidusb · May 10, 2023, 4:30pm

It will be available in the next release yes.
Not enough time lately to work on this but it will be updated anytime soon.

torstein · May 16, 2023, 6:31pm

Anyone here with deye inverters? I have made the possibility to change the wanted soc from the emhass recommended soc. But the issue for me is that the battery will be fully charged within the expensive grid hours and later that day i will be giving away free energy. Have anyone solved this some how limiting the pv power to charge batteries when using zero export to ct?

torstein · May 19, 2023, 7:39am

I have found a way to set charging amps down on the deye inverter using modbus same way as setting wanted soc. But is it a recommended way doing this using p_batt_forcast?

unit_of_measurement: W
friendly_name: Battery Power Forecast
battery_scheduled_power: 
- date: '2023-05-19 10:00:00+02:00'
  p_batt_forecast: '-380.0'
- date: '2023-05-19 11:00:00+02:00'
  p_batt_forecast: '0.0'
- date: '2023-05-19 12:00:00+02:00'
  p_batt_forecast: '0.0'
- date: '2023-05-19 13:00:00+02:00'
  p_batt_forecast: '-3242.0'
- date: '2023-05-19 14:00:00+02:00'
  p_batt_forecast: '0.0'
- date: '2023-05-19 15:00:00+02:00'
  p_batt_forecast: '-493.0'
- date: '2023-05-19 16:00:00+02:00'
  p_batt_forecast: '-698.0'
- date: '2023-05-19 17:00:00+02:00'
  p_batt_forecast: '354.0'
- date: '2023-05-19 18:00:00+02:00'
  p_batt_forecast: '411.0'
- date: '2023-05-19 19:00:00+02:00'
  p_batt_forecast: '-1147.0'
- date: '2023-05-19 20:00:00+02:00'
  p_batt_forecast: '-206.0'
- date: '2023-05-19 21:00:00+02:00'
  p_batt_forecast: '989.0'

nordpool:

raw_today: 
- start: '2023-05-19T00:00:00+02:00'
  end: '2023-05-19T01:00:00+02:00'
  value: 0.969
- start: '2023-05-19T01:00:00+02:00'
  end: '2023-05-19T02:00:00+02:00'
  value: 0.941
- start: '2023-05-19T02:00:00+02:00'
  end: '2023-05-19T03:00:00+02:00'
  value: 0.907
- start: '2023-05-19T03:00:00+02:00'
  end: '2023-05-19T04:00:00+02:00'
  value: 0.871
- start: '2023-05-19T04:00:00+02:00'
  end: '2023-05-19T05:00:00+02:00'
  value: 0.887
- start: '2023-05-19T05:00:00+02:00'
  end: '2023-05-19T06:00:00+02:00'
  value: 0.918
- start: '2023-05-19T06:00:00+02:00'
  end: '2023-05-19T07:00:00+02:00'
  value: 0.972
- start: '2023-05-19T07:00:00+02:00'
  end: '2023-05-19T08:00:00+02:00'
  value: 1.201
- start: '2023-05-19T08:00:00+02:00'
  end: '2023-05-19T09:00:00+02:00'
  value: 1.19
- start: '2023-05-19T09:00:00+02:00'
  end: '2023-05-19T10:00:00+02:00'
  value: 1.028
- start: '2023-05-19T10:00:00+02:00'
  end: '2023-05-19T11:00:00+02:00'
  value: 0.831
- start: '2023-05-19T11:00:00+02:00'
  end: '2023-05-19T12:00:00+02:00'
  value: 0.707
- start: '2023-05-19T12:00:00+02:00'
  end: '2023-05-19T13:00:00+02:00'
  value: 0.708
- start: '2023-05-19T13:00:00+02:00'
  end: '2023-05-19T14:00:00+02:00'
  value: 0.678
- start: '2023-05-19T14:00:00+02:00'
  end: '2023-05-19T15:00:00+02:00'
  value: 0.677
- start: '2023-05-19T15:00:00+02:00'
  end: '2023-05-19T16:00:00+02:00'
  value: 0.75
- start: '2023-05-19T16:00:00+02:00'
  end: '2023-05-19T17:00:00+02:00'
  value: 0.835
- start: '2023-05-19T17:00:00+02:00'
  end: '2023-05-19T18:00:00+02:00'
  value: 0.968
- start: '2023-05-19T18:00:00+02:00'
  end: '2023-05-19T19:00:00+02:00'
  value: 1.069
- start: '2023-05-19T19:00:00+02:00'
  end: '2023-05-19T20:00:00+02:00'
  value: 1.067
- start: '2023-05-19T20:00:00+02:00'
  end: '2023-05-19T21:00:00+02:00'
  value: 1.063
- start: '2023-05-19T21:00:00+02:00'
  end: '2023-05-19T22:00:00+02:00'
  value: 1.057
- start: '2023-05-19T22:00:00+02:00'
  end: '2023-05-19T23:00:00+02:00'
  value: 1.053
- start: '2023-05-19T23:00:00+02:00'
  end: '2023-05-20T00:00:00+02:00'
  value: 0.971

markpurcell · May 19, 2023, 7:54am

I do the same to set the charging_amps for my EV:

Charging Amps = Power / Phases / Voltage

Just be careful with the charging and discharging (negative and positive values) and what your MODBUS expects.

service: number.set_value
data:
  value: >-
    {{(states('sensor.p_deferrable2')|int(0) /
    (state_attr('sensor.duka_charger_power','charger_phases')*3/2)|int(1)     /
    max(220,state_attr('sensor.duka_charger_power','charger_volts')|int(230)))|round(0)}}
target:
  entity_id: number.duka_charging_amps