EMHASS: An Energy Management for Home Assistant

prinzkafka · June 7, 2022, 3:16pm

Thank you. I tried this:

value_template: >
   {% set powerload = states('sensor.electricity_rietschener_str_24_total_power') | float(default=0) %}
   {% set heizpatrone = states('sensor.heizpatrone_power') | float(default=0) %}
   {% set geschirrspueler = states('sensor.geschirrspuler_current_consumption') | float(default=0) %}
   {% set waschmaschine = states('sensor.waschmaschine_power_consumption') | float(default=0) %}
   {% set value = ( powerload - heizpatrone - geschirrspueler - waschmaschine ) | round(1,default=0) %}    
   {{ value }}

but as you can see from the picture, there is still an unknown state in the database after each restart.
Could someone please post his working template sensor as an example?

markpurcell · June 7, 2022, 8:07pm

My energy plan for today, optimised by EMHASS. Total forecasts value is $60 credit.

Today’s s.qld energy plan. Buy low (10¢ / kWh forecast at midday) and sell high ($13/ kWh forecast at 6pm price spike)! The grid needs your help.

High prices this morning ($1/ kWh forecast @ 7am) and a price spike tonight ($15/ kWh @ 6pm) present opportunity to discharge the battery to the grid for a double cycle.

Very low prices, finally, over the midday solar soak (10¢/ kWh forecast between 10am - 3pm) is the ideal window to charge everything and run all your power hungry loads. With a forecast feed in tariff of 0¢ try to self consume 100% of any generated solar as any exports at that time are literally worthless.

davidusb · June 7, 2022, 9:05pm

So try to catch that wrong value. Something like this:

value_template: >
   {% set powerload = states('sensor.electricity_rietschener_str_24_total_power') | float(default=0) %}
   {% set heizpatrone = states('sensor.heizpatrone_power') | float(default=0) %}
   {% set geschirrspueler = states('sensor.geschirrspuler_current_consumption') | float(default=0) %}
   {% set waschmaschine = states('sensor.waschmaschine_power_consumption') | float(default=0) %}
   {% set value = ( powerload - heizpatrone - geschirrspueler - waschmaschine ) | round(1,default=0) %} 
   {% if (value == 'unavailable') or (value == 'unknown') %}
      {{ 0.0 }}
   {% else %}
      {{ value }}

prinzkafka · June 8, 2022, 5:58am

Thank you. I have changed it that way. Still after every restart one unknown state in the database.

davidusb · June 8, 2022, 7:17am

Well that template sensor should be fine for the new values that will arrive to your base sensors. However the unknown values that where stored in the database are still there. If this is the case, here is a guide to modify database values manually: How to fix statistics data (e.g. energy data)

And are you still having the same error in emhass?

prinzkafka · June 8, 2022, 7:34am

Yes, same error with this template sensor:

[2022-06-08 09:32:00,090] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 81, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 454, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 107, in get_data
    data = response.json()[0]
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Extra data] 500 Internal Server Error
Server got itself in trouble: 4

davidusb · June 10, 2022, 9:59am

Hi, I just released a new add-on version with some improvements >> v0.2.18: Release EMHASS add-on v0.2.18 · davidusb-geek/emhass-add-on · GitHub

Added new sensors for the grid power and the total value of the cost function after an optimization.
Added a new method to obtain a mix forecast and take into account now/current measured values. This will work when using MPC optimization and you can define the two new runtime parameters alpha and beta as passed data when calling naive-mpc-optim. See this description in the docs: The forecast module — emhass 0.3.16 documentation
Improved the documentation a little (still some work to do on this)

markpurcell · June 13, 2022, 6:39am

emhass with the ‘now’ values really makes a smooth set of calculations for its forecasts.

Below you can see the measured values in the top graphic and the forecast values from emhass, running mpc-optim every minute.

I really like how you can see the grid power forecast at 3:40 pm slowly increment up as the PV production reduces to maintain the fixed load (5.5 kW for deferrable1 - my pool heater) and then at 4pm deferrable2 my variable EV charger matches PV production minus the baseload and slowly increments down with PV production.

At about 3:30pm you can see deferrable1 switch on and off and on and off and on. I have a hysteresis delay (5 minutes on, 2 minutes off), to smooth out the switching so these rapid changes aren’t actually passed through to the heater pump.

You can see some slight differences to the forecasts, at 2:40pm my battery is charging instead of deferrable1 (my pool heater) operating, I intervened getting the battery SOC to 100% before sunset. At 3:40pm the battery was discharging (top graphic), whilst the bottom graphic wanted to draw from the grid, I am still trying to get that level of fine control over my battery.

davidusb · June 13, 2022, 8:57am

Nice to see the improvements working!

The fine tuning is very case specific. These should be sorted out with custom automation rules. I saw that you had already worked out a lot of these specific rules. EMHASS should be used as a top level intelligent energy management, but you will still need those low level rule-based management for fine tuning.

As for the new mixed forecast using now/current values, you can play with the settings of the weight of now values over forecast values that will be used. This is done by passing parameters alpha and beta at runtime. They are both fixed by default at 0.5. See the equation for this here in the docs: The forecast module — emhass 0.8.0 documentation

Cheers

Gunth · July 5, 2022, 1:45pm

I tried to install this addon but when trigger a perfect-optim i recieve a error in my log, did i miss somthing in my configuration ?

I’m running it as an add-on for Home Assistant OS with the latest version (0.2.19)

[2022-06-30 20:00:34,713] ERROR in optimization: It was not possible to find a valid solver for Pulp package
[2022-06-30 20:00:34,715] INFO in optimization: Status: Not Solved
[2022-06-30 20:00:34,715] WARNING in optimization: Cost function cannot be evaluated, probably None
[2022-06-30 20:00:34,729] ERROR in app: Exception on /action/perfect-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 143, in action_call
    opt_res = perfect_forecast_optim(input_data_dict, app.logger)
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 161, in perfect_forecast_optim
    opt_res = input_data_dict['opt'].perform_perfect_forecast_optim(df_input_data, input_data_dict['days_list'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/optimization.py", line 460, in perform_perfect_forecast_optim
    opt_tp = self.perform_optimization(data_tp, P_PV, P_load,
  File "/usr/local/lib/python3.9/dist-packages/emhass/optimization.py", line 384, in perform_optimization
    opt_tp["P_grid"] = [P_grid_pos[i].varValue + P_grid_neg[i].varValue for i in set_I]
  File "/usr/local/lib/python3.9/dist-packages/emhass/optimization.py", line 384, in <listcomp>
    opt_tp["P_grid"] = [P_grid_pos[i].varValue + P_grid_neg[i].varValue for i in set_I]
TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType'

davidusb · July 5, 2022, 2:10pm

Hi, as the log says, the system did not found a valid solver. The default solver works just fine on amd64 machines, but not so in others. You just need to modify the solver in the configuration params.

This can be controlled in the configuration file with parameters lp_solver and lp_solver_path. The options for lp_solver are: ‘PULP_CBC_CMD’, ‘GLPK_CMD’ and ‘COIN_CMD’. If using ‘COIN_CMD’ as the solver you will need to provide the correct path to this solver in parameter lp_solver_path, ex: ‘/usr/bin/cbc’.

Gunth · July 5, 2022, 2:25pm

Thanks it work with GLPK_CMD, i 'm running the addon on a Raspberry Pi4 that’s the reason why it was failing …

haraldov · July 7, 2022, 10:44am

I installed the Emhass addon and in the configuration changed the default sensor to:

      - name: "Power_load_no_var_loads"
        unit_of_measurement: "W"
        device_class: power
        state: > 
          {% set powerload = states('sensor.power_adresse_xx') | float(default=0) %}
          {% set vvb = states('sensor.bryter_varmvannsbereder_electric_consumed_w') | float(default=0) %}
          {% set vaskemaskin = states('sensor.strombryter_vaskemaskin_electrical_measurement') | float(default=0) %}
          {% set value = ( powerload - vvb - vaskemaskin) | round(1,default=0) %}
          {{ value }}

When I trigger a “Perfect optimization” or “Day-ahead optimization” I get error in the log. The recorded history of sensor.power_load_no_var_loads is only one day old. Is this error related to how many days the recorder has kept the data?

[2022-07-07 12:35:35,609] INFO in command_line: Setting up needed data
[2022-07-07 12:35:35,615] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-07-07 12:35:35,652] ERROR in retrieve_hass: The retrieved JSON is empty, check that correct day or variable names are passed
[2022-07-07 12:35:35,653] ERROR in retrieve_hass: Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
[2022-07-07 12:35:35,653] ERROR in app: Exception on /action/perfect-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 64, in set_input_data_dict
    rh.get_data(days_list, var_list,
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 130, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
[2022-07-07 12:35:36,992] INFO in command_line: Setting up needed data
[2022-07-07 12:35:36,994] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2022-07-07 12:35:39,318] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2022-07-07 12:35:39,319] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-07-07 12:35:39,328] ERROR in retrieve_hass: The retrieved JSON is empty, check that correct day or variable names are passed
[2022-07-07 12:35:39,328] ERROR in retrieve_hass: Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
[2022-07-07 12:35:39,328] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 77, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 476, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 130, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment

haraldov · July 7, 2022, 2:15pm

The perfect optimization are now running without error. I was solved when I waited 2 days (historic_days_to_retrieve). The day-ahead optimization is still giving error, but I hope it is solved when I wait a few hours.

[2022-07-07 16:04:14,019] INFO in web_server: EMHASS server online, serving index.html...
[2022-07-07 16:04:16,751] INFO in command_line: Setting up needed data
[2022-07-07 16:04:16,755] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-07-07 16:04:17,091] INFO in web_server:  >> Performing perfect optimization...
[2022-07-07 16:04:17,091] INFO in command_line: Performing perfect forecast optimization
[2022-07-07 16:04:17,093] INFO in optimization: Perform optimization for perfect forecast scenario
[2022-07-07 16:04:17,094] INFO in optimization: Solving for day: 5-7-2022
[2022-07-07 16:04:17,162] INFO in optimization: Status: Optimal
[2022-07-07 16:04:17,163] INFO in optimization: Total value of the Cost function = -2.31
[2022-07-07 16:04:17,171] INFO in optimization: Solving for day: 6-7-2022
[2022-07-07 16:04:17,222] INFO in optimization: Status: Optimal
[2022-07-07 16:04:17,222] INFO in optimization: Total value of the Cost function = -4.5
/usr/local/lib/python3.9/dist-packages/emhass/web_server.py:46: FutureWarning:
Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.

davidusb · July 7, 2022, 9:14pm

Ok, nice that the perfect optimization is running, effectively the historic_days_to_retrieve has to match the available data on your HA database.
What is the error on the day-ahead optimization?

haraldov · July 8, 2022, 1:05am

When I push the “Day-Ahead Optimization” button in the web page I get this: “ERROR in retrieve_hass”. But when I push the “Perfect Optimization” I do not get the error.

Home Assistant 2022.7.0
Supervisor 2022.07.0
Operating System 8.2
Frontend 20220706.0 - latest

2022-07-08 02:59:14,383] INFO in web_server: EMHASS server online, serving index.html...
[2022-07-08 02:59:26,345] INFO in command_line: Setting up needed data
[2022-07-08 02:59:26,372] INFO in forecast: Retrieving weather forecast data using method = scrapper
[2022-07-08 02:59:27,715] INFO in forecast: Retrieving data from hass for load forecast using method = naive
[2022-07-08 02:59:27,716] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-07-08 02:59:27,724] ERROR in retrieve_hass: The retrieved JSON is empty, check that correct day or variable names are passed
[2022-07-08 02:59:27,725] ERROR in retrieve_hass: Either the names of the passed variables are not correct or days_to_retrieve is larger than the recorded history of your sensor (check your recorder settings)
[2022-07-08 02:59:27,725] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 134, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 77, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 476, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 130, in get_data
    self.df_final = pd.concat([self.df_final, df_day], axis=0)
UnboundLocalError: local variable 'df_day' referenced before assignment
[2022-07-08 02:59:55,840] INFO in command_line: Setting up needed data
[2022-07-08 02:59:55,843] INFO in retrieve_hass: Retrieve hass get data method initiated...
[2022-07-08 02:59:56,325] INFO in web_server:  >> Performing perfect optimization...
[2022-07-08 02:59:56,325] INFO in command_line: Performing perfect forecast optimization
[2022-07-08 02:59:56,328] INFO in optimization: Perform optimization for perfect forecast scenario
[2022-07-08 02:59:56,328] INFO in optimization: Solving for day: 6-7-2022
[2022-07-08 02:59:56,417] INFO in optimization: Status: Optimal
[2022-07-08 02:59:56,418] INFO in optimization: Total value of the Cost function = -8.68
[2022-07-08 02:59:56,423] INFO in optimization: Solving for day: 7-7-2022
[2022-07-08 02:59:56,483] INFO in optimization: Status: Optimal
[2022-07-08 02:59:56,484] INFO in optimization: Total value of the Cost function = -6.87
/usr/local/lib/python3.9/dist-packages/emhass/web_server.py:46: FutureWarning:
Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.

davidusb · July 8, 2022, 6:34am

In your configuration you need to check that in both var_PV and var_load, they have the correct name as in your HA sensors and yes that there is enough history data.

haraldov · July 8, 2022, 8:57am

I suspect the ha recorder does not have enough data for “sensor.power_adresse_xx” (only one and a half day recording). I will wait. I am going to report back later.

      - name: "Power_load_no_var_loads"
        unit_of_measurement: "W"
        device_class: power
        state: > 
          {% set powerload = states('sensor.power_adresse_xx') | float(default=0) %}
          {% set vvb = states('sensor.bryter_varmvannsbereder_electric_consumed_w') | float(default=0) %}
          {% set vaskemaskin = states('sensor.strombryter_vaskemaskin_electrical_measurement') | float(default=0) %}
          {% set value = ( powerload - vvb - vaskemaskin) | round(1,default=0) %}
          {{ value }}

haraldov · July 8, 2022, 8:15pm

The “Day-Ahead Optimization” is now working. The solution was to wait for 2 days of recording.

Next question:
I have 12 solar cell modules (REC_Solar_REC295TP2) on my garage roof with three AP System QS1 microinvertere (Discontinued APsystems QS1 - APsystems USA | The global leader in multi-platform MLPE technology) placed in the center with four solar cell modules attached. So I have 12 panels and 3 QS1 microinverteres. The microinverters are attached to one 220 V power cabel.

On the house roof I have 6 solar cell modules (REC_Solar_REC295TP2). Four of the panels are attaced to one QS1 and two of them are attached to APSystem YC600. So I have 6 panels and 1 QS1 microinverteres + 1 YC600 (Discontinued APsystems YC600 - APsystems USA | The global leader in multi-platform MLPE technology) pr string. The microinverters are attached to one 220 V power cabel.

Can you verify if pv configuration is correct? I know the “list_surface_tilt” and “list_surface_azimuth” is not correct. I must measure it outside first.

davidusb · July 8, 2022, 9:48pm

Great!
From your description your configuration seems good to me. What you should do is validate that it is good by comparing the PV forecast that will be produced with your actual power production. If they match very well on a clear sky day then it is validated.