EMHASS: An Energy Management for Home Assistant

rcruikshank · July 10, 2023, 9:41am

Also a lot of errors in logs now:

2023-07-10 19:31:00,560 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 174, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 110, in set_input_data_dict
    rh.get_data(days_list, var_list,
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 140, in get_data
    df_tp = df_raw.copy()[['state']].replace(
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5920, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 419, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/blocks.py", line 580, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1292, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1237, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1098, in astype_nansafe
    result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1181, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: 'NOTRUN'
2023-07-10 19:31:00,598 - web_server - INFO - Setting up needed data
2023-07-10 19:31:00,600 - web_server - INFO -  >> Publishing data...
2023-07-10 19:31:00,601 - web_server - INFO - Publishing data to HASS instance
2023-07-10 19:31:00,617 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = -1.5
2023-07-10 19:31:00,629 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 500.63
2023-07-10 19:31:00,638 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2023-07-10 19:31:00,648 - web_server - INFO - Successfully posted to sensor.p_deferrable1 = 0.0
2023-07-10 19:31:00,657 - web_server - INFO - Successfully posted to sensor.p_deferrable2 = 0.0
2023-07-10 19:31:00,668 - web_server - INFO - Successfully posted to sensor.p_deferrable3 = 0.0
2023-07-10 19:31:00,678 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 0.0
2023-07-10 19:31:00,688 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 10.0
2023-07-10 19:31:00,697 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = 502.13
2023-07-10 19:31:00,707 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = -8.08
2023-07-10 19:31:00,718 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.45
2023-07-10 19:31:00,728 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.36
2023-07-10 19:32:00,043 - web_server - INFO - Setting up needed data
2023-07-10 19:32:00,043 - web_server - ERROR - ERROR: The passed data is either not a list or the length is not correct, length should be 48
2023-07-10 19:32:00,044 - web_server - ERROR - Passed type is <class 'list'> and length is 48
2023-07-10 19:32:00,045 - web_server - INFO - Retrieve hass get data method initiated...
2023-07-10 19:32:00,648 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 174, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 110, in set_input_data_dict
    rh.get_data(days_list, var_list,
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 140, in get_data
    df_tp = df_raw.copy()[['state']].replace(
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5920, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 419, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/blocks.py", line 580, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1292, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1237, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1098, in astype_nansafe
    result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1181, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: 'NOTRUN'
2023-07-10 19:32:00,679 - web_server - INFO - Setting up needed data
2023-07-10 19:32:00,681 - web_server - INFO -  >> Publishing data...
2023-07-10 19:32:00,681 - web_server - INFO - Publishing data to HASS instance
2023-07-10 19:32:00,694 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = -1.5
2023-07-10 19:32:00,704 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 500.63
2023-07-10 19:32:00,716 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2023-07-10 19:32:00,727 - web_server - INFO - Successfully posted to sensor.p_deferrable1 = 0.0
2023-07-10 19:32:00,736 - web_server - INFO - Successfully posted to sensor.p_deferrable2 = 0.0
2023-07-10 19:32:00,747 - web_server - INFO - Successfully posted to sensor.p_deferrable3 = 0.0
2023-07-10 19:32:00,757 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 0.0
2023-07-10 19:32:00,766 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 10.0
2023-07-10 19:32:00,776 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = 502.13
2023-07-10 19:32:00,786 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = -8.08
2023-07-10 19:32:00,798 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.45
2023-07-10 19:32:00,809 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.36

markpurcell · July 10, 2023, 9:45am

The issue is the Amber predictions does not always have 48 elements.

In your day-ahead optimisation we had prediction_horizon set to 33 elements which is a safe number (amber always has 33 elements), so if you change that to 33 it should be functional.

rcruikshank · July 10, 2023, 9:54am

After correcting everything? How did that get corrupted? Was working then not?

markpurcell · July 10, 2023, 9:59am

reduce prediction_horizon to 33 and soc_final to 0.05

rcruikshank · July 10, 2023, 11:09am

Thanks Mark. I think its back to normal now. Corruption in the shell commands. I have to put it down to Visual Studio Code being too clever. Changed punctuation throughout the shell command scripts.

Also I had solcast updating every 30 mins when new accound only get 10 API calls a day. Had to change it to a scan interval or 2:24.

markpurcell · July 10, 2023, 11:31am

markpurcell:

        "pv_power_forecast": {{
          ([states('sensor.APF_Generation_Entity')|int(0)] +
          state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list +
          state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list
          )| tojson
        }},
        "prediction_horizon": {{
          min(48, (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
        }},

@rcruikshank here are better templates for prediction_horizon and pv_power_forecast, the latter depends on GitHub - oziee/ha-solcast-solar: Solcast Integration for Home Assistant which will manage your 10 API calls and also make it available for the Home Assistant Energy dashboard.

rcruikshank · July 10, 2023, 9:53pm

ok thanks for that. The ha-solcast-solar HACS doesn’t seem to support the rooftop resource or the rooftop version has been discontinued.

markpurcell · July 11, 2023, 12:51am

I have the SolCast hobbiest account with two arrays:

Which then gets pulled into the SolCast integration:

rcruikshank · July 11, 2023, 2:19am

ok thanks. Will let the dust settle on current config and set it up.

rcruikshank · July 11, 2023, 11:06am

Mark
How often do you run post_amber_forecast?

markpurcell · July 11, 2023, 11:11am

I don’t run day-ahead-optim.

I run mpc-optim every minute, in order to include updated prices, load and solar production.

This is now accurate enough that my EV charging will adjust down to the require watt if an unscheduled load; oven, dishwasher comes online.

s2719632 · July 11, 2023, 11:38am

Just to confirm Mark, you only run mpc-optim and dont run post_amber_forecast as its own shell command? How frequently do you run the publish data shell command? For battery control does this need to be frequently, say every minute, if we are using mpc optim? Thanks

markpurcell · July 11, 2023, 11:52am

As I mentioned in my earlier post I am using a restful_command directly rather than shell_command & curl.

Each minute my automation calls the naive_mpc_optim and publish_data end points.

s2719632 · July 11, 2023, 11:55am

Thank you for your reply. I was having difficulty debugging the rest_command when I tried it. I have had success using a shell command instead. Will try your rest_command with fresh eyes. Thank you again for the time you put in assisting here

bakerboy908 · July 11, 2023, 12:11pm

Slowly starting to get things working nicely, Today i decluttered my automation for controlling inverter modes.

I have created a template sensor that decides what “Mode” emhass is in, Chargine, Discharging, Exporting Solar or General (Self consumption)

This means now i dont need to touch my complicated automation for inverter control, only the relativly simple template sensor.

- sensor:
    - unique_id: emhass_mode
      name: "EMHASS Mode"
      state: >-
        {%set bat_power = states('sensor.p_batt_forecast') | float %}
        {%set load = states('sensor.p_load_forecast') | float%}
        {%set grid = states('sensor.p_grid_forecast') | float%}
        {%set solar = states('sensor.p_pv_forecast') | float %}


        {% if bat_power + load < 0 and grid > 0 %}
          charge 
        {% elif bat_power - load > 0 and grid < 0 and solar <= 0%}
          discharge
        {% elif bat_power == 0 and grid < 0 %}
          exporting_solar 
        {%elif bat_power <= 0 and grid > 0%}
          backup 
        {%else %} 
          general 
        {%- endif %}

@davidusb A toggle i would love to see added to EMHASS is no discharging to the grid while the sun is shinning

rcruikshank · July 11, 2023, 1:35pm

Oh ok you don’t run post_amber_forecast at all either. Just post_mpc_optim_solcast and publish_data every minute using restful command. ok thanks again

Overlordt · July 11, 2023, 6:43pm

I’ve been pushing Tibber prices from a REST sensor and some templating to EMHASS.
Tibber current day prices start at 00:00 hours. But it looks like EMHASS uses the first value as the current or next hour? depending on when the data is published this seems to create an unwanted offset.

I could push only future prices but won’t that mess with the expected number of values/hours?

Only pushing forecast data at 00:00 hours could solve this I guess. But what happens after a restart and no data is present? Waiting till midnight cannot be the solution in a production like environment, right?

rcruikshank · July 12, 2023, 1:45am

@markpurcell ok you don’t run post_amber_forecast at all. Just post_mpc_optim_solcast and publish_data every minute? thanks again for your help.

markpurcell · July 12, 2023, 3:26am

What I do is utilise the prediction_horizon variable, either set it to the known minimum (for Amber that is 33) or dynamically calculate by counting the number of price forecasts.

See my example above.

rcruikshank · July 12, 2023, 3:47am

I’ve got the prediction horizon set to 33 at the moment (see below). However, I count 48 elements in each array for each of load_cost_forecast, prod_price_forecast and pv_power_forecast.

Should I change this to 48? What is the effect of this change?

I’m also still running post_amber_optim at 5:50am when I don’t think I need to if I’m going straight to 1 minute MPC optimisations.

Also, SOC init and SOC final can go to 100 and 0 (or 1 to 0 should I say) for sonnen I think as the user batery state is 10% less than real battery capacity, if my understanding of these two data is correct.

post_mpc_optim_solcast:
    "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":[0.2, 0.19, 0.2, 0.17, 0.17, 0.17, 0.17, 0.17, 0.16, 0.16, 0.16, 0.17, 0.17, 0.21, 0.31, 0.27, 0.2, 0.17, 0.17, 0.16, 0.13, 0.13, 0.12, 0.09, 0.1, 0.11, 0.11, 0.13, 0.39, 0.4, 0.39, 0.39, 0.42, 0.45, 0.48, 0.56, 0.56, 0.56, 0.51, 0.5, 0.28, 0.27, 0.27, 0.23, 0.26, 0.21, 0.21, 0.2], \"prod_price_forecast\":[0.09, 0.09, 0.09, 0.07, 0.07, 0.07, 0.07, 0.07, 0.06, 0.06, 0.06, 0.07, 0.07, 0.11, 0.2, 0.16, 0.1, 0.07, 0.07, 0.06, 0.02, 0.02, 0.0, -0.02, -0.01, -0.0, 0.0, 0.02, 0.3, 0.31, 0.3, 0.3, 0.34, 0.36, 0.38, 0.45, 0.46, 0.46, 0.42, 0.41, 0.17, 0.16, 0.16, 0.13, 0.15, 0.11, 0.11, 0.09], \"pv_power_forecast\":[3244, 3333, 3359, 3304, 3192, 2976, 2712, 2384, 2003, 1497, 926, 227, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 99, 666, 1200, 1677, 2107, 2529, 2922, 3166], \"prediction_horizon\":33,\"soc_init\":0.97,\"soc_final\":0.05,\"def_total_hours\":[2,2,2,2]}' http://localhost:5000/action/naive-mpc-optim"
This template listens for the following state changed events:

Entity: sensor.cecil_st_feed_in_forecast
Entity: sensor.cecil_st_feed_in_price
Entity: sensor.cecil_st_general_forecast
Entity: sensor.cecil_st_general_price
Entity: sensor.solcast_24hrs_forecast
Entity: sensor.sonnenbatterie_84324_state_charge_user