EMHASS: An Energy Management for Home Assistant

[davidusb]

@markpurcell

@davidusb very big release today with a whole bunch of additional functionality, which is great to see the project moving forward.

Sorry about this. I did tested the add-on before the release but with the default config with two deferrable loads and everything was working fine.

Then went to my personal configuration where I have 3 deferrable loads and found a similar error, realizing what @michaelpiron and @GeoDerp did, which is to have the same number of list items as your number of deferrable loads. After this everything is working fine again.
I did had go to share folder and purge (erase) the .pkl files there.

Please test again following this procedure. It worked for me using the day-ahead optimization, everything is back on track.

[checking12 (Sander)]

0.6.0 did resolve this issue for me

2024-01-30 09:07:02,319 - web_server - INFO - Launching the emhass webserver at: http://0.0.0.0:5000
2024-01-30 09:07:02,321 - web_server - INFO - Home Assistant data fetch will be performed using url: http://supervisor/core/api
2024-01-30 09:07:02,322 - web_server - INFO - The data path is: /share
2024-01-30 09:07:02,325 - web_server - INFO - Using core emhass version: 0.7.1
waitress   INFO  Serving on http://0.0.0.0:5000

now i can start figuring out things I wanna do.

In that context i have a question too:
I have 3 different ‘battery’ vehicles to charge on a smart outlet, however their charging capacity and speed is different and the need for charging is also unpredictable.
Is there any way i can have this add on try to charge the devices with the surplus on PV output?
I was thinking in direction:
A - turn on smart plug, check Watt usage to determine if charger would activate and then use the EM to determine what moment it should charge.
or
B - have EM turn on smart plug and be done with charging if nothing is charging

[michaelpiron]

I removed all pkl and csv files from share folder, and restarted the add-on.
However, the error message remains:

[GeoDerp]

@michaelpiron whats the configuration look like?

[davidusb]

@michaelpiron

I removed all pkl and csv files from share folder, and restarted the add-on.
However, the error message remains:

So even after the lists are correct you have this error?
Probably specific to MPC.

[110hs]

I updated through the automatic update, which identified the new release, and rebooted following the installation, also to reload my updated restful commands.

I have only 1 (@davidusb I think the number of deferrable loads is not the key to the problem) fake deferrable load (0 hours operating - it was mentioned it’s better than passing nothing) and by heart I think the configuration was showing 0 as start and end operating time for it; but actually I pass these parameters “runtime” so should make no difference.

This is how I updated my command to make it work with v.0.6.0 of the add-on:

emhass_mpc_optim:
  url: http://localhost:5000/action/naive-mpc-optim
  method: POST
  timeout: 30
  content_type: 'application/json'
  #model-type is ignored unless you use mlforecaster in the config file
  payload: >-
    { "prediction_horizon": {{ prediction_horizon }}, "pv_power_forecast": {{ pv_power_forecast }}, "num_def_loads": {{ num_def_loads }}, "P_deferrable_nom": {{ P_deferrable_nom }}, "def_total_hours": {{ def_total_hours }}, "treat_def_as_semi_cont": {{ treat_def_as_semi_cont }}, "def_start_timestep": {{ def_start_timestep }}, "def_end_timestep": {{ def_end_timestep }},  "set_def_constant": {{ set_def_constant }}, "soc_init": {{ soc_init }}, "soc_final": {{ soc_final }}, "load_cost_forecast": {{ load_cost_forecast }}, "prod_price_forecast": {{ prod_price_forecast }}, "alpha": {{ alpha }}, "beta": {{ beta }}, "model_type": {{ '"' ~ model_type ~ '"' }} }

And here a portion of my automation which I updated and was passing the parameters

    - service: rest_command.emhass_mpc_optim
      data:
        num_def_loads: 1 #if you don't have any it's still better to set this to something and then set the power to 0
        P_deferrable_nom: [0]
        def_total_hours: [2] #if you don't have any it's still better to set this to something and then set the power to 0
        def_start_timestep: [0]
        def_end_timestep: [0]
        treat_def_as_semi_cont: [1]
        set_def_constant: [0]
        alpha: 0.25
        beta: 0.75
        model_type: 'KNN' # this parameter is ignored unless you use mlforecaster in the config file 
        soc_init: >-
          {{ (states('sensor.sonnen_usoc_filtered')|float(0)) / 100 }}
        soc_final: 0.3
        prediction_horizon: ....
        pv_power_forecast: ....
        load_cost_forecast: ....
        prod_price_forecast: ....

At the moment I’ve reverted to the previous version without the start/end deferrable windows.

My current (0.5.4) working configuration

hass_url: empty
long_lived_token: empty
costfun: self-consumption
logging_level: DEBUG
optimization_time_step: 30
historic_days_to_retrieve: 2
method_ts_round: first
set_total_pv_sell: false
lp_solver: COIN_CMD
lp_solver_path: /usr/bin/cbc
set_nocharge_from_grid: false
set_nodischarge_to_grid: true
set_battery_dynamic: false
battery_dynamic_max: 0.9
battery_dynamic_min: -0.9
weight_battery_discharge: 0
weight_battery_charge: 0
load_forecast_method: naive
sensor_power_photovoltaics: sensor.production_pv_w
sensor_power_load_no_var_loads: sensor.consumption_filtered_w
number_of_deferrable_loads: 1
list_nominal_power_of_deferrable_loads:
  - nominal_power_of_deferrable_loads: 0
list_operating_hours_of_each_deferrable_load:
  - operating_hours_of_each_deferrable_load: 2
list_peak_hours_periods_start_hours:
  - peak_hours_periods_start_hours: "07:00"
list_peak_hours_periods_end_hours:
  - peak_hours_periods_end_hours: "23:00"
list_treat_deferrable_load_as_semi_cont:
  - treat_deferrable_load_as_semi_cont: true
load_peak_hours_cost: 0.1907
load_offpeak_hours_cost: 0.1419
photovoltaic_production_sell_price: 0.065
maximum_power_from_grid: 5000
list_pv_module_model:
  - pv_module_model: SunPower_SPR_E20_327
  - pv_module_model: SunPower_SPR_P19_395_COM
list_pv_inverter_model:
  - pv_inverter_model: SMA_America__SB3000TL_US_22__240V_
  - pv_inverter_model: SMA_America__STP50_US_40__480V_
list_surface_tilt:
  - surface_tilt: 17
  - surface_tilt: 17
list_surface_azimuth:
  - surface_azimuth: 225
  - surface_azimuth: 225
list_modules_per_string:
  - modules_per_string: 9
  - modules_per_string: 14
list_strings_per_inverter:
  - strings_per_inverter: 1
  - strings_per_inverter: 1
set_use_battery: true
battery_discharge_power_max: 7100
battery_charge_power_max: 7100
battery_discharge_efficiency: 0.95
battery_charge_efficiency: 0.95
battery_nominal_energy_capacity: 27000
battery_minimum_state_of_charge: 0
battery_maximum_state_of_charge: 1
battery_target_state_of_charge: 0.3

[michaelpiron]

Here’s my configuration:

costfun: profit
logging_level: DEBUG
set_total_pv_sell: false
set_nocharge_from_grid: false
set_nodischarge_to_grid: false
sensor_power_photovoltaics: sensor.solcast_forecast_data
sensor_power_load_no_var_loads: sensor.power_load_no_var_loads
number_of_deferrable_loads: 2
list_nominal_power_of_deferrable_loads:
  - nominal_power_of_deferrable_loads: 2000
  - nominal_power_of_deferrable_loads: 7360
list_operating_hours_of_each_deferrable_load:
  - operating_hours_of_each_deferrable_load: 18
  - operating_hours_of_each_deferrable_load: 1
list_start_timesteps_of_each_deferrable_load:
  - start_timesteps_of_each_deferrable_load: 0
  - start_timesteps_of_each_deferrable_load: 3
list_end_timesteps_of_each_deferrable_load:
  - end_timesteps_of_each_deferrable_load: 0
  - end_timesteps_of_each_deferrable_load: 9
list_peak_hours_periods_start_hours:
  - peak_hours_periods_start_hours: "05:54"
  - peak_hours_periods_start_hours: "10:24"
list_peak_hours_periods_end_hours:
  - peak_hours_periods_end_hours: "09:24"
  - peak_hours_periods_end_hours: "11:54"
list_treat_deferrable_load_as_semi_cont:
  - treat_deferrable_load_as_semi_cont: true
  - treat_deferrable_load_as_semi_cont: false
list_set_deferrable_load_single_constant:
  - set_deferrable_load_single_constant: false
  - set_deferrable_load_single_constant: false
load_peak_hours_cost: 0.1907
load_offpeak_hours_cost: 0.1419
photovoltaic_production_sell_price: 0.065
maximum_power_from_grid: 9000
list_pv_module_model:
  - pv_module_model: CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M
list_pv_inverter_model:
  - pv_inverter_model: Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_
list_surface_tilt:
  - surface_tilt: 30
list_surface_azimuth:
  - surface_azimuth: 205
list_modules_per_string:
  - modules_per_string: 16
list_strings_per_inverter:
  - strings_per_inverter: 1
set_use_battery: false
battery_nominal_energy_capacity: 5000

sensor.solcast_forecast_data

sensor.power_load_no_var_loads

I’m also analyzing the code, but don’t understand the cause of the issue so far.

[GeoDerp]

Thanks for that @michaelpiron.

Try adding load_negative: true (or false)

[michaelpiron]

Tried that, but error message stays the same

[markpurcell]

@davidusb

I did had go to share folder and purge (erase) the .pkl files there.

I have a few .pkl files, which should I remove?

[110hs]

I decided to try again the update.

This is the configuration following the update:

costfun: self-consumption
logging_level: DEBUG
set_total_pv_sell: false
set_nocharge_from_grid: false
set_nodischarge_to_grid: true
sensor_power_photovoltaics: sensor.production_pv_w
sensor_power_load_no_var_loads: sensor.consumption_filtered_w
number_of_deferrable_loads: 1
list_nominal_power_of_deferrable_loads:
  - nominal_power_of_deferrable_loads: 0
list_operating_hours_of_each_deferrable_load:
  - operating_hours_of_each_deferrable_load: 2
list_start_timesteps_of_each_deferrable_load:
  - start_timesteps_of_each_deferrable_load: 0
list_end_timesteps_of_each_deferrable_load:
  - end_timesteps_of_each_deferrable_load: 0
list_peak_hours_periods_start_hours:
  - peak_hours_periods_start_hours: "07:00"
list_peak_hours_periods_end_hours:
  - peak_hours_periods_end_hours: "23:00"
list_treat_deferrable_load_as_semi_cont:
  - treat_deferrable_load_as_semi_cont: true
list_set_deferrable_load_single_constant:
  - set_deferrable_load_single_constant: false
load_peak_hours_cost: 0.1907
load_offpeak_hours_cost: 0.1419
photovoltaic_production_sell_price: 0.065
maximum_power_from_grid: 5000
list_pv_module_model:
  - pv_module_model: SunPower_SPR_E20_327
  - pv_module_model: SunPower_SPR_P19_395_COM
list_pv_inverter_model:
  - pv_inverter_model: SMA_America__SB3000TL_US_22__240V_
  - pv_inverter_model: SMA_America__STP50_US_40__480V_
list_surface_tilt:
  - surface_tilt: 17
  - surface_tilt: 17
list_surface_azimuth:
  - surface_azimuth: 225
  - surface_azimuth: 225
list_modules_per_string:
  - modules_per_string: 9
  - modules_per_string: 14
list_strings_per_inverter:
  - strings_per_inverter: 1
  - strings_per_inverter: 1
set_use_battery: true
battery_nominal_energy_capacity: 27000
hass_url: empty
long_lived_token: empty
optimization_time_step: 30
historic_days_to_retrieve: 2
method_ts_round: first
lp_solver: COIN_CMD
lp_solver_path: /usr/bin/cbc
set_battery_dynamic: false
battery_dynamic_max: 0.9
battery_dynamic_min: -0.9
weight_battery_discharge: 0
weight_battery_charge: 0
load_forecast_method: naive
battery_discharge_power_max: 7100
battery_charge_power_max: 7100
battery_discharge_efficiency: 0.95
battery_charge_efficiency: 0.95
battery_minimum_state_of_charge: 0
battery_maximum_state_of_charge: 1
battery_target_state_of_charge: 0.3

Something I noticed right after the update is that the configuration was showing 2 start/end deferrable window items I had to manually re-align (see picture)

The command and automation are the same I posted above.

Still same error:

2024-01-30 10:01:05,561 - web_server - INFO - Setting up needed data
2024-01-30 10:01:05,563 - web_server - INFO - Retrieve hass get data method initiated...
2024-01-30 10:01:07,150 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 50, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 112, in set_input_data_dict
    rh.prepare_data(retrieve_hass_conf['var_load'], load_negative = retrieve_hass_conf['load_negative'],
  File "/usr/local/lib/python3.11/dist-packages/emhass/retrieve_hass.py", line 208, in prepare_data
    self.df_final[new_var_replace_zero] = self.df_final[new_var_replace_zero].fillna(0.0)
                                          ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 3767, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/base.py", line 5877, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/usr/local/lib/python3.11/dist-packages/pandas/core/indexes/base.py", line 5938, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['s', 'e', 'n', 's', 'o', 'r', '.', 'p', 'r', 'o', 'd', 'u', 'c', 't',\n       'i', 'o', 'n', '_', 'p', 'v', '_', 'w'],\n      dtype='object')] are in the [columns]"

Rolling back again for the moment.

[GeoDerp]

I wonder if its my mistake with:

       params['retrieve_hass_conf']['var_interp'] = options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_PV']), options.get('sensor_power_load_no_var_loads',params['retrieve_hass_conf']['var_load'])

might need to be:

        params['retrieve_hass_conf']['var_interp'] = [options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_PV']), options.get('sensor_power_load_no_var_loads',params['retrieve_hass_conf']['var_load'])]

Will look over my code and test now

[michaelpiron]

Not sure if that’s the cause, because you didn’t change that part of the code, you just moved the function build_params from web_server.py to utils.py .
And that code worked before.

[michaelpiron]

@GeoDerp in the emhass-add-on code, I see that practically all config/option parameters have been made optional, which seems a huge relaxation.
Is the code equipped to deal with the optionality of all these parameters?

Could the issue come from there?
https://github.com/davidusb-geek/emhass-add-on/blame/262566259b4692b8a9cd961b510895db8a82a05c/emhass/config.yml#L108

[GeoDerp]

The support for optional options come from: Make options optional by GeoDerp · Pull Request #158 · davidusb-geek/emhass · GitHub on EMHASS

It is possible that something bypasses build_params, or there is an error in it.

[davidusb]

@GeoDerp

I wonder if its my mistake with:

       params['retrieve_hass_conf']['var_interp'] = options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_PV']), options.get('sensor_power_load_no_var_loads',params['retrieve_hass_conf']['var_load'])

might need to be:

        params['retrieve_hass_conf']['var_interp'] = [options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_PV']), options.get('sensor_power_load_no_var_loads',params['retrieve_hass_conf']['var_load'])]

Will look over my code and test now

Yes this is a problem, that need to be a list

[davidusb]

@GeoDerp

The support for optional options come from: Make options optional by GeoDerp · Pull Request #158 · davidusb-geek/emhass · GitHub on EMHASS

It is possible that something bypasses build_params, or there is an error in it.

I moved build_params to utils.py and added a unittest to effectively test it.
However as it is a new test not everything is fully tested on that method.

[davidusb]

@GeoDerp

Will look over my code and test now

These two variables should be lists:

        params['retrieve_hass_conf']['var_replace_zero'] = options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_replace_zero'])
        params['retrieve_hass_conf']['var_interp'] = options.get('sensor_power_photovoltaics',params['retrieve_hass_conf']['var_PV']), options.get('sensor_power_load_no_var_loads',params['retrieve_hass_conf']['var_load']

[davidusb]

@markpurcell

I have a few .pkl files, which should I remove?

For me it was just load_forecast_mlf.pkl that was posing problems.
If you erase this then you need to re-fit and tune the ML model again.

[GeoDerp]

Fix optional bugs

Went all the way around the trace to find that one