EMHASS: An Energy Management for Home Assistant

[lordwizzard]

Also python is on either 3.10 or 3.11 currently in the releases. My question is why has yours not updated? The RED line error.

[davidusb]

That error has absolutely nothing to do with a Python version. Python 3.9 is still completely supported but yes we should update to 3.10. Just have to find the time to test that

[markpurcell]

My curl command is getting quite complex to manage:

EMHASS: An Energy Management for Home Assistant

No problem, as soc_final is so far away it has little impact on the initial forecasts, so I set it to zero to see the maximum that EMHASS could return over the full 24 hours. Warning. My full mpc_optimisation call is quite complex and I don’t recommend people start here with this level of complexity. I actually have five different shell commands that I can set from the straight forward day-ahead to the complex MPC, so if things go wrong I can easily roll back to a working configuration. pos…

I have now switched to a dedicated rest_command, which is still complex, but less so I also get errors reported in the logs directly, which is helpful:

rest_command:
  naive_mpc_optim:
    url: http://localhost:5000/action/naive-mpc-optim
    method: POST
    content_type: 'application/json'
    payload: >-
      {
        "prod_price_forecast": {{
          ([states('sensor.amber_feed_in_price')|float(0)] +
          (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list))
          | tojson 
        }},
        "load_cost_forecast": {{
          ([states('sensor.amber_general_price')|float(0)] + 
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list) 
          | tojson 
        }},
        "load_power_forecast": {{
          ([states('sensor.power_load_no_var_loads')|int] +
          (states('input_text.fi_fo_buffer').split(', ')|map('multiply',1000)|map('int')|list)[1:]
          )| tojson 
        }},
        "pv_power_forecast": {{
          ([states('sensor.APF_Generation_Entity')|int(0)] +
          state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list +
          state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list
          )| tojson
        }},
        "prediction_horizon": {{
          min(48, (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
        }},
        "alpha": 1,
        "beta": 0,
        "num_def_loads": 6,
        "def_total_hours": {{[states('sensor.def_total_hours_pool_filter')|int(0),
                              states('sensor.def_total_hours_pool_heatpump')|int(0),
                              states('sensor.def_total_hours_ev')|int(0),
                              states('sensor.def_total_hours_hvac')|int(0),
                              states('sensor.def_total_hours_hws')|int(0),
                              states('input_number.car2_def_total_hours')|int(0)]}},
        "P_deferrable_nom": [1300, 5000, 11500, 4000, 600, 3500],
        "treat_def_as_semi_cont": [1, 1, 0, 0, 1, 0],
        "set_def_constant": [0, 0, 0, 0, 0, 0],
        "soc_init": {{ (states('sensor.filtered_powerwall_soc')|int(0))/100 }},
        "soc_final": 0.0
      }
  publish_data:
    url: http://localhost:5000/action/publish-data
    method: POST
    content_type: 'application/json'
    payload: '{}'

[davidusb]

Done! Python 3.10 is now supported.
Possible Python versions are 3.8, 3.9 and 3.10

[rcruikshank (Robert Cruikshank)]

Hi
Trying to set up EMHASS but finding the instructions difficult. Would be very appreciative of some help.

• I have 5kW PV.
• A Fronius Primo 6.0-1 inverter.
• I’m using solcast to predict PV production.
• A 10 kWh (9 kWh usable) sonnen eco 9.43 battery.
• I use the Sonnenbatterie HACS addon by weltmeyer to obtain all the battery metrics.
• I can use POST to switch the battery between charge, discharge, self consumption and TOU modes and probably others that I haven’t discovered yet.
• I’m on the Australian Amber Electric service (They don’t support Sonnen).
• I’m using the Amber Electric integration in HA to get the pricing data etc.
• Deferable loads might be pool pump, dish washer, dryer, washing machine but I currently don’t monitor power consumption for any of these loads.
• I have one EV, Tesla model Y.
• I’ve been using Charge HQ to divert feed-in to the car but suspect I can replace that with EMHASS.

I’ve set up the following in HA:

• Automation

- alias: EMHASS day-ahead optimization
  trigger:
    platform: time
    at: 05:30:00
  action:
  - service: shell_command.dayahead_optim
- alias: EMHASS publish data
  trigger:
  - minutes: /5
    platform: time_pattern
  action:
  - service: shell_command.publish_data

• configuration.yaml

shell_command:
  dayahead_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim"
  publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"
  post_amber_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":{{(
          state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
          }},\"load_cost_forecast\":{{(
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
          }},\"prediction_horizon\":33}' http://localhost:5000/action/dayahead-optim"
  post_emhass_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":{{(
          state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
          }},{{states('sensor.solcast_24hrs_forecast')}},\"load_cost_forecast\":{{(
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
          }}}' http://localhost:5000/action/dayahead-optim"
  post_mpc_optim_solcast: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{(
          ([states('sensor.amber_general_price')|float(0)] +
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:48])
          }}, \"prod_price_forecast\":{{(
          ([states('sensor.cecil_st_feed_in_price')|float(0)] +
          state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)[:48]) 
          }}, \"pv_power_forecast\":{{states('sensor.solcast_24hrs_forecast')
          }}, \"prediction_horizon\":48,\"soc_init\":{{(states('sensor.sonnenbatterie_84324_state_charge_user')|float(0))/100
          }},\"soc_final\":0.05,\"def_total_hours\":[2,0,0,0]}' http://localhost:5000/action/naive-mpc-optim"

and

# Solar forecast for EMHASS
  - platform: rest
    name: "Solcast Forecast Data"
    json_attributes:
      - forecasts
    resource: https://api.solcast.com.au/rooftop_sites/YYYYYYYYYYYYY/forecasts?format=json&api_key=XXXXXXXXXXXXXXXXXXXXXXXXXX&hours=24
    method: GET
    value_template: "{{ (value_json.forecasts[0].pv_estimate)|round(2) }}"
    unit_of_measurement: "kW"
    device_class: power
    scan_interval: 8000
    force_update: true

  - platform: template
    sensors:
      solcast_24hrs_forecast :
        value_template: >-
          {%- set power = state_attr('sensor.solcast_forecast_data', 'forecasts') | map(attribute='pv_estimate') | list %}
          {%- set values_all = namespace(all=[]) %}
          {% for i in range(power | length) %}
          {%- set v = (power[i] | float |multiply(1000) ) | int(0) %}
          {%- set values_all.all = values_all.all + [ v ] %}
          {%- endfor %} {{ (values_all.all)[:48] }}

I assume I have to change the automation to also call the amber, EMHASS and solcast curl shell commands? But not sure if this replaces the default dayahead_optim and publish_data curl commands.

Haven’t even looked into what I do with the returned results from EMHASS. Just wanted to get this first stage set up correctly.

Do I need to measure deferable loads so I can subtract them from house consumption figure I get from battery? If so, I’ll have to go out and gets some power points that monitor power.

Am I going about this in the right way. What changes do I need to make and what next.

Thanks in advance.

[markpurcell]

Looks like you have a lot of components.

I would start with a basic system.

If you can control your battery charge/ discharge via a post command, then you should be able to get the same level of functionality as SmartShift.

Your EV and pool pump will also integrate well and you should be able to get excellent optimisation results. You will need a smart socket for the pool but the EV should be able to integrate directly.

You only need to call one optimisation and one plublish content, the others are just spending that at in additional functionality.

As it is setup it will call the simple dayahead once a day at 0530 optimisation which is good and then publish these results every 5 minutes.

You can find my post here on how you can connect other devices:
Running Devices When Energy is Cheaper and Greener

[Octofinger (Ivar Edebro)]

@davidusb did you have a chance to look at the errors I posted above in EMHASS: An Energy Management for Home Assistant - #728 by Octofinger?
Seems emhass is choking on microseconds in HA sensor timestamps.

[davidusb]

You have a problem retrieving your data from Home Assistant but with the given information I’m not able to reproduce that. I’m able to perfectly retrieve data from HA using the legacy Python code on a virtual environment, that’s proven to work so there must be a problem with your configuration.
You can maybe test using the standalone docker container, it is more practical to find the issue.
Otherwise open a new issue on github providing a way to reproduce the error.

[rcruikshank (Robert Cruikshank)]

Yes, I do have a basic system in place. I migrated to Node-Red about 4 years ago for automations.

• Discharge battery when FiT goes above 51c and return to self-consumption when FiT goes below 49c
• Charge the battery at 05:30 for a half hour if the amber general price is low and the battery is below 15% SoC
• Charge the battery at 15:30 for an hour if the amber general price is low and the battery is below 80% SoC
• Pool pump only comes on if the battery is 100% SoC and there is at least 1.5 kW PV for limited time depending on season
• Turn on aircon and pool pump when FiT goes negative and battery is 100% SoC or PV Feed-in is positive

I don’t usually charge the car at home as there are too many free chargers located around me to bother, although that’s getting difficult with more and more EVs on the road.

Difficult limiting use of other loads (washing machines etc.) as people will use them whenever they want despite my focus on energy consumption, but I’m willing to try.

So, my main aim is to see what EMHASS will deliver to the logic. But it’s difficult to configure without reading through pages of hard to understand explanations of control systems. I don’t really need to know the details of MPC etc, just the different configuration steps to achieve the different modes of operation.

[Octofinger (Ivar Edebro)]

Just to make sure, you also get microseconds from HA in last_changed and last_updated timestamps? Or is it just me?
I’ll give the docker version a try.
Thanks for helping out!

[davidusb]

No, I’ve never had these problems with microseconds. Your problem is with your configuration.
In the --config argument you need to pass the path with the yaml file included like this:

emhass --action 'dayahead-optim' --config './config_emhass.yaml' --costfun 'profit'

[Octofinger (Ivar Edebro)]

That’s what I find so hard with the documentation. The documentation is clear, but the application doesn’t behave as the documentation states. If I run the command above, I get the below. Apparently, the application takes the --config argument as a path to the folder where a config.yaml is located.

> emhass --action 'dayahead-optim' --config './config_emhass.yaml' --costfun 'profit'
C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\pvlib\forecast.py:20: UserWarning: The forecast module algorithms and features are highly experimental. The API may change, the functionality may be consolidated into an io module, or the module may be separated into its own package.
  warnings.warn(
2023-06-13 11:07:35,394 - emhass.command_line - INFO - Setting up needed data
INFO:emhass.command_line:Setting up needed data
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Scripts\emhass.exe\__main__.py", line 7, in <module>
  File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\command_line.py", line 182, in main
    input_data_dict = setUp(config_path, costfun, logger)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\command_line.py", line 30, in setUp
    retrieve_hass_conf, optim_conf, plant_conf = get_yaml_parse(config_path)
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Ivar\Documents\Programmering\Python\emhass\emhassenv\Lib\site-packages\emhass\utils.py", line 70, in get_yaml_parse
    with open(config_path + '/config.yaml', 'r') as file:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: './config_emhass.yaml/config.yaml'

[davidusb]

Do you have the same result if you provide and absolute path instead of a relative path like this?

emhass --action 'dayahead-optim' --config '/home/user/emhass/config_emhass.yaml' --costfun 'profit'

[sladey]

Hi David,

After a HASS restart, I am seeing the following error when a run the predict and publish for the new ML Forcaster.

2023-06-14 08:45:33,181 - web_server - INFO - Setting up needed data
2023-06-14 08:45:33,182 - web_server - INFO - Retrieve hass get data method initiated...
2023-06-14 08:45:34,437 - web_server - INFO -  >> Performing a machine learning forecast model predict...
2023-06-14 08:45:34,442 - web_server - ERROR - Exception on /action/forecast-model-predict [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 216, in action_call
    df_pred = forecast_model_predict(input_data_dict, app.logger)
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 361, in forecast_model_predict
    predictions = mlf.predict(data_last_window)
  File "/usr/local/lib/python3.9/dist-packages/emhass/machine_learning_forecaster.py", line 210, in predict
    predictions = self.forecaster.predict(steps=self.num_lags,
  File "/usr/local/lib/python3.9/dist-packages/skforecast/ForecasterAutoreg/ForecasterAutoreg.py", line 637, in predict
    check_predict_input(
  File "/usr/local/lib/python3.9/dist-packages/skforecast/utils/utils.py", line 620, in check_predict_input
    raise ValueError(
ValueError: `last_window` has missing values.

I have run the “fit” and “tune” steps. Any thoughts? I am running 0.3.13 of the add on.

[davidusb]

Hi. I will need to test this but I see why the fit and tune methods worked and not the prediction. This is because there are NaN’s in your data probably caused by the HA restart and they are not treated for the prediction and the last window of data needed for that prediction. They are only treated for fit and tune. This should be fixed and a linear interpolation is a good solution for small windows of missing values. However if you have too many missing values it can lead to unexpected results. But I guess is the best we can do in that case.
It should be fixed if you wait for correct non missing data for at least the number of lags used by your predictor, ex. 48h, this value is shown in the logs when launching the fit and tune methods.
Could you please open an issue on github to fix this?

[torstein]

Just a question about the forecast, almost every day is spot on exacly correct forecast. But if there are som sky or rain it miss a lot like this.

The solarpanel that is configured in emhass is:
Phono_Solar_Technology_Co__Ltd__PS360MH_24_T, Dont understand where to find these list in the documentation for other 360W mono crystal panels. if we could try replacing with another panels see if results will be more close on all days?

[sladey]

'last_window' has missing values error after HASS restart

thanks David.

[davidusb]

The first thing to do is to know what is your forecast method?

[torstein]

day ahead optim

[davidusb]

What is your PV forecast method? Web scrap? Solcast? Solar.Forecast? Your own method and passing a list of values?