EMHASS add-on: An energy management optimization add-on for Home Assistant OS and supervised

110hs · May 6, 2024, 8:10pm

There is always the possibility to do things better, but you should keep into consideration that this project is not something commercial and has grown exponentially from the initial build. Actually David has done a terrific job in making this tool available to the community and developing it in what I can assume is his spare time, with the help of just a few great contributors and experienced users available to help and guide EMHASS newbies.
I can share my personal experience and I can tell you it took at least one month to me before I could implement something; I had to read a lot on the forum (here as well) and deeply study the documentation, which has improved a lot over the time.
If you think something should be changed or done differently David is always open to contribution (in the code, in the documentation, …) on the GitHub page of the project.

Coming to your question about the reason why you have to indicate the PV panels and the inverter in that funny way: because at the beginning the tool was making use of a DB developed by a 3rd group and that was the way the devices were recorded. Each string corresponds to a specific model that corresponds to specific electrical parameters.
Indicating your panel and inverter models is needed if you want to use the scrapping method, because it will convert the solar forecast into PV [W] production needed by EMHASS to predict and optimize your battery usage.
In addition to that, as the 3rd party DB was pretty outdated, David kindly implemented a newer and expanded database with more models; to make our life easier he also released a webapp where you can filter the results and look for your model or the one that is the closest to yours. (https://emhass-pvlib-database.streamlit.app/) [doc reference: https://emhass.readthedocs.io/en/latest/forecasts.html]

So do not give up, it’s not something you should expect to have up and running in minutes but not even days. If you have doubts post a message and somebody will try to help

Mjines · May 6, 2024, 9:27pm

I have to be honest. I didn’t expect a reply (and definitely not this fast).
And since I don’t tend to log on often, this would’ve been buried in my messages at some point.

I have looked at your references as well as the webapp. Unfortunately it seems like the manufacturer is not included in the DB, which isn’t too bad now that I got to understand this system a bit more though. I guess I will have to search for a panel with similar electrical characteristics.

I must admit that “at least to me” it would make more sense to skip this “extra step” and just ask the user the required parameters directly. Most people that implement this solution will presumably know their panel name and be able to look up the parameters in the datasheet.

However, I also have to admit that I neither have the knowledge, time nor Endurance to actively develop such a system from scratch. Likely, as you stated, in one’s free time.

So I understand that if I want to use this software, I will have to adapt to the method chosen by the developer(s). I could propose ideas to improve the usability and intuitiveness of the software. But this would require a redesign and reevaluation of the entire project. Which isn’t feasible just for one person who spend just a few hours with this project up until this point. I have to live with what other people deem “a good solution” even if that’s “not a good solution” to other people.

One can’t complain about something for free, which I did out of frustration primarily driven by tiredness (quite late in my Timezone) which still isn’t an excuse, obviously.

I have the habit of implementing solutions in under a day with all my HA related projects. Which worked decently great up until now. This however will likely require reading every single word of the doc (multiple times) to extrapolate the meaning of certain configuration options.

I don’t particularly like this process as it reminds me of (Desktop vs Terminal). The Terminal is more powerful but requires n^10 times more understanding than an intuitive GUI, which takes seconds to do 60-80% the Terminal can.

Usually at this point i would readjust my needs and settle with an overpriced premade solution. But since not a single automation solution exists that ties in greatly with HA or costs 6 figures, this is the best solution.
It will require much more time though than initially anticipated. And considering the “inaccurate” quality of irradiance forecasts with low-frequency updates in “free tiers” the providers grant us, this whole efficiency boost gained becomes marginal at best, presumably.

Will I still do it, absolutely, otherwise I wouldn’t have started HA anyway because the time sunken into it would have sufficed to toggle a light switch manually until I’m a thousand years old.

This seems like a curse, until it works, eventually (hopefully).

Thank you for your kind and fast reply.

110hs · May 6, 2024, 10:10pm

Scrapper works quite well in any case, I think there is a limit on how frequently you can ask the website to elaborate the forecast for your location (from what I understood clearoutside.com generates the forecast for your location upon request, but the site won’t create a new one before one hour or so - I think to keep the burden low) so this is a good starting point to get some forecast coming in and seeing how EMHASS works. Some of us use solcast (but new users have a strict limit on the number of API calls) others (like me) forecast.solar.
Forecast.solar has a free tier that gives you today and tomorrow forecast which in my case is enough, but you have to heavily work with the jinja code to rearrange the forecast buckets, if you plan to run EMHASS computation more frequently than 1h.
If you do not want to do so forecast.solar also has a cheap (14€/year) personal tier with 30 minutes forecast and 3 days forecast but in any case I believe you’ll have to play with the API calls to download the data, format it and pass to EMHASS (I’m not sure this service is fully implemented in EMHASS and you can just pass the APIkey like it seems possible to do with solcast).

So to start with I suggest you try with the scrapper method and see how it goes.
You can try to just insert a random PV and inverter model and then refine it to match yours as much as possible (you see the results after performing a search in the webapp and you can compare the electrical parameters with your inverter).

Argogma · September 30, 2024, 2:42pm

Is it possible with the Add-On to implement the Deferrable load thermal model? I find no input in the Add-On Configuration page.
At the moment i am working with these shell-commands:
trigger_tibber_solcast: “curl -i -H ‘Content-Type: application/json’ -X POST -d ‘{"pv_power_forecast": {{ [states(‘sensor.inverter_power_kombiniert’)|int(0)]| list + (state_attr(‘sensor.solcast_pv_forecast_prognose_heute’, ‘detailedHourly’) | map(attribute=‘pv_estimate’) | map(‘multiply’, 1000) | map(‘round’, 0) | list + state_attr(‘sensor.solcast_pv_forecast_prognose_morgen’, ‘detailedHourly’) | map(attribute=‘pv_estimate’) | map(‘multiply’, 1000) | map(‘round’, 0) | list)[now().hour:][:23] }}, "load_cost_forecast":{{states(‘sensor.tibber_forecast’) }}}’ http://localhost:5000/action/dayahead-optim”

trigger_tibber_solcast_mpc_zwei: “curl -i -H ‘Content-Type: application/json’ -X POST -d ‘{"pv_power_forecast": {{ [states(‘sensor.inverter_power_kombiniert’)|int(0)]| list + (state_attr(‘sensor.solcast_pv_forecast_prognose_heute’, ‘detailedHourly’) | map(attribute=‘pv_estimate’) | map(‘multiply’, 1000) | map(‘round’, 0) | list + state_attr(‘sensor.solcast_pv_forecast_prognose_morgen’, ‘detailedHourly’) | map(attribute=‘pv_estimate’) | map(‘multiply’, 1000) | map(‘round’, 0) | list)[now().hour:][:23] }}, "load_cost_forecast":{{states(‘sensor.tibber_forecast’) }}, "alpha":1, "beta":0,"prediction_horizon":24, "soc_init":{{states(‘sensor.battery_state_of_capacity’)|float(0)/100 }},"soc_final":0.1}’ http://localhost:5000/action/naive-mpc-optim”
now i want to regulate my hot-water tank (200L with heating element). And next month i will get a heat pump, which i want to optimize
Any suggestion how to do this with shell-command or the add-on configuration yaml?

markpurcell · September 30, 2024, 10:06pm

The thermal model works well with the add on, but is not configurable via the GUI, you need to adapt the shell command.

I have started a GitHub discussion here if you would like to share you experiences:

768g · February 11, 2025, 6:43pm

I am getting the following error when installing the v0.12.6 of the add-on on a Raspberry Pi 3B+:

Can’t install ghcr.io/davidusb-geek/emhass:v0.12.6: 404 Client Error for http+docker://localhost/v1.47/images/ghcr.io/davidusb-geek/emhass:v0.12.6/json: Not Found (“No such image: Package emhass · GitHub”)

Is this likely a “me” problem or a GitHub issue?

davidusb · February 17, 2025, 8:17pm

Are you upgrading to v0.12.6?
It depends on your architecture so I can’t say with certitude, but we did dropped support for some old arm architectures that were becoming very hard to maintain

Jan-Willem · February 20, 2025, 11:00pm

Hi all, i am struggling quite a bit with the MPC optimization. The day ahead optimization works fine, but i can’t get the MPC to start running.
I always get an error:

[2025-02-20 23:51:51 +0100] [49] [INFO] Retrieve hass get data method initiated...
[2025-02-20 23:51:51 +0100] [49] [ERROR] Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/web_server.py", line 414, in action_call
    input_data_dict = set_input_data_dict(
                      ^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/command_line.py", line 262, in set_input_data_dict
    if not rh.prepare_data(
           ^^^^^^^^^^^^^^^^
  File "/app/src/emhass/retrieve_hass.py", line 397, in prepare_data
    self.df_final[new_var_replace_zero] = self.df_final[
                                          ^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "[''] not in index"

the curl command supplied is:

curl -i -H 'Content-Type:application/json' -X POST -d '{"pv_power_forecast":[1, 70, 141, 246, 513, 753, 1049, 1797, 1697, 3078], "prediction_horizon":10, "operating_hours_of_each_deferrable_load":2, "start_timesteps_of_each_deferrable_load":0, "end_timesteps_of_each_deferrable_load":0}' http://192.168.178.80:5000/action/naive-mpc-optim

the log states that runtime parameters are parsed correctly, so that doesn’t seem to be it. I also tried several more simple curl commands, e.g. with only 1 deferrable load + prediction horizon only. Also copying examples from the documentation doesn’t seem to work.

Does anyone have any tips or advice how to solve this?

davidusb · February 21, 2025, 6:22am

What is your load power forecast method?
To debug try just passing some dummy values at runtime with the load_power_forecast key. With this we could try identify if there is a problem with that.

Jan-Willem · February 21, 2025, 9:49am

I use a sensor that returns a constant value of the load_power_forecast. This because i have an all electric heatpump which i want to implement as a deferrable load later on.

I’ve tried passing dummy data, unfortunately the cause is somewhere else:

[2025-02-21 10:41:09 +0100] [49] [INFO] Passed runtime parameters: {'load_power_forecast': [1350, 1350, 1350, 1350, 1350, 1350, 1350, 1350, 1350, 1350], 'pv_power_forecast': [1, 70, 141, 246, 513, 753, 1049, 1797, 1697, 3078], 'prediction_horizon': 10, 'operating_hours_of_each_deferrable_load': 2, 'start_timesteps_of_each_deferrable_load': 0, 'end_timesteps_of_each_deferrable_load': 0}
[2025-02-21 10:41:09 +0100] [49] [INFO]  >> Setting input data dict
[2025-02-21 10:41:09 +0100] [49] [INFO] Setting up needed data
[2025-02-21 10:41:10 +0100] [49] [INFO] Retrieve hass get data method initiated...
[2025-02-21 10:41:12 +0100] [49] [ERROR] Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/web_server.py", line 414, in action_call
    input_data_dict = set_input_data_dict(
                      ^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/command_line.py", line 262, in set_input_data_dict
    if not rh.prepare_data(
           ^^^^^^^^^^^^^^^^
  File "/app/src/emhass/retrieve_hass.py", line 397, in prepare_data
    self.df_final[new_var_replace_zero] = self.df_final[
                                          ^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/frame.py", line 4108, in __getitem__
    indexer = self.columns._get_indexer_strict(key, "columns")[1]
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6200, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "/app/.venv/lib/python3.12/site-packages/pandas/core/indexes/base.py", line 6252, in _raise_if_missing
    raise KeyError(f"{not_found} not in index")
KeyError: "[''] not in index"

Compared to the standard installation i have one deviation; use a solcast forecast, which i automatically update in a HA sensor. From this HA sensor i derive another sensor which i write into a csv file. It is that csv file that i use for the day ahead optimization. I wouldn’t exepct this to be the cause of the issue, as (1) the day ahead works fine, and (2) i add dummy variables in the runime. Nevertheless it might be good to know as i assume it is a difference with the majority of running EMHASS configurations…

davidusb · February 21, 2025, 11:42am

Can you please check in your configuration what at the sensors listed under this option: " Sensor to replace NAN values with 0s".
Check if your HA configuration has those sensors available. Otherwise put there the correct names to your sensors.

Jan-Willem · February 21, 2025, 3:00pm

the sensors that are listed there are:

 "sensor_linear_interp": [
    "sensor.afgeleid_vermogen_eigenverbruik"
  ],
  "sensor_power_load_no_var_loads": "sensor.afgeleid_vermogen_eigenverbruik",
  "sensor_power_photovoltaics": "sensor.linenjw_mn0vk8kdahchepydf5ctk9brodxp7p50_realtime_power",
  "sensor_replace_zero": [
    "sensor.linenjw_mn0vk8kdahchepydf5ctk9brodxp7p50_realtime_power",
    "sensor.afgeleid_vermogen_eigenverbruik"
  ],

All sensor’s are available in the HA states section. That being said: there is something peculiar going on with the sensorpower_load_no_var_loads (the “sensor.afgeleid_vermogen_eigenverbruik”). When i check the sensor in states it gives the value 0. However, the sensor is a statistical sensor (helper / sampling size 100, max age 1:30:00, percentile 70, precision 2). The preview of this helper is 1.59.
So apparently this sensor is set to 0 by the mpc, and not(?) by the day ahead optimization. Only thing is i don’t understand why inserting dummy loads doesn’t solve the issue…

Jan-Willem · February 21, 2025, 3:04pm

when running the MPC again after toying around with these sensors (there was a blank line in the EMHASS config, which i removed) i get a different error message now:

[2025-02-21 16:01:08 +0100] [49] [ERROR] Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 1511, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 919, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 917, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/.venv/lib/python3.12/site-packages/flask/app.py", line 902, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/web_server.py", line 472, in action_call
    opt_res = naive_mpc_optim(input_data_dict, app.logger)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/command_line.py", line 663, in naive_mpc_optim
    opt_res_naive_mpc = input_data_dict["opt"].perform_naive_mpc_optim(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/optimization.py", line 1456, in perform_naive_mpc_optim
    self.opt_res = self.perform_optimization(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/app/src/emhass/optimization.py", line 731, in perform_optimization
    if def_total_hours[k] > 0:
       ~~~~~~~~~~~~~~~^^^
TypeError: 'int' object is not subscriptable

So apparantly i am using an integer variable, which causes an error. Should it be a float?

Jan-Willem · February 21, 2025, 3:19pm

got it!
The issue was that in the shell command i didn’t put brackets [ ] around the start timestep and end timesteps of the curl command. This caused it not to be a list…I’ve run the first succesfull MPC optimization now, many thanks for your support!

davidusb · February 21, 2025, 3:31pm

Glad to hear that!

Johan71 · February 23, 2025, 9:28pm

~~Nordpool will switch to a 15-minute price on June 11, otherwise the arrangement will be the same with a 24-hour price, etc. What should be done then with emhass?~~

Already asked the question

tbrasser · February 24, 2025, 5:50pm

I guess format everything into 15 minute, in my case that means going from 24x60min to 96x15min. The per-hour/half-hour json lists can be padded in templates.

768g · March 6, 2025, 6:15pm

Yes, I was on 32 bit (Raspberry Pi 3B+). Will upgrade to 64 bit as soon as I get a chance.

768g · March 20, 2025, 7:44pm

Hi Guys,

Loaded 64 bit HAOS on RPi 3B +
With 0.12.8 picked up the following error:

[2025-03-20 21:13:47 +0200] [56] [INFO] Booting worker with pid: 56
/app/.venv/lib/python3.12/site-packages/pulp/tests/test_pulp.py:1776: SyntaxWarning: invalid escape sequence '\d'
  """
/app/.venv/lib/python3.12/site-packages/pulp/tests/test_pulp.py:1950: SyntaxWarning: invalid escape sequence '\d'
  command_line, option="strong", grp_pattern="\d+"
[2025-03-20 21:15:06 +0200] [56] [INFO] Obtaining parameters from config.json:

Also as the RPi 3 B+ only has 1GB of R.A.M. EMHASS add-on is having difficulty running. Any possibility of modularising it so that it only loads the optimisation for the module selected?

   Building emhass @ file:///app
      Built emhass @ file:///app
Uninstalled 1 package in 38ms
Installed 1 package in 8ms
[2025-03-20 21:22:56 +0200] [22] [INFO] Starting gunicorn 23.0.0
[2025-03-20 21:22:56 +0200] [22] [INFO] Listening at: http://0.0.0.0:5000 (22)
[2025-03-20 21:22:56 +0200] [22] [INFO] Using worker: gthread
[2025-03-20 21:22:56 +0200] [23] [INFO] Booting worker with pid: 23
[2025-03-20 21:24:19 +0200] [22] [ERROR] Worker (pid:23) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:24:19 +0200] [34] [INFO] Booting worker with pid: 34
[2025-03-20 21:25:34 +0200] [22] [ERROR] Worker (pid:34) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:25:35 +0200] [45] [INFO] Booting worker with pid: 45
[2025-03-20 21:26:12 +0200] [22] [ERROR] Worker (pid:45) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:26:12 +0200] [53] [INFO] Booting worker with pid: 53
[2025-03-20 21:27:26 +0200] [22] [ERROR] Worker (pid:53) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:27:26 +0200] [64] [INFO] Booting worker with pid: 64
[2025-03-20 21:28:57 +0200] [22] [ERROR] Worker (pid:64) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:28:58 +0200] [75] [INFO] Booting worker with pid: 75
[2025-03-20 21:30:26 +0200] [22] [ERROR] Worker (pid:75) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:30:26 +0200] [86] [INFO] Booting worker with pid: 86
[2025-03-20 21:31:52 +0200] [22] [ERROR] Worker (pid:86) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:31:52 +0200] [97] [INFO] Booting worker with pid: 97
[2025-03-20 21:33:22 +0200] [22] [ERROR] Worker (pid:97) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:33:23 +0200] [108] [INFO] Booting worker with pid: 108
[2025-03-20 21:34:05 +0200] [22] [ERROR] Worker (pid:108) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:34:06 +0200] [119] [INFO] Booting worker with pid: 119
[2025-03-20 21:35:29 +0200] [22] [ERROR] Worker (pid:119) was sent SIGKILL! Perhaps out of memory?
[2025-03-20 21:35:29 +0200] [130] [INFO] Booting worker with pid: 130
[2025-03-20 21:36:21 +0200] [22] [INFO] Handling signal: term