EMHASS: An Energy Management for Home Assistant

[adammcdonagh]

Are you still using solcast to forecast solar? And, are you calling the EMHASS API to update the forecast throughout the day/night?
If so, how are you handling passing in the correct number of samples to ensure your end battery SoC is correctly calculated? It looks like your graph extends for 24hrs into the future from when you took the screenshot.

At the moment I’m using the following, inspired by the docs, which appear to have been based on your post above in the first place. However I’m only calling this once a day (at 23:30), when my lower import rate starts. Since I don’t get any money for exporting, I’m storing all that I can use, and having to export the rest… This is today for example:

I’d like to keep the solar forecast up-to-date. It looks like you are still plotting a 24hr window constantly? Or are you also only calling it once?

Here’s my current config API calls:

publish_data: 'curl -i -H "Content-Type:application/json" -X POST -d ''{}'' http://localhost:5000/action/publish-data'
post_mpc_optim_solcast: 'curl -i -H "Content-Type: application/json" -X POST -d ''{"prediction_horizon":48,"pv_power_forecast":{{ states("sensor.solcast_24hrs_forecast") }}, "soc_init":{{ (states("sensor.battery_soc")|float(0))/100 }}}'' http://localhost:5000/action/naive-mpc-optim'

The optimisation call is run at 2330, and the publish just after that.

My EMHASS config uses the following for the SoC defaults:

battery_minimum_state_of_charge: 0.3
battery_maximum_state_of_charge: 0.9
battery_target_state_of_charge: 0.35

Also, I assume you’re doing something clever with your car, to ensure that it doesn’t drain from the powerwall (or maybe Tesla just does that all for you). I’ve also got an EV, but since we aren’t charging every day, I’ve set the deferrable load hrs to 0, so that it doesnt show up. At present I’m just changing my battery state to not discharge whenever the car is charging (it’s an MG4 with an Ohme charger, so I’ve got no smart control over it via Home Assistant)

Thanks!

[markpurcell]

@adammcdonagh

Are you still using solcast to forecast solar? And, are you calling the EMHASS API to update the forecast throughout the day/night?
If so, how are you handling passing in the correct number of samples to ensure your end battery SoC is correctly calculated? It looks like your graph extends for 24hrs into the future from when you took the screenshot.

Yes, I continue to employ Solcast for solar predictions, and I am indeed invoking the EMHASS API to update these forecasts around the clock.

I’m implementing a complex template formula for the PV forecast that leverages Solcast, as illustrated in the YAML code below:

\"pv_power_forecast\":{{[states('sensor.APF_Generation_Entity')|int(0)]
            + state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list 
            + state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list}}

[states('sensor.APF_Generation_Entity')|int(0)]

Live Data

[4136]

This data is the first entry since I’m running the EMHASS MPC optimization every minute. Therefore, I use the live data for all my forecast variables (like load, price, cost, and PV). If you’re implementing a daily optimization process, injecting live data isn’t necessary.

Today’s Forecast

+ state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list 

• detailedForecast attribute returns the 30 minute interval forecasts from solcast_forecast_today
• period_start greaterthan utcnow() ensures I only return future forecasts
• extract only the pv_estimate which is the forecast energy (kWh) over the 30 minutes
• multiply by 2000 to convert forecast energy (kWh) into the 30 minute average power (W) required by EMHASS

Tomorrow’s Forecast

+ state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list

Add in forecast_tomorrow in case forecast_today after utcnow is less than the required 48

[markpurcell]

@adammcdonagh

Also, I assume you’re doing something clever with your car, to ensure that it doesn’t drain from the powerwall (or maybe Tesla just does that all for you). I’ve also got an EV, but since we aren’t charging every day, I’ve set the deferrable load hrs to 0, so that it doesnt show up. At present I’m just changing my battery state to not discharge whenever the car is charging (it’s an MG4 with an Ohme charger, so I’ve got no smart control over it via Home Assistant)

I’m doing three things with my car, unfortunately charging the car still drains the household battery by default, even with all Tesla. To overcome this I have an automation to set the Powerwall backup reserve to the SOC_forecast from EMHASS when charging the EV. The Powerwall won’t discharge below this backup reserve so when I’m charging my EV it pulls from excess solar and the grid, which is what I want.

The second thing I do is have a complex template to calculate the def_hours for EV charging to inject into EMHASS.

    - name: def_total_hours_ev
      state: "{{is_state('automation.p_deferable2_automation','on')|abs
             * (is_state('device_tracker.duka_location_tracker','home')|abs) 
             * (is_state('binary_sensor.duka_charger', 'on') | abs) 
             * (((states('number.duka_charge_limit')|int(0) - states('sensor.duka_battery')|int(0))/10+0.9)|int(0))|int(0)}}" 

Here’s a more detailed breakdown of the calculation:

1. is_state('automation.p_deferable2_automation','on')|abs: This checks if the automation p_deferable2_automation is currently on. This confirms that I actually have my EV charging automation switched on.
2. is_state('device_tracker.duka_location_tracker','home')|abs: This checks if the device duka_location_tracker is currently at home. This confirms the car is at home.
3. is_state('binary_sensor.duka_charger', 'on') | abs: This checks if the binary sensor duka_charger is currently on. This confirms the location tracker for the car is home.
4. (((states('number.duka_charge_limit')|int(0) - states('sensor.duka_battery')|int(0))/10+0.9)|int(0))|int(0): This calculates the number of hours needed to charge the EV based on the current battery level (sensor.duka_battery) and the charge limit (number.duka_charge_limit). The difference between the charge limit and the current battery level is divided by 10 (because the charger adds 10 units of charge per hour), then rounded up to the nearest whole number by adding 0.9 and converting to an integer.

The result of each of these four parts is then multiplied together to get the final state of the def_total_hours_ev sensor. If any of the conditions are not met (i.e., the automation is off, the device tracker is not at home, or the charging cable is not plugged in), then the state will be 0. Otherwise, it will be the calculated number of hours needed to charge the EV to the specified limit.

The final piece is to set the EV charging_amps to match the EMHASS forecast for EV charging (deferrable load 2).

[adammcdonagh]

Thank you! I appreciate the detailed explaination

[adammcdonagh]

One question outstanding I think, was about the final battery SoC. If you’re calling the model evey minute, what are you doing for the soc_final value? Surely if you’re forecasting the next 24hrs every minute, that end SoC value will be all over the place?

Can I see your current shell command template that you’re using to call the EMHASS API please?

Thanks

[markpurcell]

No problem, as soc_final is so far away it has little impact on the initial forecasts, so I set it to zero to see the maximum that EMHASS could return over the full 24 hours.

Warning. My full mpc_optimisation call is quite complex and I don’t recommend people start here with this level of complexity. I actually have five different shell commands that I can set from the straight forward day-ahead to the complex MPC, so if things go wrong I can easily roll back to a working configuration.

  post_mpc_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{(
          ([states('sensor.amber_general_price')|float(0)] +
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:48])
          }}, \"load_power_forecast\":{{
          [states('sensor.power_load_no_var_loads')|int] +(states('input_text.fi_fo_buffer').split(', ')|map('multiply',1000)|map('int')|list)[1:]
          }}, \"prod_price_forecast\":{{(
          ([states('sensor.amber_feed_in_price')|float(0)] +
          (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)))
          }}, \"pv_power_forecast\":{{[min(15000,(states('sensor.APF_Generation_Entity')|int(0)
                                      /(states('sensor.solaredge_i1_active_power_limit')|int(0)
                                        +states('sensor.solaredge_i2_active_power_limit')|int(0))*200)|int(0))]
            + state_attr('sensor.solcast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list 
            + state_attr('sensor.solcast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow())
              | map(attribute='pv_estimate')|map('multiply',2000)|map('int')|list
          }}, \"prediction_horizon\":{{min(48,
          (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
          }}, \"alpha\":1, \"beta\":0, \"soc_init\":{{(states('sensor.filtered_powerwall_soc')|int(0))/100
          }}, \"soc_final\":0.0, \"def_total_hours\":[{{states('sensor.def_total_hours_pool_filter')
          }},{{states('sensor.def_total_hours_pool_heatpump')
          }},{{states('sensor.def_total_hours_ev')
          }},{{states('sensor.def_total_hours_hvac')
          }},{{states('sensor.def_total_hours_hws')
          }}]}' http://localhost:5000/action/naive-mpc-optim"

Here’s a breakdown of each part of the JSON payload:

1. load_cost_forecast: This is a list of the current electricity price from the ‘amber_general_price’ sensor, plus a list of future prices extracted from the ‘amber_general_forecast’ sensor. The list is trimmed or extended to have exactly 48 elements.
2. load_power_forecast: This is a list starting with the current load from the ‘power_load_no_var_loads’ sensor, followed by values extracted from the ‘fi_fo_buffer’ input text, which are multiplied by 1000 and converted to integers.
3. prod_price_forecast: This is a list of the current feed-in electricity price from the ‘amber_feed_in_price’ sensor, plus a list of future prices extracted from the ‘amber_feed_in_forecast’ sensor.
4. pv_power_forecast: This starts with the current generation from the ‘APF_Generation_Entity’ sensor, normalized to a maximum of 15000 and a minimum of 0. This is followed by the solar forecasts for today and tomorrow extracted from the ‘solcast_forecast_today’ and ‘solcast_forecast_tomorrow’ sensors, which are multiplied by 2000 and converted to integers.
5. prediction_horizon: This is the minimum of 48 and one more than the number of forecasts in the ‘amber_feed_in_forecast’ sensor.
6. alpha: This is a fixed value of 1.
7. beta: This is a fixed value of 0.
8. soc_init: This is the current state of charge of the Powerwall, obtained from the ‘filtered_powerwall_soc’ sensor and divided by 100.
9. soc_final: This is a fixed value of 0.0.
10. def_total_hours: This is a list of values obtained from the ‘def_total_hours_pool_filter’, ‘def_total_hours_pool_heatpump’, ‘def_total_hours_ev’, ‘def_total_hours_hvac’, and ‘def_total_hours_hws’ sensors.

[adammcdonagh]

Thanks again

[markpurcell]

Minimal configuration to inject Amber Electric prices:

shell_command:
  dayahead_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim"
  publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data "

  post_amber_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":{{(
          state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
          }},\"load_cost_forecast\":{{(
          state_attr('sensor.amber_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
          }},\"prediction_horizon\":33}' http://localhost:5000/action/dayahead-optim"

Which should return something like:

  post_amber_forecast: "curl -i -H 'Content-Type: application/json' -X POST -d '{\"prod_price_forecast\":[-0.04, 0.01, 0.02, 0.04, 0.03, 0.07, 0.14, 0.16, 0.21, 0.4, 0.34, 0.33, 0.19, 0.19, 0.19, 0.17, 0.16, 0.1, 0.12, 0.14, 0.14, 0.12, 0.1, 0.1, 0.1, 0.07, 0.07, 0.07, 0.07, 0.07, 0.05, 0.06, 0.08, 0.1, 0.13, 0.44, 0.37, 0.29, 0.09, 0.07, 0.07, 0.04, 0.02, 0.02, 0.0, -0.01, -0.01, 0.01],\"load_cost_forecast\":[0.03, 0.08, 0.1, 0.12, 0.11, 0.15, 0.23, 0.25, 0.31, 0.51, 0.45, 0.44, 0.28, 0.29, 0.29, 0.26, 0.25, 0.18, 0.22, 0.24, 0.24, 0.22, 0.19, 0.2, 0.19, 0.16, 0.16, 0.16, 0.16, 0.16, 0.14, 0.15, 0.18, 0.19, 0.22, 0.57, 0.48, 0.39, 0.18, 0.15, 0.15, 0.12, 0.1, 0.1, 0.08, 0.07, 0.07, 0.08],\"prediction_horizon\":33}' http://localhost:5000/action/dayahead-optim"

Try this on your command line to see if it is working correctly.

[adammcdonagh]

Bit of a weird one today… I’m assuming this is a bug.
The max charge rate of my battery is set to 3300W. However the model today seems to be suggesting a charge rate of 5.3KW

I’m passing in a SoC of 0.11 (which is what it is), but the graph is refusing to believe me and is showing 0.3 instead:

My guess is that it doesn’t expect the battery to every be below 30% (my settings). However in reality today it discharged beyond 30%, even though I don’t want EMHASS to allow this as part of the prediction

Battery config is:

set_use_battery: true
battery_discharge_power_max: 3000
battery_charge_power_max: 3300
battery_discharge_efficiency: 0.95
battery_charge_efficiency: 0.95
battery_nominal_energy_capacity: 12000
battery_minimum_state_of_charge: 0.1
battery_maximum_state_of_charge: 0.9
battery_target_state_of_charge: 0.35

And my new shell command:

curl -i -H "Content-Type: application/json" -X POST -d ''{"prediction_horizon":{{ state_attr("sensor.populate_solcast_forecast", "steps_left") }},"pv_power_forecast":{{ states("sensor.solcast_24hrs_forecast") }}, "soc_init":{{ (states("sensor.battery_soc")|float(0))/100 }}}'' http://localhost:5000/action/naive-mpc-optim

[davidusb]

Check the logs for the optimization status. Maybe you are falling under an infeasible condition due to that low initial SOC? Check that status in the logs. If the optimization is feasible then all your constraints should be respected, including those SOC min/max constraints.

[adammcdonagh]

Yep, I think it did say infeasible. What’s the suggested way to handle this?

[torstein]

tried to activate no_discharge_to_grid:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 331, in <module>
    params = build_params(params, options, args.addon)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 125, in build_params
    params['optim_conf'][19]['set_nodischarge_to_grid'] = options['set_nodischarge_to_grid']
IndexError: list index out of range

Not able to open web gui after this, not able to revert config to get it working again.

this is after upgrading to 11

[davidusb]

Yes there were some issues on the latest release. They are solved now. Search for updates and install the latest version.

[Remco_Ploeg]

I had problem with starting the service since a couple of days. So I started from clean, only changed the long live token with a correct one and tried to run the service with the standard values in the configuration file. But still receive these kinds of errors:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun emhass (no readiness notification)
s6-rc: info: service legacy-services successfully started
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 4 (char 3)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 314, in <module>
    config_hass = response.json()
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Extra data: line 1 column 4 (char 3)

Any idea what I am doing wrong? Used the standard model and inverter in the standard config file and didn’t changed anything.

[davidusb]

The first step is to understand why it is not converging. Sometimes it is just one of the input data that didn’t have correct values. On occasions with the passed input values we are able to conclude that the problem is truly infeasible. For example if you have not much PV production and you are demanding your battery to store energy but in fact there is not enough energy to store. You can maybe do these check ups and try to understand the root cause. Most of the time we are able to explain infeasible conditions.

[davidusb]

Are you using the add-on? If this is the case there is no need for the long lived access token. Or are you trying to set the docker standalone mode ?

[Octofinger (Ivar Edebro)]

I’m getting so stuck when trying to find my inverter and panel modules in PVLib. The config docs link to a document from 2019… I’m feeling rather stupid right now.
I’ve got JA Solar JAM66S30 modules that are 500W each.
The inverter is a Sungrow SH10RT with the small SBR096 (9.6kWh) battery.
Should I just grab whatever and then use SolCast or just pass the PV estimates myself to the service? I find it a bit problematic that the service relies on a list of available hardware in such a rapidly changing marked as home PV systems is today. I’d rather just code the required parameters of my installation.

How do I get out of this trouble?

[davidusb]

EHMASS does not “relies” on this list of outdated hardware.
That’s why there are many option to provide your PV estimates:

• You have the PVLib option with the “scrapping” method which is great because the PV modelling is very detailed using PVLib. But I agree the database is a bit outdated. If you can’t find a suitable hardware for you then just simply don’t use this option. Move on to the next option.
• Pass your estimates as lists of values. You will need to format your list using templates. See the docs.
• There is also the option to use Solcast and Forecas.Solar directly from inside EMHASS without passing any list of values. These services are fully implemented. The problem with these will be that if you are already using them on your HA instance (for example for the energy dasboard), then you will have problems with the limited number of API calls per day. But if this is not a problem then use these.

[markpurcell]

Piece of art!

What EMHASS was forecasting for my afternoon optimisation:

What my devices reported as actual power consumption:

[davidusb]

Very impressive!
These machine learning algorithms are magic. And it is good to note that we are not using neural netwroks or anything more complicated than that, just some classic machine learning algorithms