EMHASS: An Energy Management for Home Assistant

[markpurcell]

Today’s plan. Feed-in price stays above 60¢/ kWh all day (peaks at $15/ kWh tonight), which is higher than the 40¢ / kWh general price tomorrow morning!

EMHASS daily optimisation is saying to shift all deferrable loads to tomorrow to maximise battery charging (to export over tonight’s peak @ $10-$15) and maximise exports during the day @ 60¢.

A lot of interest in EMHASS from the Australian Amber users facebook group as we try and optimise our way through a crazy electricity market. Redirecting...

[davidusb]

Nice discussion. I’m currently working on a hopefully better load forecast module. I’m a data scientist, so now that EMHASS seems quite stable the next step is to improve those forecast. The naive persistence approach is basic but robust. I’m testing to see what kind of model could we use to obtain better load forecasts, better than the naive method. This is currently narrowed to RNN (Recurrent neural networks) and LSTM ( Long short-term memory) models. They seem to give some nice results. I’m working on this right now.

[kc_au (Kel)]

@davidusb A small feature request would be to put totals on the columns:
cost_profit cost_fun_profi
or like a 24 hour forecast of cost/profit so you can see how the model is going to perform as a whole for the period.

[davidusb]

Hi, yes this can be done. An additional table that sums up the results shown in the table. The total cost is already been printed on the add-on logger though. This result is printed every time that an optimization routine is launched.

[markpurcell]

That m it’s would be useful to have the sum of cost brought through into HA, or just bring cost_profit and others with attributes, when could then be summed inside HA.

apexcharts will sum data series, so it would be good to put in the header.

Could I also ask you consider bringing the other variables across, like SOC and p_grid as that can help with control.

[davidusb]

This can be done for p_grid.

For the SOC this is already done. Aren’t you able to see sensor.soc_batt_forecast with its future values as attributes?

[davidusb]

Advanced a bit on new load forecasting methods.

Here is a quite decent result for the N-BEATS model:

[markpurcell]

The pattern of life.

Interesting to see how each day is similar but different.

Of course the Mon-Fri pattern is often similar but different to the Sat-Sun pattern, so it would be interesting to see how the forecasting model accommodates that.

My observation of the current EMHASS forecasting is that not surprisingly if yesterday was abnormal (ie higher or lower consumption) that then flows into the current optimisation. Your approach with N-BEATS and other advanced models will address this.

One suggestion from me is to include the now actual power load value into the modelling. (sensor.power_load_no_var_loads)

As you are aware I am running the mpc optimisation updated every minute. Into that model I am injecting the buy/ sell values, and using method_ts_round first, I include the current now value plus the subsequent forecast values for future timeslots.

Given EMHASS already had access to the now values for load (sensor.power_load_no_var_loads) and production (sensor.power_photovoltaics), it would seem to make sense when using method_ts_round first to use the actual now values when developing the forecasts.

In practice I would see these now values replacing the now values in any EMHASS forecast and subsequent calculations. For example I can see in the optimisation results many examples when EMHASS sets the battery power value to match baseload or PV production values, to produce a net zero effect. Pvlib at that instant maybe forecasting 3000W production and then EMHASS calculates p_batt_forecast to -3000W to balance. But sensor.power_photovoltaics may be 2000W, thus p_batt_forecast should be calculated to match at -2000W, because if we set to -3000W we need to draw that additional 1000W power from the grid.

I am aware I can inject the now values myself, as I am doing with the buy/ sell values. But if I generate my own forecasts for PV production including the now value, I loose access to the benefits of pvlib and if I generate my own load forecasts including the now value I would loose access to N-BEATS or other advanced modelling.

I hope that makes sense, but happy to discuss further if useful.

[markpurcell]

Oh, sorry I missed the inclusion of that one.

I am finding control of the battery a frustrating issue, nothing to do with EMHASS.

Looking at Home Solar battery systems like the Victron @kc_au it seems you can send the desired battery control value in W directly - so p_batt_forecast from EMHASS can be sent directly to the Victron and it will match for both charging and discharging.

I have a Tesla Powerwall2 and monitoring and control via the undocumented API is somewhat limited. I’m using the excellent teslapy library which provides the best level of control possible.

The battery is capable of charging/ discharging at +/- 5 kW, but getting it to achieve this on command is difficult.

I can set backup mode which sets charging from the grid at 3.3 kW (usually) which is a close proxy for charging on demand, albeit a bit slower than max. So when p_batt_forecast is below -3300 I can switch to backup mode and when it is above -3300 I switch to self_consumption mode.

In self_consumption mode excess solar (upto 5 kW) is stored in the battery and excess load (upto 5 kW) is provided by the battery. So this is a great mode to balance peaks and troughs.

I can also set backup_reserve as desired % minimum charge. I can easily take sensor.soc_batt_forecast and use that to set backup_reserve. When this value is over the actual SOC the battery will charge from the grid, when this value is below SOC the battery will discharge to meet household load only, but only down to backup_reserve. I could set backup_reserve after every optimisation and that would be partially effective. But wouldn’t support the peaks and troughs, as the battery would not discharge below backup_reserve.

So what about discharging to the grid?
Possible, but complex.

I can set autonomous mode on the battery that takes into consideration history of solar production and load and a set time based control schedule with windows for super_offpeak, offpeak, mid_peak and peak. All with set buy/ sell prices for each window.

The general strategy the battery follows is to ensure there is sufficient battery charge to cover peak, by charging during offpeak. To maximise returns this includes discharging the battery (5 kW to the grid) during high price sell windows (peak) and charging the battery (5 kW from the grid) during low cost buy windows (off-peak).

I can switch to autonomous mode via a script, but cannot (as yet) use a script to change the windows or prices.

Now of course EMHASS is more flexible and might want to charge from the grid at anytime conditions are right, especially when optimising for wholesale pricing that changes continuously.

Typically the times we want to discharge to the grid are those times when the grid is under extreme demand, needs additional capacity and thus the wholesale price shoots up (in my case upto $13/ kWh) to drive additional supply to meet that demand. Generally the sunrise and sunset morning and evening peaks.

I have set in the battery high price peak windows for 0600-0700 & 1700-1800 and my automation is if p_batt_forcecast is greater than 4000W and it is between these hours I switch to autonomous mode which sets the battery to discharge (5 kW to the grid) during the times of most need and afterwards I switch back to self_consumption mode.

So possible, but complex with a Powerwall2. I guess they are trying to limit over-cycling the battery. But the Victron style level of control does have its advantages.

[kc_au (Kel)]

@markpurcell I set my Grid Set Point to a value in W. So if I set it to -5000, the Victron inverter will target export 5000 W. if my house consumption is 2000W, the inverter will output 7000W to meet its target. 0 = self consume. + 5000 would be take 5000 from grid, and excess into battery, shortage from battery.

Just confirmation on how the Victron works. What I cant do, is set the Victron to export 3kw from battery, ignore house loads. and it takes 5-6 seconds to get data, and then 5-6 seconds to push, so I cant update it live.

Pros and Cons of each. I am struggling with visualising EMHASS data atm., but once I can see the data that EMHASS is asking my batt to do, I can do the automations. Wether I take the value from EMHASS direct, or apply some logic/rules to it. Unsure at this point.

Loving what I am seeing so far though!

[davidusb]

The more advanced models should adapt very well to the hour of the day and the day of the week (weekdays, weekends, etc). This is because I am using that information as encoded covariate inputs to the models. This is very interesting in deed as these values are perfectly known in the future. The overall quality of the forecast will also depend on the quantity of data used when training the model. We should provide as much data as possible. This is multivariate, so l am also adding as inputs the historical values of weather data for example. Something I’ve not done yet is to also provide occupation data, but I will do in the future. This is looking great. I am still thinking if this should be a complete new add-on or if I will directly integrate this in emhass. The problem that I see is that the docker image will be bigger and harder to compile for arm archs.

As for the now values, these are of course already integrated in the forecast models. The models often find a direct correlation with the closest data lags of the history values. When feeding history values to the models the now values are the last historical values, so they are taken into consideration when forecasting.
I’m not sure if I’m missing something else on your analysis about this.

[markpurcell]

My apologies, I think I haven’t explained myself for the ‘now’ variable.

Take for a specific example pv_forecast over the last 30 minutes (1630-1700 - sunset is 1703):

pvlib has calulated pv_forecast as 729 W and p_load constant at 896W and as a consequence p_batt_forecast has been set a constant 166W for the full 30 minute period.

I am running mpc_optim every minute.

emhass has already access to the top graph variables for load and pv, but doesn’t use them when calculating the optimisation, it only looks back over the last 24 hrs, I don’t believe it uses the current value.

So p_batt should be increasing after each mpc_optimisation to reflect the actual differences (now values) for Solar Power and Load Power.

This is just a minor example, but I sometimes have errors between forecasts and actuals of 1000 W or 2000 W, which would make a substantive difference to the published p_batt_forecast value.

Now for pv_forecast I could inject the ‘now’ values, with method_ts_round first using something like:

"pv_now": {{(states('sensor.powerwall_solar_now')|float(0))}}
"solcast_future_forecasts": {{state_attr('sensor.solcast_forecast_data', 'forecasts')|map(attribute='pv_estimate')| list}}
"pv_power_forecasts:"{{([states('sensor.powerwall_solar_now')|float(0)] + 
                       (state_attr('sensor.solcast_forecast_data', 'forecasts')|map(attribute='pv_estimate')| list))[:48]}}

pv_now would be updated and injected for each mpc_optim run and the remainder of the forecasts would be fairly stable.

"pv_now": 0.05

"solcast_future_forecasts": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.383, 2.8362, 4.7992, 6.8943, 8.5046, 10.0137, 11.0883, 12.0041, 12.6009, 12.8405, 13.0011, 12.7743, 12.5328, 11.8008, 10.8927, 9.6497, 7.8646, 6.2096, 4.2492, 2.1112, 0.3171]

"pv_power_forecasts:"[0.05, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0.383, 2.8362, 4.7992, 6.8943, 8.5046, 10.0137, 11.0883, 12.0041, 12.6009, 12.8405, 13.0011, 12.7743, 12.5328, 11.8008, 10.8927, 9.6497, 7.8646, 6.2096, 4.2492, 2.1112]

[davidusb]

Oh yes I do get it now. Ok this is specific to MPC. You are right, when using MPC it is just using the forecast from the day ahead optimization. Those forecast should be updated every time. I’ll fix this.

[davidusb]

Ok, taking a deeper look into this, the dayahead and the MPC optimization implementations are just fine. The scraping method for PV forecast and the naive method for load consumption forecast are just what they are with their respective limitations. To make the current/now values act as input to the forecast we will need what I’m doing right now for the load forecast: train a machine learning regression model. What I realize now is that we need to do this also for the PV forecast. We have a PV forecast obtained from scraping a website with data from GFS model. We then use PVLib to convert from irradiance to watts. Now we can use this along with the history of real PV production values to train a regression model that will be more accurate because it will be fitted to the latest lags of the real data. Ok I’ll put all this on the roadmap and continue working on these better machine learning based forecast models. So there is nothing to fix, just need to provide better forecasts. Of course the user is always open to provide their own forecasts via the runtime passed data when using MPC.

[prinzkafka (Mario Damian)]

Hi, I have a problem with my template sensor for the sensor_power_load_no_var_loads in the EMHASS Addon

This is the Log:

[2022-06-05 15:11:44,615] ERROR in app: Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2077, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1525, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1523, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1509, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 130, in action_call
    input_data_dict = set_input_data_dict(config_path, str(config_path.parent), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 81, in set_input_data_dict
    P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
  File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 454, in get_load_forecast
    rh.get_data(days_list, var_list)
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 107, in get_data
    data = response.json()[0]
  File "/usr/local/lib/python3.9/dist-packages/requests/models.py", line 917, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: [Errno Extra data] 500 Internal Server Error
Server got itself in trouble: 4

This is how I calculate the Template Sensor:

emhass_power_no_var_load:
            value_template: >
                {% set powerload = states('sensor.electricity_rietschener_str_24_total_power') | float %}
                {% set heizpatrone = states('sensor.heizpatrone_power') | float %}
                {% set geschirrspueler = states('sensor.geschirrspuler_current_consumption') | float %}
                {% set waschmaschine = states('sensor.waschmaschine_power_consumption') | float %}
                {% set value = ( powerload - heizpatrone - geschirrspueler - waschmaschine ) | round(1) %}
                {{ '{:.1f}'.format(value) }}
            unit_of_measurement: 'W'

When I use the sensor of my electricity meter, everything works smoothly, so I assume that I made a mistake somewhere with the template sensor. Maybe someone has an idea where the error could be?

[prinzkafka (Mario Damian)]

I think I figured out what the problem might be. The template sensor sometimes has an unknown state. Could this be the problem? How can I work around this?

[davidusb]

This a breaking change in Home Assistant templates with the new 2022.6 release. You’ll need to catch those unknown values with some if else statements in your template.

[davidusb]

You can also specify default values inside the failing function on the template. Take a look here: Fail template functions when no default specified by emontnemery · Pull Request #71687 · home-assistant/core · GitHub

[prinzkafka (Mario Damian)]

Thanks for the tips, have tried both - also both together - after a restart the first state will be an unknown state in the database - here is the code of the template sensor. Any other ideas?

{% set powerload = states('sensor.electricity_rietschener_str_24_total_power') | float(default=0) %}
{% set heizpatrone = states('sensor.heizpatrone_power') | float(default=0) %}
{% set geschirrspueler = states('sensor.geschirrspuler_current_consumption') | float(default=0) %}
{% set waschmaschine = states('sensor.waschmaschine_power_consumption') | float(default=0) %}
{% set value = ( powerload - heizpatrone - geschirrspueler - waschmaschine ) | round(1) %}    
{% if (states('sensor.electricity_rietschener_str_24_total_power') != "unavailable") %}
   {{ '{:.1f}'.format(value) }}
{% else %}
   0
{% endif %}

[davidusb]

@prinzkafka

round(1)

Try round(1,default=0). I had to do this with my templates…
You will have to do this in templates where you use these functions: round , multiply , log , sin , cos , tan , asin , acos , atan , atan2 , sqrt , timestamp_custom , timestamp_local , timestamp_utc , as_timestamp , strptime , float , int .