Thanks you for the reply. I going to follow your advice.
I have filtered the entiteties following this guide: How to keep your recorder database size under control, which helped my a lot to get the database smaller.
With Home Assistant 2023.4.0 we get a new database format which has the benefit to have* Smaller (deduplication), less disk usage. Maybe this will help me.
Here is my updated template for the âsensor.power_load_no_var_loadsâ without any Home Assistant long-term statistics transformation.
- name: "Power load no var loads"
unique_id: fbfeef21-1aa3-4a54-b781-426f46fef597
unit_of_measurement: W
device_class: power
state: >
{% set powerload = states('sensor.power_adresse') | float(default=0) %}
{% set vvb = states('sensor.bryter_varmvannsbereder_electric_consumption_w') | float(default=0) %}
{% set varmekabler = states('sensor.power_varmekabler') | float(default=0) %}
{% set varmepumpe = states('sensor.strombryter_varmepumpe_electric_consumption_w') | float(default=0) %}
{% set value = ( powerload - vvb - varmekabler - varmepumpe) | round(1,default=0) %}
{{ value }}
Hi, I have run into some issues with the current setup of EMHASS. Weirdly enough the automated forecast works and shows valid results but after 24 hours when it should do the same thing again in the morning there is a failure to compute. Here are my mistake protocols.
2023-04-05 09:44:26,211 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 597, in get_load_forecast
forecast_out.index = self.forecast_dates
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5596, in __setattr__
return object.__setattr__(self, name, value)
File "pandas/_libs/properties.pyx", line 70, in pandas._libs.properties.AxisProperty.__set__
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 769, in _set_axis
self._mgr.set_axis(axis, labels)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 214, in set_axis
self._validate_set_axis(axis, new_labels)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/base.py", line 69, in _validate_set_axis
raise ValueError(
ValueError: Length mismatch: Expected axis has 8 elements, new values have 24 elements
Within a couple minutes I switched the length of the time slots to 30 minutes and this mistake result showed up.
2023-04-05 10:03:53,977 - web_server - ERROR - Exception on /action/dayahead-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2528, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1823, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1799, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 170, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 91, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'])
File "/usr/local/lib/python3.9/dist-packages/emhass/forecast.py", line 597, in get_load_forecast
forecast_out.index = self.forecast_dates
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5596, in __setattr__
return object.__setattr__(self, name, value)
File "pandas/_libs/properties.pyx", line 70, in pandas._libs.properties.AxisProperty.__set__
File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 769, in _set_axis
self._mgr.set_axis(axis, labels)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 214, in set_axis
self._validate_set_axis(axis, new_labels)
File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/base.py", line 69, in _validate_set_axis
raise ValueError(
ValueError: Length mismatch: Expected axis has 5 elements, new values have 48 elements
I changed the sensors in the setup but no change. If I restart HA everything works fine, I can optimize, publish and automate. Just the next day there will be this error. The error also persists when I only restart EMHASS. Here is also my config file, if it is of any use.
hass_url: empty
long_lived_token: empty
costfun: self-consumption
optimization_time_step: 30
historic_days_to_retrieve: 2
method_ts_round: nearest
set_total_pv_sell: false
lp_solver: COIN_CMD
lp_solver_path: /usr/bin/cbc
set_nocharge_from_grid: false
load_forecast_method: naive
sensor_power_photovoltaics: input_number.nullsensor
sensor_power_load_no_var_loads: input_number.nullsensor_2
number_of_deferrable_loads: 3
list_nominal_power_of_deferrable_loads:
- nominal_power_of_deferrable_loads: 450
- nominal_power_of_deferrable_loads: 60
- nominal_power_of_deferrable_loads: 50
list_operating_hours_of_each_deferrable_load:
- operating_hours_of_each_deferrable_load: 3
- operating_hours_of_each_deferrable_load: 8
- operating_hours_of_each_deferrable_load: 4
list_peak_hours_periods_start_hours:
- peak_hours_periods_start_hours: "02:54"
- peak_hours_periods_start_hours: "17:24"
list_peak_hours_periods_end_hours:
- peak_hours_periods_end_hours: "15:24"
- peak_hours_periods_end_hours: "20:24"
list_treat_deferrable_load_as_semi_cont:
- treat_deferrable_load_as_semi_cont: true
- treat_deferrable_load_as_semi_cont: true
- treat_deferrable_load_as_semi_cont: true
load_peak_hours_cost: 4
load_offpeak_hours_cost: 4
photovoltaic_production_sell_price: 0
maximum_power_from_grid: 9000
list_pv_module_model:
- pv_module_model: Avancis_PowerMax_STRONG_130
- pv_module_model: Antaris_Solar_AS_P_230
list_pv_inverter_model:
- pv_inverter_model: SolarEdge_Technologies_Ltd___SE3000__240V_
- pv_inverter_model: ABB__MICRO_0_3_I_OUTD_US_240__240V_
list_surface_tilt:
- surface_tilt: 90
- surface_tilt: 40
list_surface_azimuth:
- surface_azimuth: 180
- surface_azimuth: 135
list_modules_per_string:
- modules_per_string: 7
- modules_per_string: 1
list_strings_per_inverter:
- strings_per_inverter: 1
- strings_per_inverter: 1
set_use_battery: false
battery_discharge_power_max: 1000
battery_charge_power_max: 1000
battery_discharge_efficiency: 0.95
battery_charge_efficiency: 0.95
battery_nominal_energy_capacity: 5000
battery_minimum_state_of_charge: 0.3
battery_maximum_state_of_charge: 0.9
battery_target_state_of_charge: 0.6
And what is the shell command that you are using to launch the dayahead-optim?
this is it
shell_command:
dayahead_optim: "curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://192.168.1.105:5000/action/dayahead-optim"
publish_data: "curl -i -H \"Content-Type:application/json\" -X POST -d '{}' http://192.168.1.105:5000/action/publish-data"
Ok thanks.
Well the problem seems to be with sensor sensor_power_load_no_var_loads:input_number.nullsensor_2.
What you need to do is double check how are you building that sensor. In that part of the code the retrieved data for your case does not have the correct expected length. Any possibilities that your are resetting the sensor at midnight?
First of all, I would like to thank you for the addon! In spite of trying to set up Emhass for about half of a year, I am still struggling to understand a few things that prevents me from using it to its full potential. Is there a way to create a deferrable load that doesnât have an operating_hours_of_each_deferrable_load but instead, if there is excess PV power production, it will always run. During the day, my water heater keeps a lower temperature, and before my family showers, it preheats water to a higher temperature. When there is excess energy, that my house canât use with its own consumption, I wanted to use EMHASS to switch the deferrable load and turn on the water heater. If I specify the operating_hours_of_each_deferrable_load it does turn on while the PV production, but if I set the time for 1 hour, it just turns on for 1 hour. But I am unable to estimate how many hours of sun there will be, and preset the time in the morning. Is there a way to make deferrable_loads that will turn on for no predefined time, to use excess PV energy? If there is a sunny day, I would like to be able to preheat a pool, charge non PV batteries and other stuff that will not turn on when there is a super cloudy day. It is unclear to me whether I missed this function or if Emhass works mostly within predefined working hours and mostly based on peak/off-peak electricity. For me, the peak electricity cost is the same as the off-peak, so I am not using this function. Thank you for any suggestion!
I do something similar.
For my hot water system, I just scheduled it to run for 6 hours and the optimisation almost always selects the solar production time to run.
You could get a little more sophisticated and set the hours to run based on the hours of sunlight expected. Which is what I do for my pool filter pump so it runs for more hours in the summer (when the days are longer) than in the winter. You can see the optimisation almost matches the water heater.
I only run my pool heater for the number of hours that I have very cheap electricity (< 5¢/ kWh), but you could also setup the run hours based on the forecast solar production. I e. 0 hours is solar forecast is below a threshold and then maybe an additional hour for each additional increment in production. My pool heater is 5 kW, so I would run for an additional hour for each 5 kWh of solar production over say a 30 kWh threshold.
I would recommend you start with just one system, hot water and get that working correctly before you try and get multiple systems working.
Thanks for the reply, the sensor is just a helper putting out a constant zero value. I changed it to a random other sensor since I donât use the perfect optimization, but it still does not work properly. But anyway, Iâm about to submit the thesis for that project so it wonât matter much anymore.
But thanks anyway
Thanks for all your detailed responses.
Regarding the list of things to do, I could help with adding support for heatpump modelling?
Specifically I would like to use this paper: Modeling a Domestic All-Electric Air-Water Heat-Pump System for Discrete-Time Simulations
I prefer this paper because itâs very recent, I know the authors personally (and I can ask them questions if needed 
) and I have access to the full python simulation code, which I previously used in one of my âresearchâ projects. Unfortunately IEEE asks money to download the paper but I can send it to you for free.
No problem. Like I said we can discuss it on github and probably a proof of concept on simple Python scripts is a good idea before adding to the final code.
Iâll have a look to that paper. Donât worry Iâve access to IEEE. At a first glance it doesnât treat the LP problem. But 'll have a better look.
The paper from Langer et al is not old (2020) and it provides the complete linear programming model formulation for heat pumps.
I have two air to water heat pumps for domestic hot water (600 W load) and pool heating (three phase 5 kW load) that I am optimising with EMHASS.
I find both of them work fairly well with EMHASS.
Domestic hot water
My 300l system needs to run for about 2 hours per day consuming about 2 kWh to maintain our device. By default it heats to 61°C, but I also have an SG Ready relay (controlled by Shelly) that superheats to 65°C.
Ideally this would run twice, once at the start of the solar day and once the end of the solar day to get me through the night, I have this programmed in EMHASS to run over a non continuous 6 hour window:
Three phase pool heater
I really just run this on an pure low cost basis. I set the def_hours based on the number of hours that the price forecast is extremely_low (< 5¢/ kWh), EMHASS then schedules this along with my other loads and it works quite well.
I donât think there is much to be improved for air to water heatpumps optimisation, maybe hydronic heating could be a use case that would benefit from modelling.
HVAC
My biggest challenge with heat pump optimisation is my HVAC system. Currently we are still having warm days so there is reasonable alignment between the hottest part of the day and the highest solar production.
I schedule def_hours based on the number of hours the weather forecast is over a user defined set point, I have this set as a continuous load that can take values between 0-4 kW, whilst I cannot control power directly my automation adjusts the set point up and down to increase and decrease power consumption:
This works reasonably well as you can see from the scheduling although there could be some improvements.
You can see the scheduling of the HVAC is start / stop switching the system from 0 to 4000W in adjacent time slots, I would much prefer an average approach of maybe 2.5 kW over three adjacent time slots.
I would like the ability to âsupercoolâ when cheap energy is bountiful, covering periods just before a price spike. Iâm my graphic above supercooling from 3-4pm would be ideal utilisation of cheap energy.
Of course the actual power consumption of my HVAC isnât always the same as the deferrable loads forecast, as it is a function of external temp and setpoint.
Finally in the late afternoon window, after 4pm, I generally have a ongoing need for cooling, but if course EMHASS doesnât schedule over that expensive period. I find I generally just manually switch on my HVAC during these periods.
Winter scheduling for heating doesnât align with Solar production, so I generally turn off EMHASS HVAC scheduling over winter and just have that appear in my household load forecast for subsequent days.
Very cool. I think the general advantage of modelling the heatpump would be the ability to look ahead, temperature wise. Together with temperature forecasts this could help you plan the cycles more efficiently. But ofcourse, we need some practical evidence first before we can make any conclusions.
Possibly something like a user defined power forecast for individual loads that could be injected into the optimisation model could be useful.
I know what the forecast external temperature profile is for tomorrow.
A simple power model would be 1kW per degree over say 24°C to a maximum of 4kW and maybe an additional 1kW at 3pm, as my buy price increases at 4pm. So with a one hour time slot the desired power profile for tomorrow would be:
Temp: [22,24,25,26,27,26,26,26,25,24,22,21,20,20]
Power:[ 0, 0, 1, 2, 3, 2, 2, 3, 2, 1, 0, 0, 0, 0]
EMHASS could then use this as another fixed constraint to optimise against.
Providing this and using it as an additional constraint can be an option. But this means that you know that model of Temp vs Power consumption.
Another option is to model that based on the Temp and some parameters that are characteristic of the thermal behavior of your house, and to do all this inside EMHASS.
The other goal that I have in mind is that this should be useful to simplify the control of the heat pump. So you will need to provide the Temp forecast and EMHASS will be able to give the optimal temperature setting for your heat pump. Mark, do you think that this approach can be useful? I currently donât have a heat pump so it is not very clear to me what will be the easiest variables to interface with.
The easiest values to get from a typical home would be:
. External temperature (either sensor or cloud based)
. Internal temperature (sensor)
. Desired set point (input number)
. HVAC actual power consumption (sensor or estimate)
Trying to match power consumption to a set point is difficult.
Some HVAC systems have a Demand Response Enabled Device interface which gives control over power consumption (0, 50, 75, 100) % options which would be ideal. Controlling Air conditioner using Demand Enabled Response Device per AS/NZS 4755
Whilst that gives more control the simpler way is to change the temperature set point, via a simple lookup table.
Desired internal temp 24 °C, increase power in 25% intervals for 25, 26, 27, 28 °C external temp.
The fixed power profile can be utilised in other use cases so could be worthwhile and I would imagine relatively simple to include.
A LP modelling of HVAC performance would be a welcome, but advanced, feature for EMHASS.
We could use data about house temperature characteristics as input to the model. For example, knowledge about insulation can make a huge difference in house thermal demands when itâs freezing outside. What would be even more amazing is if we could discover these model parameters from data via machinelearning. Kind of like how a smart house thermostat learns your thermal profile over time.
Hello, just wondering but have you heard about Energy Flexibility Interface? (EFI) EFI - Energy Flexibility Interface
David,
I have had some time to experiment with the ML module over the weekend.
I now have ML models for my buy/ sell prices, which is quite exciting.
Thanks for this extension.
2023-04-23 12:40:41,646 - web_server - INFO - >> Performing a machine learning forecast model fit...
2023-04-23 12:40:41,649 - web_server - INFO - Performing a forecast model fit for price_forecast
2023-04-23 12:40:41,669 - web_server - INFO - Training a KNeighborsRegressor model
2023-04-23 12:40:41,720 - web_server - INFO - Elapsed time for model fit: 0.05129742622375488
2023-04-23 12:40:42,042 - web_server - INFO - Prediction R2 score of fitted model on test data: 0.4393118941478219
2023-04-23 12:40:44,704 - web_server - INFO - Setting up needed data
2023-04-23 12:40:44,711 - web_server - INFO - Retrieve hass get data method initiated...
2023-04-23 12:40:45,562 - web_server - INFO - >> Performing a machine learning forecast model fit...
2023-04-23 12:40:45,565 - web_server - INFO - Performing a forecast model fit for price_forecast
2023-04-23 12:40:45,589 - web_server - INFO - Training a KNeighborsRegressor model
2023-04-23 12:40:45,610 - web_server - INFO - Elapsed time for model fit: 0.02150583267211914
2023-04-23 12:40:45,865 - web_server - INFO - Prediction R2 score of fitted model on test data: 0.4393118941478219
2023-04-23 12:40:45,872 - web_server - INFO - Performing simple backtesting of fitted model
2023-04-23 12:40:46,799 - web_server - INFO - Elapsed backtesting time: 0.9260625839233398
2023-04-23 12:40:46,799 - web_server - INFO - Backtest R2 score: 0.9048172716630719
Thatâs really nice!
That was the goal of the custom ML model, you can actually build a forecast of whatever sensor you have in HA, provided that it have enough data and that it is stationary.
Prod price and load cost are good examples. What it would be nice now is that you compare those results with the real forecast obtained from your provider. I donât expect that we can obtain much better results for prices or weather forecasts, because typically forecasts provider will be using very complex models. In the case of weather forecasts they are a combination of satellite and/or radar data along with deep neural networks and the such. Here we are using more simple (but efficient) ML models.
The real deal with this ML module was the need for a better load power forecast model. And it works really well for that.
One thing that I will add in the future is the possibility to pass exogenous data for training and prediction. For example train a model to forecast the load power based on the historical values of the load power itself and other additional data as the outdoor temperature, humidity, house occupancy, etc. The main issue to this is that you will have to provide future estimates of those exogenous data. For weather data those estimates are easy to obtain. But for other types of data such as occupancy you will have to build them yourself and in that case we can use the emhass custom ML model approach. That can be interesting to evaluate with the issue of error propagating along those different forecast models.
