Ok, thanks for that. It’a not clear in the manual. It’s a difficult read in spots but great work. very appreciative to the developer.
Hey @davidusb when I load the pkl file in table1 I see three lots of data, each 48 entries long. There is nothing missing as far as I can tell. I have tried to enable DEBUG level logging via the ad-on config UI but that does not seem to take effect. I am a bit stuck now.
Must this all be integers? Cant it be a float?
I don’t know if I have a time zone issue or if I’m just not using dayahead-optim right. I can’t get the PV forecast sent to the function to align with actual times.
Example:
At 08:35 I call dayahead-optim with the following PV forecast:
pv_power_forecast: [1.726, 3.264, 4.613, 5.789, 6.391, 6.79, 6.71, 6.233, 5.775, 5.273, 4.226, 2.71, 1.568, 0.356, 0.0, 0.0, 0.0, 0.0, 0.0, 0.01, 0.188, 0.454, 0.683, 0.627]
My intention here is that the forecast is 1.726 kWh between 08-09, 3.264 kWh between 9-10 and so forth. But, when I look at the optimization results table, the forecast is one hour off. For some reason, the optimization believes that the first forecast in the list is for the time 07-08.
What am I missing?
Server time and timezone where both HA and EMHASS is running is correct.
~$ date -I'seconds'
2023-06-27T08:48:25+02:00
EDIT: I realize I have the same problem with price forecast passed to dayahead-optim. The question then probably is: What is the starting time of dayahead-optim? Do I need to pass a few hours of historical data to it?
Thanks. It’s not intuitive what ‘first’ or ‘last’ means in terms of timestamp rounding. Maybe something got lost in translation. But I did some tests from your pointer.
Forecast data i send is per hour. Time now is 18:33, so the first value in forecast parameters sent to dayahead-optim is for the time period 18:00-19:00.
Solcast sensor attributes look like this:
- period_start: '2023-06-28T18:00:00+02:00'
pv_estimate: 3.4184
- period_start: '2023-06-28T19:00:00+02:00'
pv_estimate: 1.3334
...
The PV power data sent to dayahead-optim looks like this: pv_power_forecast: [3418, 1333, 416, 113...
With method_ts_round: 'first', dayahead-optim gives me this result:
With method_ts_round: 'last', dayahead-optim gives me this result:
While method_ts_round: 'nearest', dayahead-optim gives me this result, which probably is consistent since the time is nearer to 19 than 18 now.
The results are consistent with the setting I believe, but why does it start two hours earlier? Time now is between 18:30 and 19, so last/nearest should give an optimization from 19, not from 17. Rounding to ‘first’ should give an optimization starting from 18, not from 16.
Sure, I can adapt my forecast data, but I believe the behavior is wrong. Maybe I should open an issue for it.
@davidusb can you please have a look at issue #90?
https://github.com/davidusb-geek/emhass/issues/90
You made a comment and closed it. I have replied, but I can’t reopen the issue so I’m not sure you got my reply. Please just have a look at my reply and decide whether it’s reason enough to reopen the issue or if you want to leave it closed. I will accept either, but please just review my comment.
Yes I’ve just reopened the issue. It will be treated. I’ve just had two very busy weeks but I will soon get some time to work on this and other issues.
The two hours difference is odd.
When I inject values they appear in the current timeslot.
Can you connect to your HomeAssistant container and check the timezone setting there?
Trying to catch up with all the info in this thread. Lots of good stuff to boil my head with. I’ve tried to implement everything up until actually executing automations to defer loads or charge/discharge etc. Want to make sure it’s working correctly before I go that far.
I’ve a 5 kWp system, half facing NNW and the other half facing ENE and all on one Fronius Primo 6.0-1 inverter.
So, I’ve set this up in solcast taking their advice and configuring an azimuth of 10° (halfway between the two directions my panels face) and tilt of 21°.
I’m also with Amber Electric and on the NSW AusGrid bonus trial.
I’ve installed the EMHASS add-on as a Home Assistant OS user. HA is running on a VM under Proxmox on an Intel NUC along with other things like Tuya convert and a docker instance running TeslaMate.
Configuration for EMHASS:
hass_url: empty
long_lived_token: empty
costfun: profit
logging_level: INFO
optimization_time_step: 30
historic_days_to_retrieve: 2
method_ts_round: first
set_total_pv_sell: false
lp_solver: COIN_CMD
lp_solver_path: /usr/bin/cbc
set_nocharge_from_grid: false
set_nodischarge_to_grid: false
set_battery_dynamic: false
battery_dynamic_max: 0.9
battery_dynamic_min: -0.9
load_forecast_method: naive
sensor_power_photovoltaics: sensor.sonnenbatterie_84324_production_w
sensor_power_load_no_var_loads: sensor.house_power_consumption_less_deferrables
number_of_deferrable_loads: 2
list_nominal_power_of_deferrable_loads:
- nominal_power_of_deferrable_loads: 1500
- nominal_power_of_deferrable_loads: 750
list_operating_hours_of_each_deferrable_load:
- operating_hours_of_each_deferrable_load: 2
- operating_hours_of_each_deferrable_load: 2
list_peak_hours_periods_start_hours:
- peak_hours_periods_start_hours: "02:54"
- peak_hours_periods_start_hours: "17:24"
list_peak_hours_periods_end_hours:
- peak_hours_periods_end_hours: "15:24"
- peak_hours_periods_end_hours: "20:24"
list_treat_deferrable_load_as_semi_cont:
- treat_deferrable_load_as_semi_cont: true
- treat_deferrable_load_as_semi_cont: true
load_peak_hours_cost: 0.1907
load_offpeak_hours_cost: 0.1419
photovoltaic_production_sell_price: 0.065
maximum_power_from_grid: 14490
list_pv_module_model:
- pv_module_model: CSUN_Eurasia_Energy_Systems_Industry_and_Trade_CSUN295_60M
list_pv_inverter_model:
- pv_inverter_model: Fronius_International_GmbH__Fronius_Primo_5_0_1_208_240__240V_
list_surface_tilt:
- surface_tilt: 21
list_surface_azimuth:
- surface_azimuth: 350
list_modules_per_string:
- modules_per_string: 17
list_strings_per_inverter:
- strings_per_inverter: 1
set_use_battery: true
battery_discharge_power_max: 3300
battery_charge_power_max: 3300
battery_discharge_efficiency: 0.95
battery_charge_efficiency: 0.95
battery_nominal_energy_capacity: 10000
battery_minimum_state_of_charge: 0.1
battery_maximum_state_of_charge: 0.9
battery_target_state_of_charge: 0.1
Deferrable loads:
-
I did have an Arlec GridConnect power point for the pool pump with Tasmota firmware but as that doesn’t monitor power usage, I’ve replaced it with a Zigbee power point that does monitor power. The power measurement is being subtracted from the total consumption see below.
-
I’ve installed a Tuya power monitoring plug behind the dish washer and added the local tuya HACS to integrate it (I hate Tuya but bit my tongue). This is also being subtracted from total consumption, see below.
-
I’ll also installed a zigbee power point for the dryer and washing machine (when I get around to it).
-
I’ve also install the Tesla HACS for my Model Y Performance. I only have a dumb Tesla charger that came with my old Model 3 so have to depend on load data and control via the car itself. Not using it yet.
Only the first two deferrable loads are being subtracted from total home consumption at this point:
- platform: template
sensors:
dw_power:
unique_id: dw_power
friendly_name: "Dish Washer Power"
value_template: "{{ states.switch.dw_switch.attributes.current_consumption | float(0) }}"
unit_of_measurement: W
- platform: template
sensors:
house_power_consumption_less_deferrables:
unit_of_measurement: W
unique_id: house_power_consumption_less_deferrables
value_template: >-
{% set consumption = states.sensor.sonnenbatterie_84324_meter_consumption_4_2_w_total.state|float(0) %}
{% set deferrable1 = states.sensor.garage_power_point_power.state|float(0) %}
{% set deferrable2 = states.sensor.dw_power.state|float(0) %}
{{ (consumption - (deferrable1 + deferrable2))|float(0) }}
So I’m getting this consumable sensor from the sonnen battery (sensor.sonnenbatterie_84324_meter_consumption_4_2_w_total).
Am I right in assuming that it should go negative if I’m selling energy into the grid?
The battery I have is a sonnen eco 9.43 10 kWh (9 kWh usable, although I think it’s getting closer to 8 now that it’s nearly 5 years old, you can upgrade them to 15 I think but not sure it’s worth it).
I have control of battery mode (Auto Self Consumptionm, manual and ToU) and charging and discharging via API thanks to @julianlu.
So to integrate the solcast forecast data:
sensor:
# Solar forecast for EMHASS
- platform: rest
name: "Solcast Forecast Data"
json_attributes:
- forecasts
resource: https://api.solcast.com.au/rooftop_sites/SOLCAST_RESOURCE_ID/forecasts?format=json&api_key=SOLCAST_API_KEY&hours=24
method: GET
value_template: "{{ (value_json.forecasts[0].pv_estimate)|round(2) }}"
unit_of_measurement: "kW"
device_class: power
scan_interval: 00:30
force_update: true
- platform: template
sensors:
solcast_24hrs_forecast:
value_template: >-
{%- set power = state_attr('sensor.solcast_forecast_data', 'forecasts') | map(attribute='pv_estimate') | list %}
{%- set values_all = namespace(all=[]) %}
{% for i in range(power | length) %}
{%- set v = (power[i] | float |multiply(1000) ) | int(0) %}
{%- set values_all.all = values_all.all + [ v ] %}
{%- endfor %} {{ (values_all.all)[:48] }}
I see some discussion about suitability of ‘Solcast Forecast’ Data being kW not W but it appears we only use solcast_24hrs_forecast which corrects this?
Next are the shell commands:
shell_command:
dayahead_optim: 'curl -i -H "Content-Type: application/json" -X POST -d ''{}'' http://localhost:5000/action/dayahead-optim'
publish_data: 'curl -i -H "Content-Type: application/json" -X POST -d ''{}'' http://localhost:5000/action/publish-data'
post_amber_forecast:
'curl -i -H ''Content-Type: application/json'' -X POST -d ''{"prod_price_forecast":{{(
state_attr(''sensor.amber_feed_in_forecast'', ''forecasts'')|map(attribute=''per_kwh'')|list)
}},"load_cost_forecast":{{(
state_attr(''sensor.amber_general_forecast'', ''forecasts'') |map(attribute=''per_kwh'')|list)
}},"prediction_horizon":33}'' http://localhost:5000/action/dayahead-optim'
post_emhass_forecast:
'curl -i -H ''Content-Type: application/json'' -X POST -d ''{"prod_price_forecast":{{(
state_attr(''sensor.amber_feed_in_forecast'', ''forecasts'')|map(attribute=''per_kwh'')|list)
}},{{states(''sensor.solcast_24hrs_forecast'')}},"load_cost_forecast":{{(
state_attr(''sensor.amber_general_forecast'', ''forecasts'') |map(attribute=''per_kwh'')|list)
}}}'' http://localhost:5000/action/dayahead-optim'
post_mpc_optim_solcast:
'curl -i -H "Content-Type: application/json" -X POST -d ''{"load_cost_forecast":{{(
([states(''sensor.amber_general_price'')|float(0)] +
state_attr(''sensor.amber_general_forecast'', ''forecasts'') |map(attribute=''per_kwh'')|list)[:48])
}}, "prod_price_forecast":{{(
([states(''sensor.amber_feed_in_price'')|float(0)] +
state_attr(''sensor.amber_feed_in_forecast'', ''forecasts'')|map(attribute=''per_kwh'')|list)[:48])
}}, "pv_power_forecast":{{states(''sensor.solcast_24hrs_forecast'')
}}, "prediction_horizon":48,"soc_init":{{(states(''sensor.sonnenbatterie_84324_state_charge_user'')|float(0))/100
}},"soc_final":0.05,"def_total_hours":[2,0,0,0]}'' http://localhost:5000/action/naive-mpc-optim'
This is where I get a bit confused with all the different methods.
I understand the basic ‘dayahead_optim’ is a good place to start and that’s what I’m running now. But can I run the other post commands since I have the forecast data to use?
Do I run them as well as the dayahead_optim or instead of?
Do I run all three post commands?
I have the following node-red flows to execute these various curl commands (although I’ve left them as shell commands for the time being and called them from node-red).
So at the moment I’m only running
- dayahead_optim at 05:30 and
- publish_data every 5 mins
The battery controls are simply a set of posts linked back to 4 buttons
I don’t actually use the 5th TOU mode, but I’ve put it in anyway.
These lead to the flows below:
Apart from the buttons that I can control manually, there are three flows that also control the battery state.
- The first is an afternoon charge that activates if the Amber price is “very low” and the battery is less than 80% charges. Its a rainy day charge to get me over the evening hump at a better price.
- The second is an early morning charge starting at 03:00 early morning flow (at the top) which charges the battery if it’s less that 20% charged and then lets it sit idle for 2 hours until people start to wake up and consume energy (first coffee and heaters on in winter).
- The last one down the bottom is to catch any spikes in the FiT what seem to happen from time to time where the tariff can hit $12 per kWh. Only problem is I’m often burning everything out of the 3.3kW output of the battery in winter time so can’t take advantage sometimes unless I run around turning aircons and heaters off. 5 adults in the house at the moment so can be difficult.
So, these are my rules based configurations that I will replace with flows that use the output of EMHASS when its working correctly.
What I’m getting so far is:
Seems to have died in the last day. Probably me mucking around too much.
Any advice greatly appreciated.
No this is not ok. This sensor should represent only your power consumption, not your net power balance. So it should be all positive values. Take your total load consumption and subtract the deferrables power. I guess that your problem here is that you are not using your total load but your net power.
You either use the day-ahead optimization once a day like what you already have or either you can use the MPC command to update your optimization results at a higher frequency (every 5min, 10min, or whatever you want). This last approach is interesting if you have high frequency forecasts updates available.
There is a situation where it could be interesting for me to use both optimization at the same time. We can launch the day-ahead optimization early in the morning to plan the battery SOC for the day, then use the MPC recurrently to follow that optimal SOC trajectory.
David
Thankyou.
For Sonnen owners I think the correct sensor would be one of the following:
- sensor.sonnenbatterie_XXXXX_consumption_w or
- sensor.sonnenbatterie_XXXXX_consumption_avg
Probably the first one.
I’ll experiment with the day-ahead and the MPC command.
Hi,
I have been diving deeper into the code to look into adding the heatpump model we discussed before.
- I am in general not happy with how the project is setup. The files are very large with single classes that implement way too much functionality. For example the forecast class can get weather data, calculate power with PVLib and can get power forecasts from solcast and forecast.solar. IMHO these should all be separate classes with appropriate abstractions.
- I’m not sure how you could ever effectively unit test the code base in its current state and it is also very difficult to read, understand and manevour.
- I think the PV forecasting part should be split from the main project. To me it’s just an input to energy management and can be used for many other things as well (energy trading and visualizations for example).
- In general I think EMHASS is too technical and too difficult to setup. To make it much more user friendly I suggest making an integration which will handle most of the configuration automatically, with minimal user input (perhaps a few blueprints for appliance control). Everything else should just work out of the box without any YAML config required.
I personally think the topic of energy management will explode in the near future. In NL it is already getting a lot attention this year due to insane electricity prices recently. Tomorrow we have 16 hours of negative eletricity prices, of which 3 hours hit the absolute record low of -€500 / MWh. We expect this to get exponentially worse in the coming decade and thus platforms like EMHASS will become more and more important.
I’m curious to hear what others think.
To be brutally honest - emhass has been on my wishlist for over a year and will probably stay there for at least another- it looks really complex and I am afraid of the time I will need to spend.
That being said l, it is very impressive and I don’t think your comments do just to what @davidusb has created - yes, with the energy transition and more and more equipment in the house competing for resources optimization software is a real must have and a different user experience will be required to increase adoption of Emhass but let’s not forget where HA itself started . This is a start and a very promising one. I found another one on the web which also looks promising:
Perhaps that one is a better fit?
That being said l, it is very impressive and I don’t think your comments do just to what @davidusb has created
Yes agreed, I did not mean to underplay the project itself, not many people are technically capable of making something as technically complicated as this.
I have looked into flexmeasures as part of my uni program (MSc in EE) but it is mostly centered around a server-API model while I would like to implement it in a distributed manner such as EMHASS.
I think the manual is difficult to interpret. Some assumptions are made about the technical skill level of the implementer. There are some mathematical explanations that could be in an appendix. But this is an indication of the maturity of the system. It’s only at version 0.3.14 and every support possible should be made available to davidusb to continue developing it.
Turns out it’s not difficult to implement at all. The difficulty in controlling my sonnen battery from HA is more complex. A lot of work has already been done by Mark Purcell to interface the amber electric forecast data into the system. I can imagine this has to be done for many electricity retailers around the world.
So, the complexities are more around these two interfaces. The inputs from many different forecast systems and the outputs to many different home energy systems. Have to focus on the most common (80/20 rule).
The guts of the system could be treated as a black box (so as to not scare people away) with focus on expected inputs and outputs and then focus on the different combinations of systems can be outsourced to different interested parties that have the skill and the actual systems to develop the documentation/process.
Agreed. The most difficult part would be to share a common control interface across different appliances and devices in your house, kind of like zigbee but for power control. I have given this a lot of thought and after looking at similar projects I think the only way to achieve it is by following a standard like EEBUS. I also like what SolarEdge is doing in this space but that’s proprietary and it’s just one supplier. I think if there was a common standard they would follow it…
My ideal energy management system is a black box that I can put in your house, discovers compatible appliances and then simply starts doing its job. Kind of like how HA onboarding is now, very smooth. Only then can I sell it with a good concience to my grandmother who knows how to use a phone and a laptop but would never in a million years setup a bunch of YAML code.
Thank you! Problem solved.
Both the server and homeassistant container was running properly CEST timezone, but I never specified a timezone for my emhass container. So it was defaulting to UTC.
Setting the TZ environment variable properly (to Europe/Stockholm) when running the container solved the issue. Now the optimization starts from when I expect it to.
1 Like
Hi,
There is always room for optimization in these codes, this is exactly what I said on my first post in this thread almost 2 years ago!
Rearranging classes and methods in a code is just that, optimization for better understanding and maintenance of the code.
Unlike you, I’m very happy to what it is today, mostly because I’m not a full stack programmer. I just hold a PhD in Electrical Engineering, so the electrical/energy part is really mi field of expertise.
However, even if I’m not a programmer I was able to use all these DevOps features in my code:
- Object oriented programming.
- Fully automated unit testing. Despite this assertions of yours (“I’m not sure how you could ever effectively unit test the code base in its current state”) the unit tests are there and they are really helpfull for automated testing.
- Using coverage to track the code that is being covered by the unit tests >> 88% as of today.
- Using fully automated continuous integration thank to the github workflows.
- Automated deployment of both the standalone Docker and the Add-on images.
- Automated documentation using doctrings directly from the code with Sphinx and Read the Docs.
- Code versioning with semantic versioning.
- An updated CHANGELOG.
- Etc etc etc
So I’m pretty happy with all this and I have learned a lot in the process. But yes there will be always room for optimization mostly to make it more easy for people to set it up and use it.
Frankly I’m also just happy with the fact that the core functionality of the system works as expected and is able to optimize our home energy usage. At the end of the day I’m just happy to have shared this project for others to use.
5 Likes
I am also going for a PhD in electrical engineering (finishing MSc this year) so I know what you mean.
I don’t think you understand my comments, I haven’t said anything about the functionality of the project, this is just about the organization of the code. This is about making it ready for the future. But based on your reaction, I am not sure you are thinking about that and that’s fine, but then I will now move on to a different project and I wish you good luck.