EMHASS: An Energy Management for Home Assistant

rcruikshank · July 14, 2023, 6:42am

Yes same here. However I noticed a gap in solcast data at the same or similar time:

rcruikshank · July 14, 2023, 7:36am

@markpurcell
? Now I’m getting these errors again?

2023-07-14 17:32:00,039 - web_server - INFO - Setting up needed data
2023-07-14 17:32:00,042 - web_server - INFO - Retrieve hass get data method initiated...
2023-07-14 17:32:00,573 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 174, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 110, in set_input_data_dict
    rh.get_data(days_list, var_list,
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 140, in get_data
    df_tp = df_raw.copy()[['state']].replace(
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5920, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 419, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/blocks.py", line 580, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1292, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1237, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1098, in astype_nansafe
    result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1181, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: 'NOTRUN'
2023-07-14 17:32:00,618 - web_server - INFO - Setting up needed data
2023-07-14 17:32:00,620 - web_server - INFO -  >> Publishing data...
2023-07-14 17:32:00,620 - web_server - INFO - Publishing data to HASS instance
2023-07-14 17:32:00,637 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = 1398.0
2023-07-14 17:32:00,650 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 765.76
2023-07-14 17:32:00,661 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2023-07-14 17:32:00,673 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 3135.0
2023-07-14 17:32:00,685 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 73.5
2023-07-14 17:32:00,700 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = -3767.24
2023-07-14 17:32:00,714 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = 1.87
2023-07-14 17:32:00,724 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.45
2023-07-14 17:32:00,734 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.36

I can’t remember what the cause was last time this happened and can’t find the comments above.

Shell Commands where corrupt again.

Forecast data has dies again?

rcruikshank · July 14, 2023, 9:10am

Seems to be solcast data missing. Rebooting the system too many times ends up using up all your API calls to solcast before the day is finished. Now I have no PV forecast until the clock strikes midnight in London.

Must implement oziee ha-solcast-solar asap

anybody got any good node-red flows to execute oziee solcast_solar.update_forecasts service only 10 times a day between sunrise and sunset despite system restarts or node-red reloads ets?

Anybody user this flow with solcast 10 API limit Node Red timer that persists after HA restart?

Or BigTimer ?

Mheloy · July 15, 2023, 1:47am

I’ve got two accounts… just swap the api key because I restarted HA multiple time

Mheloy · July 15, 2023, 1:50am

The only thing I can think of is the battery is close to full and solar is in excess.

rcruikshank · July 15, 2023, 1:58am

ok good idea,

rcruikshank · July 15, 2023, 2:01am

Not working at all now? Something wrong in post_mpc_optim_solcast? Been looking at it for hours.

2023-07-15 11:57:00,040 - web_server - INFO - Setting up needed data
2023-07-15 11:57:00,043 - web_server - INFO - Retrieve hass get data method initiated...
2023-07-15 11:57:00,543 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 2190, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1486, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1484, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.9/dist-packages/flask/app.py", line 1469, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
  File "/usr/local/lib/python3.9/dist-packages/emhass/web_server.py", line 174, in action_call
    input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
  File "/usr/local/lib/python3.9/dist-packages/emhass/command_line.py", line 110, in set_input_data_dict
    rh.get_data(days_list, var_list,
  File "/usr/local/lib/python3.9/dist-packages/emhass/retrieve_hass.py", line 140, in get_data
    df_tp = df_raw.copy()[['state']].replace(
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/generic.py", line 5920, in astype
    new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 419, in astype
    return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/managers.py", line 304, in apply
    applied = getattr(b, f)(**kwargs)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/internals/blocks.py", line 580, in astype
    new_values = astype_array_safe(values, dtype, copy=copy, errors=errors)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1292, in astype_array_safe
    new_values = astype_array(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1237, in astype_array
    values = astype_nansafe(values, dtype, copy=copy)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1098, in astype_nansafe
    result = astype_nansafe(flat, dtype, copy=copy, skipna=skipna)
  File "/usr/local/lib/python3.9/dist-packages/pandas/core/dtypes/cast.py", line 1181, in astype_nansafe
    return arr.astype(dtype, copy=True)
ValueError: could not convert string to float: 'NOTRUN'
2023-07-15 11:57:00,577 - web_server - INFO - Setting up needed data
2023-07-15 11:57:00,579 - web_server - INFO -  >> Publishing data...
2023-07-15 11:57:00,579 - web_server - INFO - Publishing data to HASS instance
2023-07-15 11:57:00,594 - web_server - INFO - Successfully posted to sensor.p_pv_forecast = 0.0
2023-07-15 11:57:00,603 - web_server - INFO - Successfully posted to sensor.p_load_forecast = 324.43
2023-07-15 11:57:00,614 - web_server - INFO - Successfully posted to sensor.p_deferrable0 = 0.0
2023-07-15 11:57:00,626 - web_server - INFO - Successfully posted to sensor.p_batt_forecast = 950.0
2023-07-15 11:57:00,635 - web_server - INFO - Successfully posted to sensor.soc_batt_forecast = 5.0
2023-07-15 11:57:00,644 - web_server - INFO - Successfully posted to sensor.p_grid_forecast = -625.57
2023-07-15 11:57:00,654 - web_server - INFO - Successfully posted to sensor.total_cost_fun_value = 1.87
2023-07-15 11:57:00,667 - web_server - INFO - Successfully posted to sensor.unit_load_cost = 0.17
2023-07-15 11:57:00,680 - web_server - INFO - Successfully posted to sensor.unit_prod_price = 0.07

shell_command:
  dayahead_optim: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/dayahead-optim"
  publish_data: "curl -i -H \"Content-Type: application/json\" -X POST -d '{}' http://localhost:5000/action/publish-data"
  post_amber_forecast:
    "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"prod_price_forecast\":{{(
    state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
    }},\"load_cost_forecast\":{{(
    state_attr('sensor.cecil_st_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
    }},\"prediction_horizon\":33}' http://localhost:5000/action/dayahead-optim"
  post_emhass_forecast:
    "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"prod_price_forecast\":{{(
    state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)
    }},{{states('sensor.solcast_24hrs_forecast')}},\"load_cost_forecast\":{{(
    state_attr('sensor.cecil_st_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
    }}}' http://localhost:5000/action/dayahead-optim"
  post_mpc_optim_solcast: 
    "curl -i -H \"Content-Type: application/json\" -X POST -d '{\"load_cost_forecast\":{{(
    ([states('sensor.cecil_st_general_price')|float(0)] +
    state_attr('sensor.cecil_st_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:33])
    }}, \"prod_price_forecast\":{{(
    ([states('sensor.cecil_st_feed_in_price')|float(0)] +
    state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list)[:33])
    }}, \"pv_power_forecast\":{{states('sensor.solcast_24hrs_forecast')
    }}, \"prediction_horizon\":33,\"soc_init\":{{(states('sensor.sonnenbatterie_84324_state_charge_user')|float(0))/100
    }},\"soc_final\":0.05,\"def_total_hours\":[2]}' http://localhost:5000/action/naive-mpc-optim"

Not sure if prediction horizon should be 33 only on the second last line or 33 for all elements in post_mpc_optim_solcast? doesn’t change the error either way.

Mheloy · July 15, 2023, 4:31am

try as per Marks suggestion for the prediction horizon

       "prediction_horizon": {{
          min(48, (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
        }},

had similar issues before

markpurcell · July 15, 2023, 6:39am

Your load_cost_forecast and prod_price_forecast only have 32 elements.

Please switch to a dynamic calculation of prediction_horizon and ensure it matches your called values.

markpurcell · July 15, 2023, 6:45am

Great idea, then the results are cached on your local system.

Setup a time based automation to call five times a day at 0600, 0900, 1200, 1500 & 1800, any more often than that doesn’t add any value to the optimisation. To be honest you could just update the solcast integration once a day in the morning and it would be good enough.

markpurcell · July 15, 2023, 6:57am

At the end of 12:00-12:30 time period your battery was forecast to be at 98% SOC and you then had 90 minutes of maximum negative FIT -0.04 $/ kWh, with very little P_Load and high P_PV forecast over this period. You also don’t have any deferrable loads defined that EMHASS could turn on to consume this excess energy. Can you set up your hotwater system, or a bar radiator as a deferrable load for these types of situations?

EMHASS had very little choices available, so it chose to export for a cost of $0.13 over 12:30-13:00, so it didn’t need to export in the remaining two intervals. I suppose it could have chosen to export for a cost of $0.04 for each of the three thirty minute intervals, so the outcome would have been the same.

In my system I have zero export curtailment rules set up for my solar inverter (SolarEdge) for this type of situation which prevents my system exporting. EMHASS doesn’t know how to do curtailment, but in my case when my system does curtail my P_PV(now) value does go down, but I then move it back to my forecast so EMHASS can turn on my deferrable loads.

Mheloy · July 15, 2023, 8:29am

Thanks Mark, not really worried about this. just trying to understand how the system thinks.
I’ve handled this negative by monitory my battery SOC and CT Clamp going to the grid.
If its export morething 500W for 1 minute with negative fit… it will turn on the EV charging or if I fogot to plugin, it will turn on the RCAC in heater mode…

Mheloy · July 15, 2023, 8:32am

I’ll post this just for curiosity… again trying to understand how it works

Selling for $0.01??? or it could be the setting of the end SOC at the end of optimisation which is 5%

rcruikshank · July 15, 2023, 8:37am

Mark
Thanks for your help.
Have you seen this from amber before? Do they change the size of the their forecast array?
Do you have an example of how to calculate the size of these array and include the variables in the post_mpc_optim_solcast template?
I’m not good at this template scripting I afraid. I’ll give it a go though.
Thanks
Rob

rcruikshank · July 15, 2023, 10:31am

FIrst attempt:

{% set general_forecast_sensor = state_attr('sensor.cecil_st_general_forecast', 'forecasts') %}
{% set gfs = general_forecast_sensor | length %}
{{ gfs }}

{% set feedin_forecast_sensor = state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts') %}
{% set ffs = feedin_forecast_sensor | length %}
{{ ffs }}

Prints 48
Not sure how to include this in:

state_attr('sensor.cecil_st_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)[:???])

Why is there 48 now and only 32 before I wonder?

markpurcell · July 15, 2023, 11:45am

After 12:30 they publish the next 24 hours, which is 48 elements.

You will have the full 24 hours until 3:30 , at which points the number of elements reduces. So at 04:00 you will have 47 elements, then at 04:30 you will have 46 elements until you get to 12:00 when you have 32/33 elements which is all the timeslots between 12:00 and 03:30.

Then after 12:30 you get the full 24 hours again.

rcruikshank · July 15, 2023, 11:50am

Should I just set it to 32 and be done with it? Can’t figure out how to include dynamic array counter template in shell command.

{% set general_forecast_sensor = state_attr('sensor.cecil_st_general_forecast', 'forecasts') %}
{% set gfs = general_forecast_sensor | length %}
{{ gfs }}

{% set feedin_forecast_sensor = state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts') %}
{% set ffs = feedin_forecast_sensor | length %}
{{ ffs }}

markpurcell · July 15, 2023, 9:24pm

That is correct it is trying to get to the requested 5% SOC. That is expected and the last couple of results can be ignored as once additional time slots are included at this time it will quickly work out the best optimisation for this timeslot.

markpurcell · July 15, 2023, 9:26pm

Why can’t you use the suggestion from above?

"prediction_horizon": {{
          min(48, (state_attr('sensor.amber_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
        }},

Replace sensor.amber_feed_in_forecast with your sensor.

rcruikshank · July 15, 2023, 10:43pm

I was trying to change the array size of load_cost_forecast and prod_price_forecast by changing the [:48] to something like [:{{sensor.ffg}]
Where ffg is:

  - platform: template
    sensors:
      ffs:
        value_template: >-
          {% set feedin_forecast_sensor = state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts') %}
          {% set ffs = feedin_forecast_sensor | length %}
          {{ ffs }}

Doesn’t work.

You’re saying just change the prediction horizon line (not the lines above) from this:

\"prediction_horizon\":33,\"soc_init\":{{(states('sensor.sonnenbatterie_84324_state_charge_user')|float(0))/100
#        }},

to this:

"prediction_horizon": {{
          min(48, (state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
        }},

Do I need the \“soc_init\” at the end of the output?