How are you retrieving your data? Maybe a problem with your long lived access token? Your HA URL? A firewall?
To exclude those variables, I used curl from within the same container that is running emhass, using the same URL and access token as I have specified in secrets.yaml. So there should be nothing different from what my curl command does and what emhass does. Curl returns full history, no complaints. See example below where I can get just over three days of history.
I have set days_to_retrieve: 2 in configuration, so I have a margin.
(Iāve masked some data below)
root@2b43e81d5699:/app# curl -H "Authorization: Bearer XXXXXXXXXX" -H "Content-Type: application/json" 'https://my.homeassistant.url:8123/api/history/period/2024-04-14T00%3A00%3A00%2B02%3A00?filter_entity_id=sensor.template_emhass_no_var_load_2&no_attributes' |python3 -m json.tool|head
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 3392k 100 3392k 0 0 3653k 0 --:--:-- --:--:-- --:--:-- 3652k
[
[
{
"entity_id": "sensor.template_emhass_no_var_load_2",
"state": "443.94",
"attributes": {},
"last_changed": "2024-04-13T22:00:00+00:00",
"last_updated": "2024-04-13T22:00:00+00:00",
"context": {
"id": "01HVG824J3QD58P2P1KHNK5EBW",
...
Any tips on how I can set up and debug emhass locally would be appreciated. I have cloned the repo.
does anybody know, what is wrong here? from time to time I get such errors:
2024-04-17 09:00:05,305 - web_server - INFO - Passed runtime parameters: {'set_def_constant': [True, True], 'pv_power_forecast': [6924.5, 8551.6, 11072.0, 13102.3, 14320.6, 15234.5, 15798.5, 16068.1, 16060.099999999999, 15728.300000000001, 15302.0, 14895.4, 14185.1, 13361.800000000001, 12360.6, 10981.3, 9309.900000000001, 7615.599999999999, 5822.3, 3873.9], 'def_total_hours': [3.0, 0], 'prediction_horizon': 20, 'load_cost_forecast': [16.356, 16.356, 16.0356, 16.0356, 16.056, 16.056, 15.9192, 15.9192, 16.243199999999998, 16.243199999999998, 16.080000000000002, 16.080000000000002, 16.668, 16.668, 16.5252, 16.5252, 17.590799999999998, 17.590799999999998, 20.4708, 20.4708], 'prod_price_forecast': [9.24, 9.24, 9.23, 9.23, 9.22, 9.22, 9.21, 9.21, 9.2, 9.2, 9.21, 9.21, 9.22, 9.22, 9.23, 9.23, 9.24, 9.24, 9.25, 9.25]}
2024-04-17 09:00:05,306 - web_server - INFO - >> Setting input data dict
2024-04-17 09:00:05,306 - web_server - INFO - Setting up needed data
2024-04-17 09:00:05,312 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-17 09:00:10,856 - web_server - INFO - Retrieving weather forecast data using method = list
2024-04-17 09:00:10,860 - web_server - INFO - Retrieving data from hass for load forecast using method = mlforecaster
2024-04-17 09:00:10,862 - web_server - INFO - Retrieve hass get data method initiated...
2024-04-17 09:01:25,776 - web_server - ERROR - Exception on /action/naive-mpc-optim [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 1463, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 872, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 870, in full_dispatch_request
rv = self.dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/flask/app.py", line 855, in dispatch_request
return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args) # type: ignore[no-any-return]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/web_server.py", line 108, in action_call
input_data_dict = set_input_data_dict(config_path, str(data_path), costfun,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/command_line.py", line 127, in set_input_data_dict
P_load_forecast = fcst.get_load_forecast(method=optim_conf['load_forecast_method'], set_mix_forecast=True, df_now=df_input_data)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/emhass/forecast.py", line 664, in get_load_forecast
data = pd.DataFrame.from_dict(data_dict)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 1760, in from_dict
return cls(data, index=index, columns=columns, dtype=dtype)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/frame.py", line 709, in __init__
mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 481, in dict_to_mgr
return arrays_to_mgr(arrays, columns, index, dtype=dtype, typ=typ, consolidate=copy)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 115, in arrays_to_mgr
index = _extract_index(arrays)
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/dist-packages/pandas/core/internals/construction.py", line 655, in _extract_index
raise ValueError("All arrays must be of the same length")
ValueError: All arrays must be of the same length
this is the command:
curl -i -H "Content-Type: application/json" -X POST -d '{
"set_def_constant":[true,true],
"pv_power_forecast":[7793.1, 8550.2, 11911.199999999999, 10691.1, 10649.1, 13500.9, 15624.1, 16068.1, 16060.099999999999, 15728.300000000001, 15302.0, 14895.4, 14185.1, 13361.800000000001, 12360.6, 10981.3, 9309.900000000001, 7615.599999999999, 5822.3, 3873.9],
"def_total_hours":[2.0,1],
"prediction_horizon":20,
"load_cost_forecast":[16.356, 16.356, 16.0356, 16.0356, 16.056, 16.056, 15.9192, 15.9192, 16.243199999999998, 16.243199999999998, 16.080000000000002, 16.080000000000002, 16.668, 16.668, 16.5252, 16.5252, 17.590799999999998, 17.590799999999998, 20.4708, 20.4708],
"prod_price_forecast":[9.25, 9.26, 9.25, 9.24, 9.23, 9.22, 9.21, 9.2, 9.21, 9.22, 9.23, 9.24, 9.25, 9.26, 9.25, 9.24, 9.23, 9.22, 9.21, 9.2]}' http://localhost:5000/action/naive-mpc-optim
it seems, it has to do with the load forecast. but I donāt know why. currently I use naive, and history of the sensor seems fineā¦
To check if it is a connection issue, do the same test but not from within the EMHASS container. Do it from another PC if possible in the same network or even in other network. It should even work from a distant network provided the LLA token and the correct reachable URL
It works well from another PC on the same network. I ran the exact same call from another server and got over three days of history, as expected.
Ok then try to debug using a simple python script like this one to check if you are able retrieve the data:
import pandas as pd
from requests import get, post
days_list = pd.date_range(end='2016-10-27', freq='D', periods=5)
day = days_list [0]
var = 'sensor.your_sensor'
hass_url= 'PUT YOUR HA URL HERE'
long_lived_token = 'PUT YOUR LLA TOKEN HERE'
url = hass_url+"api/history/period/"+day.isoformat()+"?filter_entity_id="+var
headers = {
"Authorization": "Bearer " + long_lived_token,
"content-type": "application/json",
}
# Then once that all this is defined there is no much to it than this:
response = get(url, headers=headers)
# And extract the data >> This is where your code is failing so check this part:
data = response.json()[0]
# If successful then print
print(data)
I have two home assistant instances for two locations. Each location have a photovoltaik. The remote location send the data with the integration āremote homeassistantā. Now i have the problem that the sensor āsensor.p_pv_forecastā in both HA instances exist. I need for the remote location a different sensor name (e.g. "sensor.p_pv_forecast_location1.
ist there a way to do that?
Yes.
It is documented here: https://emhass.readthedocs.io/en/latest/intro.html#the-publish-data-specificities
You need to use a custom name when calling the publish-data API endpoint. Like this:
shell_command:
publish_data: "curl -i -H \"Content-Type:application/json\" -X POST -d '{\"custom_pv_forecast_id\": {\"entity_id\": \"sensor.p_pv_forecast_remote_location\", \"unit_of_measurement\": \"W\", \"friendly_name\": \"Load Power Forecast on my remote location\"}}' http://localhost:5000/action/publish-data"
Or use the publish_prefix runtime parameter to the publish_data command from one of the instances so you prefix all published variables.
https://emhass.readthedocs.io/en/latest/intro.html#passing-other-data
I use it to easier find all emhass created sensors:
emhass_publish_data:
url: 'http://localhost:5000/action/publish-data'
method: 'POST'
content_type: 'application/json'
payload: '{"publish_prefix": "emhass_"}'
I ran that, with the following minor adjustment to the query. I find it a tad odd that I canāt ask for more than five days of history, even though the recorder default i 10 days in HAā¦ but the periods parameter is the same as days_to_retrieve in config, right? I have it set to 2 in my config.
days_list = pd.date_range(end='2024-04-17', freq='D', periods=3)
and got a massive response with history data.
...
{'entity_id': 'sensor.template_emhass_no_var_load_2', 'state': '458.83', 'attributes': {'unit_of_measurement': 'W', 'device_class': 'power'}, 'last_changed': '2024-04-15T21:58:25.665441+00:00', 'last_updated': '2024-04-15T21:58:25.665441+00:00', 'context': {'id': '01HVG824J3QD58P2P1KHNK5EBW', 'parent_id': None, 'user_id': None}},
...
Do you want me to transfer this discussion to an issue on Github instead?
Yes please, that would be better.
But I donāt understand how is this possible? Are you saying that with periods=5 it failed?
Nah, sorry.
It works with periods=5. I just mixed some things up hereā¦
Iāll take this to an issue on Github.
I try (copy) the command. But i donāt get the sensor (sensor.p_pv_forecast_remote_location)
Any idea?
I have to correct my post from above. The problem occurs, when using mlforecaster.
And it seems related to (failing?) forecast-model_fit
I run forecast_model_fit at 6:30 am. (sadly, I see no logs, because already gone - is there a way to access the logs at least for some days back?). all following naive-mpc-optim seem to fail then with the same error. The reason seems to be the sensor.p_load forecast, which just has a value for 6:30, but no more.
When I run forecast-model-fit now (7:15 am.), naive-mpc-optim then works.
and the sensor.p_load_forecast has all valuesā¦
Does somebody know why this happens?
At least more Logs would help to understand betterā¦
Now i have the remote sensor. The reason HA need a full restart (not quick restart). Sorry for the confusion
Awesome tool, however to make it more user friendly I have a question regarding the power and def_total_hours:
Can I obtain them from an input_number / sensor within HA instead of hard coding them into the EMHASS config file like for example the var_PV which comes from the solar power sensor?
Yes
Example:
"def_total_hours": [
{%- if states('weather.openweathermap') == 'sunny' and states('sensor.cecil_st_general_price') | float(0) < 0.13 -%}
{%- if is_state('sensor.season', 'winter') -%}
{{1}}
{%- elif is_state('sensor.season', 'summer') -%}
{{4}}
{%- else -%}
{{3}}
{%- endif -%}
{%- else -%}
{{1}}
{%- endif -%},
{%- if is_state('device_tracker.ynot_location_tracker', ['home']) -%}
{%- if is_state('binary_sensor.ynot_charger', ['on']) -%}
{{ ((80-(states('sensor.ynot_battery')|int(0)))/30*3)|int(0) }}
{%- else -%}
0
{%- endif -%}
{%- else -%}
0
{%- endif -%}
],
Passed in a RESTful POST.
Complete template for the post:
{
"prod_price_forecast": {{
([states('sensor.cecil_st_feed_in_price')|float(0)] +
(state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list))
| tojson
}},
"load_cost_forecast": {{
([states('sensor.cecil_st_general_price')|float(0)] +
state_attr('sensor.cecil_st_general_forecast', 'forecasts') |map(attribute='per_kwh')|list)
| tojson
}},
"pv_power_forecast": {{
([states('sensor.sonnenbatterie_84324_production_w')|int(0)] +
state_attr('sensor.solcast_pv_forecast_forecast_today', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list +
state_attr('sensor.solcast_pv_forecast_forecast_tomorrow', 'detailedForecast')|selectattr('period_start','gt',utcnow()) | map(attribute='pv_estimate')|map('multiply',1000)|map('int')|list
)| tojson
}},
"load_power_forecast": {{
([states('sensor.house_power_consumption_less_deferrables')|int(0)] +
states('input_text.fifo_buffer').split(',') | map('int') | list)
}},
{#"load_power_forecast": {{"[500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500, 500]"
}},#}
"prediction_horizon": {{
min(48, (state_attr('sensor.cecil_st_feed_in_forecast', 'forecasts')|map(attribute='per_kwh')|list|length)+1)
}},
"num_def_loads": 2,
"def_total_hours": [
{%- if states('weather.openweathermap') == 'sunny' and states('sensor.cecil_st_general_price') | float(0) < 0.13 -%}
{%- if is_state('sensor.season', 'winter') -%}
{{1}}
{%- elif is_state('sensor.season', 'summer') -%}
{{4}}
{%- else -%}
{{3}}
{%- endif -%}
{%- else -%}
{{1}}
{%- endif -%},
{%- if is_state('device_tracker.ynot_location_tracker', ['home']) -%}
{%- if is_state('binary_sensor.ynot_charger', ['on']) -%}
{{ ((80-(states('sensor.ynot_battery')|int(0)))/30*3)|int(0) }}
{%- else -%}
0
{%- endif -%}
{%- else -%}
0
{%- endif -%}
],
"P_deferrable_nom": [1300, {{ (states('input_number.ev_amps') | int(0) * 230)|int(0) }}],
"treat_def_as_semi_cont": [1, 0],
"set_def_constant": [0, 0],
"soc_init": {{ (states('sensor.sonnenbatterie_84324_state_charge_user')|int(0))/100 }},
"soc_final": 0.24,
"alpha": 0,
"beta": 1
}
This produces something like this:
{
"prod_price_forecast": [0.09, 0.08, 0.09, 0.11, 0.09, 0.09, 0.09, 0.09, 0.08, 0.08, 0.08, 0.09, 0.09, 0.09, 0.08, 0.06, 0.05, 0.04, 0.03, 0.03, 0.01, 0.0, 0.0, -0.01, -0.01, 0.0, 0.0, 0.0, 0.29, 0.29, 0.29, 0.3, 0.31, 0.35, 0.4, 0.59, 0.59, 0.57, 0.43, 0.42, 0.15, 0.15, 0.14, 0.11, 0.15, 0.15, 0.15, 0.14, 0.13],
"load_cost_forecast": [0.19, 0.18, 0.19, 0.21, 0.2, 0.2, 0.19, 0.19, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.18, 0.15, 0.15, 0.14, 0.13, 0.13, 0.12, 0.12, 0.11, 0.1, 0.1, 0.11, 0.11, 0.11, 0.38, 0.38, 0.38, 0.39, 0.4, 0.45, 0.5, 0.7, 0.7, 0.68, 0.53, 0.52, 0.25, 0.25, 0.24, 0.22, 0.25, 0.25, 0.25, 0.25, 0.24],
"pv_power_forecast": [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 63, 303, 770, 1197, 1612, 2003, 2416, 2655, 2730, 2680, 2489, 2323, 2213, 2151, 2161, 2100, 1985, 1705, 1321, 839, 266, 25, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 76, 409, 928, 1400, 1870, 2353, 2830, 3156, 3417, 3586, 3698, 3693, 3630, 3509, 3266, 2926, 2448, 1967, 1482, 978, 443, 44, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
"load_power_forecast": [359, 310, 294, 289, 303, 304, 314, 292, 284, 291, 301, 304, 278, 276, 280, 310, 336, 1053, 528, 752, 536, 490, 516, 1557, 1351, 1259, 1304, 393, 341, 375, 494, 459, 466, 555, 1798, 1246, 1010, 566, 686, 679, 538, 553, 541, 505, 718, 469, 462, 444, 461],
"prediction_horizon": 48,
"num_def_loads": 2,
"def_total_hours": [1,0],
"P_deferrable_nom": [1300, 3910],
"treat_def_as_semi_cont": [1, 0],
"set_def_constant": [0, 0],
"soc_init": 0.2,
"soc_final": 0.24,
"alpha": 0,
"beta": 1
}
And can be passed to EMHASS in a curl command or the way I do it is in Node-Red in an http request node.
I think thatās what youāre referring to?
1 Like
Yes! Exactly , I was missing the curl part instead of just putting only sensor.name into the emhass.conf!
Thanks a lot!
Deferrable load is moved forward and sometimes back. Is my problem that I run mpc every 5 minutes? Is it better for me, who has an electricity price based on a full hour, to run post_naive_mpc every full hour or what do I gain by running it more often?
OK, here is my first post, just started EMHASS from apr 22 at 8.00am and with help of @markpurcell and @Johan71 I managed to program an automation, see results here:
There is a catch here, the automation really wants to follow the forecasted SOClvl, so I will consume power from the grid as because of this:
I wonder if I should prevent this or just let it find itās way the coming days?
Any advice? I use day ahead-optimization once at 23::57, maybe run it more often, or as said program an automation to prevent charge from the grid.
1 Like