Predicting heat demand from forecast temperature - how to extract/manipulate weather forecast data?

[BuhJuhWuh (Jo)]

Hi all,

Here’s a challenge which I’m hoping you can help me with. I’ll outline what I’m trying to do first, then describe a couple of the specific parts I’m struggling with for starters.

The challenge:

I’m looking at using Predbat for automating our home battery, however, our electricity demand is dominated by our heat pump, and strongly linked to the outside temperature. From the last few months’ historical data I can get a straightforward quadratic curve fit formula which I can use to convert from forecast temperature to a predicted heat demand. This can then be fed as an input to Predbat as described here. I think the steps are, broadly:

1. Extract forecast temperatures from a weather forecast entity, say hourly values for the next 24 hours as per the Forecast Home entity;
2. Use my formula to convert to a set of heat demand values (maybe some other mathematical tweaks to add in future, but let’s start with the basics);
3. Arrange the new values in the prescribed format (Predbat wants this in the form of a sensor with two attributes: timestamps and incrementing heat demand).

So, some questions to start with:

Q1. Choosing the right HA tools - in other languages/platforms, I feel like I could code this in half an hour, but I’m really struggling here with the distinctions and capability differences between sensors, scripts, services, automations, etc. I haven’t found a clear map of this in the documentation anywhere - if I’ve missed something obvious, please point me to it. My current thinking is that this should be achievable in a template sensor - does this seem like the right approach? Or can I use a script to get data from a sensor/service and feed it into a new sensor?

Q2. Getting the forecast data - for step 1 above, it looks like the example here ought to be spot on. (I would then in edit the sensor attributes section at the bottom to perform my calculations and produce an output in the correct format, but let’s get the basics working first.) A few tweaks seem necessary (e.g. entity_id: weather.forecast_home not weather.home), as per this thread. But I can’t seem to get this to work at all - it just results in “Unavailable”.

To be specific, I am pasting the following into a new template sensor:

template:
  - trigger:
      - platform: time_pattern
        hours: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
    sensor:
      - name: Weather Forecast Hourly
        unique_id: weather_forecast_hourly
        state: "{{ now().isoformat() }}"
        attributes:
          forecast: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"

None of the available options in the ‘Unit of measurement’, ‘Device class’ or ‘State class’ menus seem appropriate, so I have left these blank - if these should be set to something specific, please advise.

I can see that the service call seems to work - pasting the following into Developer Tools > Services gets a sensible output:

service: weather.get_forecasts
data:
  type: hourly
target:
  entity_id: weather.forecast_home

But it doesn’t seem to be doing anything when put into the template sensor.

System info as follows, running on a RPi5/8GB:

Any pointers or suggestions much appreciated (along with requests for any crucial info you need to know which I’ve missed!).

Cheers,

Jo

[Didgeridrew]

@BuhJuhWuh

To be specific, I am pasting the following into a new template sensor:

Are you pasting this in the Template Helper creator? If so, that’s your issue.

The “State template” field in the Template Helper creator is only for the state template not for YAML configuration. From your example above the only thing that would be in that field is:

{{ now().isoformat() }}

However, the Template Helper does not currently support advanced features like triggers and actions. Sensors using advanced feature need to be set up in your YAML configuration, either directly in configuration.yaml or a separate, properly merged, file:

Splitting the Config
Packages (aka Splitting the Config: Easy Mode)

[BuhJuhWuh (Jo)]

Thanks, that’s helpful, will give that a go.

[BuhJuhWuh (Jo)]

Okay, one step forward, next hurdle: Really struggling to work out how to extract/address the hourly temperature data contained in the attribute. I can get the initial full set of forecast data (thanks @Didgeridrew ), but not having much joy handling it from there. From the reading I’ve done, selectattr(), map(), and possibly even just dot notation to get to the next level down ought to be helping get a more succinct list of the properties I want (i.e. list of timestamps & temperatures), but not really getting anywhere - any advice much appreciated.

Here’s a few things I’ve tried:

In configuration.yaml:

template:
  - trigger:
      - platform: time_pattern
        #hours: /1
        minutes: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
    sensor:
      - name: "HP Energy Prediction"
        unique_id: HP_energy_prediction
        state: "{{ now().isoformat() }}"
        attributes:
          forecast: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
          forecast2: "{{ hourly_forecast_data['weather.forecast_home'].forecast.temperature }}"
          forecast3: "{{ state_attr('weather.forecast_home','forecast') | map(attribute='temperature') }}"
          forecast4: "{{ hourly_forecast_data['weather.forecast_home'].forecast | selectattr('temperature') }}"
          temp0: "{{ hourly_forecast_data['weather.forecast_home'].forecast[0].temperature }}"

Yields the following:

Putting the following into the Dev Tools Template previewer:

{{ state_attr('weather.forecast_home','forecast') }}

{{ state_attr('weather.forecast_home','forecast') | map(attribute='temperature') }}

{{ state_attr('weather.forecast_home','forecast') | selectattr('temperature') }}

Yields the following:

[{'condition': 'cloudy', 'datetime': '2024-03-04T12:00:00+00:00', 'wind_bearing': 99.8, 'temperature': 1.2, 'templow': 0.7, 'wind_speed': 22.7, 'precipitation': 0.0, 'humidity': 87}, {'condition': 'cloudy', 'datetime': '2024-03-05T12:00:00+00:00', 'wind_bearing': 144.2, 'temperature': 8.9, 'templow': 2.0, 'wind_speed': 34.2, 'precipitation': 0.0, 'humidity': 73}, {'condition': 'partlycloudy', 'datetime': '2024-03-06T12:00:00+00:00', 'wind_bearing': 154.1, 'temperature': 9.4, 'templow': 3.0, 'wind_speed': 23.4, 'precipitation': 0.0, 'humidity': 61}, {'condition': 'sunny', 'datetime': '2024-03-07T12:00:00+00:00', 'wind_bearing': 168.1, 'temperature': 8.9, 'templow': 1.1, 'wind_speed': 22.3, 'precipitation': 0.0, 'humidity': 66}, {'condition': 'partlycloudy', 'datetime': '2024-03-08T12:00:00+00:00', 'wind_bearing': 124.4, 'temperature': 6.3, 'templow': 3.8, 'wind_speed': 9.0, 'precipitation': 0.0, 'humidity': 67}, {'condition': 'sunny', 'datetime': '2024-03-09T12:00:00+00:00', 'wind_bearing': 93.2, 'temperature': 7.1, 'templow': 0.8, 'wind_speed': 17.6, 'precipitation': 0.0, 'humidity': 59}]

<generator object sync_do_map at 0x7f6802be40>

<generator object select_or_reject at 0x7f2e6f4dc0>

Searching for those errors has only yielded a few threads which tend to tail off with “bug”

Any clues or pointers would be much appreciated.

[Didgeridrew]

#2 is returning nothing because hourly_forecast_data['weather.forecast_home'].forecast is a list of dictionaries… There is no single property called temperature.

3 & 4 aren’t errors, they are telling you that your template is returning a generator object.

@BuhJuhWuh

Any clues or pointers would be much appreciated.

What specifically do want each of them to output?

[BuhJuhWuh (Jo)]

Thank you, that gives me some more search terms to go and dig into.

Ultimately, I would like to end up with a sensor of the form specified here. To get there, I think the steps would be:

1. Extract the hourly temperature values (like temp0 above, but for the full 24 hour set) - is there a straightforward way to extract the full set? #2-4 were all my poor attempts to do this;
2. Apply a formula (a basic quadratic y = ax^2 + bx + c based on known data) to each of those temperature values to convert it to an energy value;
3. Reformat that set of ‘energy’ data so that it is in the form of an incrementing series;
4. Include the ‘last_updated’ timestamps.

[Didgeridrew]

@BuhJuhWuh

Include the ‘last_updated’ timestamps.

Where is that data coming from? With a trigger-based template sensor all the attributes are rendered follow every trigger… so you would have to also include the datetime or some other unique identifier then compare incoming value to previous values and overwrite only changed values… but to what end?

Or are do you want the datetime string from the weather forecast?

template:
  - trigger:
      - platform: time_pattern
        hours: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
      - variables:
          forecasts: '{{ hourly_forecast_data["weather.forecast_home"].forecast }}'
    sensor:      
      - name: "HP Energy Prediction"
        unique_id: HP_energy_prediction
        state: "{{ now().isoformat() }}"
        attributes:
          forecast: |
            {% set a = 1 %}
            {% set b = 2 %}
            {% set c = 3 %}
            {% set ns = namespace(x=[]) %}
            {% for f in forecasts %}  
              {% set t = f.temperature | float %}
              {% set energy = a*(t**2) + b*t + c  %}
              {% set ns.x = ns.x + [ (f.datetime, energy) ]%}
            {% endfor %}
            {{ dict(ns.x | sort(attribute=1))}}

Depending on your use and timezone, you might want to localize the datetime string by replacing f.datetime with f.datetime|as_datetime|as_local|string. I think that might make it easier when you are trying to use the data elsewhere.

EDIT: Updated to fix sorting and C/P error

[BuhJuhWuh (Jo)]

@Didgeridrew

Where is that data coming from? With a trigger-based template sensor all the attributes are rendered follow every trigger… Or are do you want the datetime string from the weather forecast?

The latter, I think. (To my mind the name ‘last_updated’ in the spec is a misnomer.)

Many thanks for this - looks very helpful - will go and have a play…

[BuhJuhWuh (Jo)]

Hmm, think we’ve still got something not quite right:

(Trying to follow the logic here, isn’t your line {% set t = f.temperature | float %} effectively trying to do the same thing as my unsuccessful attempt #2 earlier? I might have expected something like {% set t = forecasts[f].temperature | float %}, but I can see that’s not quite right either, so I may have entirely the wrong end of the stick here…)

[Didgeridrew]

@BuhJuhWuh

Hmm, think we’ve still got something not quite right:

I don’t know what you are trying to show with that screen shot… Here’s what mine looks like with the configuration from above (keep in mind my temps are F so the energy values look crazy):

@BuhJuhWuh

isn’t your line {% set t = f.temperature | float %} effectively trying to do the same thing as my unsuccessful attempt #2 earlier?

Kind of…

For each value in the list the loop:

1. Gets the Temp value
2. Runs it through the quadratic equation
3. Pairs the results from the equations with the datetime string in a tuple
4. Adds that tuple to a list

Once all the values have been looped over, the list of tuples is sorted by the energy value then turned into a dictionary.

[BuhJuhWuh (Jo)]

Thanks for the explanation of the logic - that all fits with what I was expecting it to do.

@Didgeridrew

I don’t know what you are trying to show with that screen shot…

It was to show that “Forecast {}” was the entirety of output I was getting at that point.

However, when I look at it again this morning, something has evidently refreshed/repopulated in the background and it’s looking much more plausible. Will check it over properly if/when I get a chance this evening, but thanks very much for your help so far.

[BuhJuhWuh (Jo)]

@Didgeridrew

For each value in the list the loop:

1. Gets the Temp value
2. Runs it through the quadratic equation
3. Pairs the results from the equations with the datetime string in a tuple
4. Adds that tuple to a list

Once all the values have been looped over, the list of tuples is sorted by the energy value then turned into a dictionary.

Okay, looking at the target result, I’m not sure the “list of tuples turned into a dictionary” is quite what we’re after. I think (but happy to be corrected) that it looks more like it’s a list of dictionaries, which would turn the process into:

1. Get temp value, as before;
2. Convert to energy value using quadratic, as before;
3. On each loop, form dictionary with “last_updated” and “energy” entries (containing our datetime and energy values respectively);
4. Append dictionary to list x in namespace ns

My attempt at programming this is below:

template:
  - trigger:
      - platform: time_pattern
        hours: /1
        #minutes: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
      - variables:
          forecasts: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
    sensor:      
      - name: "HP Energy Prediction"
        unique_id: HP_energy_prediction
        state: "{{ now().isoformat() }}"
        attributes:
          #forecast: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
          #temp0: "{{ hourly_forecast_data['weather.forecast_home'].forecast[0].temperature }}"
          HP_Prediction: |
            {# Choose coefficients for quadratic curve fit #}
            {% set a = 0.1536 %}
            {% set b = -3.9739 %}
            {% set c = 31.591 %}
            {% set ns = namespace(x={}) %}
            {% for f in forecasts %}  
              {# For each hourly entry, get temperature value from 'forecasts' variable and store it in 't' #}
              {% set t = f.temperature | float %}
              {# Calculate hourly energy HP prediction based on temperature and known fit curve #}
              {% set energy = "%.2f"|format((a*(t**2) + b*t + c)/24) %}
              {# Want list of dictionaries.  Create dictionary. #}
              {% set y = ({"last_updated" : f.datetime, "energy" : energy}) %}
              {# Add dictionary to list #}
              {% ns.x.append(y) %}
            {% endfor %}
            {{ ns.x }}

But something’s not working. From testing each of the steps in the Dev Tools, I think the append step is the culprit, yielding an error message:

access to attribute 'append' of 'list' object is unsafe.

Would be grateful for your thoughts. Does this approach appear to match the result I’m aiming for? Or is there a better way to achieve this?

Thanks.

[Didgeridrew]

Issues:

• append() is not an allowed method in HA’s Jinja implementation, but you can add lists to lists.
• If you define the type of a namespace’s variable as a dictionary it’s a dictionary not a list.

  - trigger:
      - platform: time_pattern
        hours: /1
        #minutes: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
      - variables:
          forecasts: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
    sensor:      
      - name: "HP Energy Prediction"
        unique_id: HP_energy_prediction
        state: "{{ now().isoformat() }}"
        attributes:
          #forecast: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
          #temp0: "{{ hourly_forecast_data['weather.forecast_home'].forecast[0].temperature }}"
          HP_Prediction: |            
            {# Choose coefficients for quadratic curve fit #}
            {% set a = 0.1536 %}
            {% set b = -3.9739 %}
            {% set c = 31.591 %}
            {% set ns = namespace(x=[]) %}
            {% for f in forecasts %}  
              {# For each hourly entry, get temperature value from 'forecasts' variable and store it in 't' #}
              {% set t = f.temperature | float %}
              {# Calculate hourly energy HP prediction based on temperature and known fit curve #}
              {% set energy = ((a*(t**2) + b*t + c)/24)|round(2) %}
              {# Want list of dictionaries.  Create dictionary. #}
              {% set y = {"last_updated" : f.datetime, "energy" : energy} %}
              {# To existing list ns.x, add dictionary (inside a list because you can only add lists to lists)  #}
              {% set ns.x = ns.x + [y] %}
            {% endfor %}
            {{ ns.x | sort(attribute='energy') | list }}

[BuhJuhWuh (Jo)]

Many thanks, that helped a lot.

In case it helps anyone else, my solution as it stands is as follows (with a few updates to convert the energy values to an incrementing series):

template:
  - trigger:
      - platform: time_pattern
        hours: /1
        #minutes: /1
    action:
      - service: weather.get_forecasts
        data:
          type: hourly
        target:
          entity_id: weather.forecast_home
        response_variable: hourly_forecast_data
      - variables:
          forecasts: "{{ hourly_forecast_data['weather.forecast_home'].forecast }}"
    sensor:      
      - name: "HP Energy Prediction"
        unique_id: HP_energy_prediction
        state: "{{ now().isoformat() }}"
        attributes:
          HP_Prediction: |            
            {# Choose coefficients for quadratic curve fit - specific to your house/heating demand #}
            {% set a = 0.1424 %}
            {% set b = -3.9174 %}
            {% set c = 38.081 %}
            {% set ns = namespace(hourly_energy_list=[]) %}
            {% set ns.incr_energy_list = [] %}
            {% for f in forecasts %}  
              {# For each hourly entry, get temperature value from 'forecasts' variable and store it in 't' #}
              {% set t = f.temperature | float %}
              {# Calculate hourly energy HP prediction based on temperature and known fit curve #}
              {% set hourly_energy = ((a*(t**2) + b*t + c)/24)|round(2) %}
              {# Add to list of hourly energy values #}
              {% set ns.hourly_energy_list = ns.hourly_energy_list + [hourly_energy] %}
              {# Calculate sum of hourly entries so far #}
              {% set incr_energy = ns.hourly_energy_list|sum %}
              {# Create dictionary entry #}
              {% set incr_entry = {"last_updated" : f.datetime, "energy" : incr_energy} %}
              {# Add to list of dictionary entries #}
              {% set ns.incr_energy_list = ns.incr_energy_list + [incr_entry] %}
            {% endfor %}
            {{ ns.incr_energy_list }}

This is working for me (can see the effect in Predbat’s load forecast column), adding the following lines in Predbat’s apps.yaml:

# Load forecast can be used to add to the historical load data (heat-pump)
# To link to Predheat
# Data must be in the format of 'last_updated' timestamp and 'energy' for incrementing kWh
  load_forecast:
    - sensor.hp_energy_prediction$HP_Prediction

Now, a final cherry on the cake would be to update the state from its placeholder timestamp to show the total forecast energy (i.e. the sum of the hourly_energy values, or the final value in the incr_energy_list). I’ve had a quick play with this.attribute.whatever - but will need to give that more attention - is it possible to grab an attribute value to use as the main state? (I understand this might be the previous value rather than the current one, but can live with that.)