How to sum values from lists, that are attributes for 2 different template sensors, if they "match" on a different attribute's list value

Configuration
1

Hi folks,

I’m new to templating (and coding in general), so there is likely an easy way to do what I am tryting to acheive. But Googling and searching hasn’t yielded an answer (or, not one that my baby-coding-brain recognised was applicable).

TL:DR question:
How can I create a template sensor that outputs a list of values, that is the sum of values of list items from 2 separate template sensors, if the corresponding value in another attribute list for each template sensor matches (e.g. the first value of template 1 attribute 1 and template 2 attribute 1 match, so give me a list entry in the new template sensor that is template 1 attribute 2 + template 2 attribute 2)

  • Maybe think of it ike an Excel vlookup using the value of attribute 1 as the “key”, and summing the corresponding values?

Longer version of intention, context, and background:

  • Overall intention: To create merged solar power forecast data to use in Home Assistant automations and as inputs into other template sensors
  • Specific intention now: I want to create a template sensor with 2 attributes: period_end_combined and pv_estimate_combined.
  • This template sensor will pull data from 2 existing (and working!) template sensors
    • The unique_ids of these 2 existing template sensors are solcast_forecast_data_se and solcast_forecast_ data_nw (which report transformed API data for predicted solar power for a split array - one facing south-east, the other north-west)
    • These template sensors each contain 2 attributes: period_end and pv_estimate
    • Each of these attributes contains a list of values. These values are “matched” between the lists, in that they were parsed from returned objects of an API request. I doubt that they are actually “linked” in any way, but the values in the first row of the list go together, as do the second row, third, etc.
  • There is an equal count of values for each of the two attributes within the same template sensor, but a different count of values in attributes between the 2 template sensors, for reasons.
  • There may also be an overlap, but not an identical replication, of the period_end list values stored in each of the parent templates’ attribute, for other reasons. This is why I want to “match” on those values.
  • The new template sensor should have 2 attributes: period_end_combined and pv_estimate_combined
    • period_end_combined will be a list of the period_end values when they appeared in both of the attribute listed values of the parent template sensors (solcast_forecast_data_se and solcast_forecast_data_nw) - basically, used as a “key” to match.
    • pv_estimate_combined will be the sum of the “corresponding” values in each of the parent template sensors’ pv_estimate attribute value list

Help?

I am also very happy to hear other approaches to the whole problem. However, the next step is to output the 2 lists (each _combined attribute) in a CSV, so an array won’t work as the output. But maybe as part of the processing? :man_shrugging:

YAML code:

Final template sensor summing the listed values in attributes of 2 parent templates where their corresponding value in another attribute matches between the two parents:

…no idea. I assume it will include something about for i in range, maybe match, some map(stateattr() ), but I honestly have no idea. I’m too baby.

“Parent” template sensors:

  - platform: template
    sensors:

      # South-East template sensor
      solcast_24hrs_forecast_se:
        unique_id: "solcast_24hrs_forecast_se"
        value_template: "OK"
        attribute_templates:
          pv_estimate: >-
            {%- set pv_estimate = state_attr('sensor.solcast_forecast_data_se', 'forecasts') | map(attribute='pv_estimate') | list %}
            {%- set values_all = namespace(all=[]) %}
            {% for i in range(pv_estimate | length) %}
              {%- set v = (pv_estimate[i] | float |multiply(1000) ) | int(0) %}
              {%- set values_all.all = values_all.all + [ v ] %}
            {%- endfor %} {{ (values_all.all) }}
          period_end: >-
            {{ (state_attr('sensor.solcast_forecast_data_se', 'forecasts') | map(attribute='period_end') | list) }}

      # North-West template sensor
      solcast_24hrs_forecast_nw:
        unique_id: "solcast_24hrs_forecast_nw"
        value_template: "OK"
        attribute_templates:
          pv_estimate: >-
            {%- set pv_estimate = state_attr('sensor.solcast_forecast_data_nw', 'forecasts') | map(attribute='pv_estimate') | list %}
            {%- set values_all = namespace(all=[]) %}
            {% for i in range(pv_estimate | length) %}
              {%- set v = (pv_estimate[i] | float |multiply(1000) ) | int(0) %}
              {%- set values_all.all = values_all.all + [ v ] %}
            {%- endfor %} {{ (values_all.all) }}
          period_end: >-
            {{ (state_attr('sensor.solcast_forecast_data_nw', 'forecasts') | map(attribute='period_end') | list) }}

Output of parent template sensors, taken from Developer Tools → States [Attributes] (same format for each, trimmed):

pv_estimate: 56, 65, 66, 69, 524, 1195, 1839, 2429, 3003, 3484, 3890, 4244, 4507, 4681, 4826, 4855, 4835, 4765, 4580, 4337, 4037, 3627, 3154, 2588, 1925, 1175, 132, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 22
period_end: 2025-01-30T21:00:00.0000000Z, 2025-01-30T21:30:00.0000000Z, 2025-01-30T22:00:00.0000000Z, 2025-01-30T22:30:00.0000000Z, 2025-01-30T23:00:00.0000000Z, 2025-01-30T23:30:00.0000000Z, 2025-01-31T00:00:00.0000000Z, 2025-01-31T00:30:00.0000000Z, 2025-01-31T01:00:00.0000000Z, 2025-01-31T01:30:00.0000000Z, 2025-01-31T02:00:00.0000000Z, 2025-01-31T02:30:00.0000000Z, 2025-01-31T03:00:00.0000000Z, 2025-01-31T03:30:00.0000000Z, 2025-01-31T04:00:00.0000000Z, 2025-01-31T04:30:00.0000000Z, 2025-01-31T05:00:00.0000000Z
...

Further background code, if it helps at all…

One of the 2 Rest sensors that get the data that the “parent” template sensors process:

sensor:
  # South-East REST sensor:
  - platform: rest
    name: Solcast forecast data SE
    unique_id: "solcast_forecast_data_se"
    resource: https://api.solcast.com.au/rooftop_sites/RooftopIDNotTelling/forecasts?format=json
    headers:
      Authorization: Bearer NotTelling
    method: GET
    device_class: power
    force_update: true
    json_attributes:
      - forecasts
    value_template: "OK"
    unit_of_measurement: "kW"
    scan_interval: 86400000   # This is a huge number on purpose

Returned JSON for REST sensor (sample):

{
  "forecasts": [
    {
      "pv_estimate": 3.31,
      "pv_estimate10": 3.0949,
      "pv_estimate90": 3.31,
      "period_end": "2025-01-26T00:30:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 3.3267,
      "pv_estimate10": 3.0437,
      "pv_estimate90": 3.3267,
      "period_end": "2025-01-26T01:00:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 3.0302,
      "pv_estimate10": 2.379,
      "pv_estimate90": 3.3311,
      "period_end": "2025-01-26T01:30:00.0000000Z",
      "period": "PT30M"
    },
  ]
}
2

This probably would not be very difficult, if only I could understand what you want. There is just too much text and too hard for me to follow. Make it simple:

  • Post several (abbreviated) dummy sensor outputs. Does not need to contain tons of sensor values, a handful or even just two would be enough.

  • Expliticly mark out which values in which lists are expected to match.

  • Explicitly mark out which values should be added together when this match is made.

  • Explicitly mark out what the expected result would be, both for the individual operation and the sensor as a whole.

My assumption is that you want to add together some values which have the same period_end time. But how may that time vary? Are they expected to be exactly identical strings between the sensors so they can be compared with a simply == or is some allowance for inexact matches necessary?

This… makes no sense. You must have some ones and twos crossed. Also what do you mean by template in this case, I assume a key? So call it a key. There are way too many things called template already in HASS…

1 Like
3

Is this what you are looking for?

image
image1516×477 26.8 KB

{% set list_1 = [ 1, 2, 3, 4] %}
{% set list_2 = [ 2, 2, 2, 2] %}
{{ zip(list_1, list_2) | map('sum') | list }}
1 Like
4

Again, I may have completely misunderstood what it is you are after… but possibly it may be something at least a little like this:

{# set entity_id = list of sensor entity ids #}

{% set period_end_combined = entity_id
| map('state_attr', 'period_end')
| sum(start=[]) | unique | sort %}

{% set pv_estimate_filler = zip(period_end_combined,
[0] * period_end_combined | count)) | list %}

{% set pv_estimate_entries = namespace(_=[]) %}
{% for entity_id in entity_id %}
  {% set pv_estimate_entries._ = pv_estimate_entries._ +
  [entity_id, (zip(state_attr(entity_id, 'period_end'),
  state_attr(entity_id, 'pv_estimate')) | list +
  pv_estimate_filler) | unique(attribute=0)
  | sort(attribute=0) | map(attribute=1) | list] %}
{% endfor %}

{% set pv_estimate_combined = zip(*(
pv_estimate_entries._ | map(attribute=1) | list
)) | map('sum') | list %}

Completely untested of course, so bound to be something I’ve not taken into consideration or typos. Probably can be rewritten to be more efficient, but this is what I came up with killing time at work :stuck_out_tongue:

How it works:

  • Expects entity_id to be a list of entity ids for your sensors.

  • Create a list of all used period_end values, as your question makes it sound like not all values will appear in all lists (otherwise this would be much easier).

  • Create a filler list mapping all possible period_end values to 0, to use when one of the sensors is missing a value for a certain period.

  • Create a namespace for storing the result, as this is the only way to get values out of a loop with the variable scope limitations in Jinja.

  • Loop over all sensors:

    • Create item list mapping period_end against pv_estimate.
    • Add our filler list at the end.
    • Filter out non-unique period_end (whose pv_estimate will be 0).
    • Sort list on period_end, then map to pv_estimate only.

  • Zip the pv_estimate vectors together, then sum each list item in the result.

1 Like
5

Thanks @Mayhem_SWE ! You’re right, lots of text. It’s the pain of being the beginner and not knowing what is relevant or useful.

You are right, I want to add together some values which have the same period_end time.

Per your points:

South-east predicted solar output at timestamp
sensor.solcast_forecast_date_se:

  • Attributes are period_end and pv_estimate
    • period_end is: [2025-01-26T00:30:00.0000000Z, 2025-01-26T01:00:00.0000000Z, 2025-01-26T01:30:00.0000000Z, … ] # (A list of timestamps)
    • pv_estimate is: [10, 230, 700, … ] # (A list of integers)

North-west predicted solar output at timestamp
sensor.solcast_forecast_date_nw:

  • Attributes are period_end and pv_estimate
    • period_end is: [2025-01-26T00:30:00.0000000Z, 2025-01-26T01:17:00.0000000Z, 2025-01-26T01:30:00.0000000Z, … ] # ( A list of timestamps)
    • pv_estimate is: [0, 100, 250, … ] # (A list of integers)

I want to match on the value of period_end in both sensors. I want an exact match. These should be treated as the matching key.

The values that should be added together are the pv_estimate values, if the match is made on period_end.

The expected result would be the following attributes and values:

  • period_end_combined: [2025-01-26T00:30:00.0000000Z, 2025-01-26T01:30:00.0000000Z, … ]
    • Only these two matched, so only they are returned. The second one didn’t match.
  • pv_estimate_combined: [10, 950, …]
    • Only 2 of the timestamps matched, so only the two that did got sum’ed

And yes, I did have a typo. Thank you for pointing it out - I’ve edited it now.

6

@TheFes Kind of! but there is a part missing.

The list_1 and list_2 data are in different sensors. Which is fine - I know how I would pull those values rather than hard-coding the values.

But, each of those sensors also has another list of timestamps. Those timestamps are associated with each value for list_1 and list_2.

I only want to add the list_1 and list_2 values together if the timestamp associated with them matches.

But this is excellent. I’ve never used zip(before, so it might be what I’m after?

7

But are the end_period vectors between the sensors always the exact same, or can they differ? If they are the same, just zip and sum.

If they can be different, normalize the output from sensors so that both lists have values for every end_period even if that value is 0. See how my example does it, which can also handle an arbitrary number of lists/sensors.

Or do you only want to sum values where both sensors provide a value, and discard the rest? Then I would start by putting together that specific list of timestamps which exist in both sensors.

1 Like
8

My 2 cts, you need to change the data to see if this does what you want

{% set d1 = {
  "forecasts": [
    {
      "pv_estimate": 10,
      "pv_estimate10": 3.0949,
      "pv_estimate90": 3.31,
      "period_end": "2025-01-26T00:30:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 20,
      "pv_estimate10": 3.0437,
      "pv_estimate90": 3.3267,
      "period_end": "2025-01-26T01:00:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 30,
      "pv_estimate10": 2.379,
      "pv_estimate90": 3.3311,
      "period_end": "2025-01-26T01:30:00.0000000Z",
      "period": "PT30M"
    },
  ]
}
%}

{% set d2 = 
{
  "forecasts": [
    {
      "pv_estimate": 1,
      "pv_estimate10": 3.0949,
      "pv_estimate90": 3.31,
      "period_end": "2025-01-26T00:30:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 2,
      "pv_estimate10": 3.0437,
      "pv_estimate90": 3.3267,
      "period_end": "2025-01-26T01:00:00.0000000Z",
      "period": "PT30M"
    },
    {
      "pv_estimate": 3,
      "pv_estimate10": 2.379,
      "pv_estimate90": 3.3311,
      "period_end": "2025-01-26T01:30:00.0000000Z",
      "period": "PT30M"
    },
  ]
}
%}
{% set ns = namespace(s=0,r=[]) %}
{% for a in d1['forecasts'] %}
{% for b in d2['forecasts'] %}
{% if a.period_end == b.period_end %}
{% set ns.s = ns.s + a.pv_estimate + b.pv_estimate %}
{% set ns.r = ns.r + [{ a.period_end  : (a.pv_estimate + b.pv_estimate) } ] %}
{% endif %}
{% endfor %}
{% endfor %}
{{ ns.s }}
{{ ns.r }}
9

Thanks @Mayhem_SWE that’s really helpful.

I did get an error when trying your suggested code:

TypeError: unsupported operand type(s) for +: 'zip' and 'zip'

But I can definitely see the logic in your proposed solution. And it looks good, except for the error.

I’m going to mark your entry as the solution, because I think it’s really good and offeres a few different ways that other people could customise it to give what they need.

For me, I think I will try to do some more processing in the parent sensor so that the item numbers for the pv_estimate values are the same for the 2 parent sensors. It seems like having to do a match on another variable is a step above what I’m capable of at the moment, even with your excellent suggestions.

Appreciate the time you (and @TheFes took to help me learn.

10

My memory said that the result of a zip() would be a list, probably because I most often feed that directly into dict() and that works. But the result is actually a “zip object”, which I guess under some circumstances is automatically cast to a list.

But not in this instance, so I’ve now modified the code and added two manual list filters which hopefully should solve things for you. (The third zip at the end does not need a list filter, as map can operate on a zip object. Just need to cast the result of the map to a list at the very end of the chain.)

1 Like