Template trigger variables: How to pass an sensor attribute's array?

heinemannj · June 27, 2024, 9:52am

template:
  - trigger:
      - platform: time_pattern
        minutes: "/1"
      - platform: state
        entity_id: sensor.solcast_pv_forecast_prognose_heute
        not_from:
          - "unknown"
          - "unavailable"
          - "none"
        not_to:
          - "unknown"
          - "unavailable"
          - "none"
    action:
      - variables:
          var: >-
            {% from '032-emhass.jinja' import globalVariables %}
            {{ globalVariables()|from_json }}
          node: "{{ state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') }}"
    sensor:
      - name: solcast_forecast_remaining_today_every_minute
        unique_id: solcast_forecast_remaining_today_every_minute
        #friendly_name: "Solcast Forecast Remaining Today (Every Minute)"
        state_class: "total"
        device_class: energy
        unit_of_measurement: "kWh"
        icon: mdi:solar-power
        state: >-
          {%- set node = state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
          {%- set ts_end_time = ((as_timestamp(now().replace(day=now().day+1).replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0))/1800)|round(0,'floor')|int * 1800) %}

          {% from '032-emhass.jinja' import forecastPeriod %}
          {{ forecastPeriod(node, 'period_start', 'pv_estimate', 1, ts_end_time)|float(0) }}
        attributes:
          solar_noon_energy: >-
            {%- set node = state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
            {%- set ts_end_time = today_at(var.batteries_charge_end_time_expected).timestamp() %}

            {% from '032-emhass.jinja' import forecastPeriod %}
            {{ forecastPeriod(node, 'period_start', 'pv_estimate', 1, ts_end_time)|float(0) }}
          solar_noon_time: "{{ var.batteries_charge_end_time_expected }}"
          last_updated: "{{ now() }}"

The data I want to pass (variable node) :

{{ state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') }}

Result type: string:

({'period_start': datetime.datetime(2024, 6, 27, 0, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 0, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 1, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 1, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 2, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 2, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 3, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 3, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 4, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 4, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 5, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 5, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0572, 'pv_estimate10': 0.0491, 'pv_estimate90': 0.0613}, {'period_start': datetime.datetime(2024, 6, 27, 6, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.191, 'pv_estimate10': 0.1513, 'pv_estimate90': 0.3506}, {'period_start': datetime.datetime(2024, 6, 27, 6, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.9809, 'pv_estimate10': 0.6884, 'pv_estimate90': 1.0786}, {'period_start': datetime.datetime(2024, 6, 27, 7, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 1.8331, 'pv_estimate10': 1.814, 'pv_estimate90': 1.8331}, {'period_start': datetime.datetime(2024, 6, 27, 7, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 2.5785, 'pv_estimate10': 2.5785, 'pv_estimate90': 2.5785}, {'period_start': datetime.datetime(2024, 6, 27, 8, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 3.3145, 'pv_estimate10': 3.2996, 'pv_estimate90': 3.3145}, {'period_start': datetime.datetime(2024, 6, 27, 8, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 3.9542, 'pv_estimate10': 3.9542, 'pv_estimate90': 3.9542}, {'period_start': datetime.datetime(2024, 6, 27, 9, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 4.5893, 'pv_estimate10': 4.2112, 'pv_estimate90': 4.5893}, {'period_start': datetime.datetime(2024, 6, 27, 9, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.0199, 'pv_estimate10': 3.8206, 'pv_estimate90': 5.0199}, {'period_start': datetime.datetime(2024, 6, 27, 10, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.3705, 'pv_estimate10': 4.4023, 'pv_estimate90': 5.3705}, {'period_start': datetime.datetime(2024, 6, 27, 10, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.687, 'pv_estimate10': 5.687, 'pv_estimate90': 5.687}, {'period_start': datetime.datetime(2024, 6, 27, 11, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.8804, 'pv_estimate10': 5.8804, 'pv_estimate90': 5.8804}, {'period_start': datetime.datetime(2024, 6, 27, 11, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.9212, 'pv_estimate10': 5.9212, 'pv_estimate90': 5.9212}, {'period_start': datetime.datetime(2024, 6, 27, 12, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.9147, 'pv_estimate10': 5.9147, 'pv_estimate90': 5.9147}, {'period_start': datetime.datetime(2024, 6, 27, 12, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.8449, 'pv_estimate10': 5.8449, 'pv_estimate90': 5.8449}, {'period_start': datetime.datetime(2024, 6, 27, 13, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.6201, 'pv_estimate10': 5.3451, 'pv_estimate90': 5.6487}, {'period_start': datetime.datetime(2024, 6, 27, 13, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 5.0787, 'pv_estimate10': 4.5493, 'pv_estimate90': 5.3362}, {'period_start': datetime.datetime(2024, 6, 27, 14, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 4.5118, 'pv_estimate10': 3.7466, 'pv_estimate90': 5.0047}, {'period_start': datetime.datetime(2024, 6, 27, 14, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 4.0165, 'pv_estimate10': 3.1704, 'pv_estimate90': 4.5683}, {'period_start': datetime.datetime(2024, 6, 27, 15, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 3.67, 'pv_estimate10': 2.8155, 'pv_estimate90': 4.116}, {'period_start': datetime.datetime(2024, 6, 27, 15, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 3.2971, 'pv_estimate10': 2.5526, 'pv_estimate90': 3.5545}, {'period_start': datetime.datetime(2024, 6, 27, 16, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 2.8761, 'pv_estimate10': 2.2693, 'pv_estimate90': 2.9703}, {'period_start': datetime.datetime(2024, 6, 27, 16, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 2.3889, 'pv_estimate10': 2.0047, 'pv_estimate90': 2.5083}, {'period_start': datetime.datetime(2024, 6, 27, 17, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 1.8572, 'pv_estimate10': 1.751, 'pv_estimate90': 1.9501}, {'period_start': datetime.datetime(2024, 6, 27, 17, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 1.1817, 'pv_estimate10': 1.1226, 'pv_estimate90': 1.2408}, {'period_start': datetime.datetime(2024, 6, 27, 18, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.6397, 'pv_estimate10': 0.6077, 'pv_estimate90': 0.6717}, {'period_start': datetime.datetime(2024, 6, 27, 18, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.5089, 'pv_estimate10': 0.4835, 'pv_estimate90': 0.5343}, {'period_start': datetime.datetime(2024, 6, 27, 19, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.408, 'pv_estimate10': 0.3876, 'pv_estimate90': 0.4284}, {'period_start': datetime.datetime(2024, 6, 27, 19, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.3261, 'pv_estimate10': 0.3098, 'pv_estimate90': 0.3424}, {'period_start': datetime.datetime(2024, 6, 27, 20, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.2595, 'pv_estimate10': 0.2465, 'pv_estimate90': 0.2725}, {'period_start': datetime.datetime(2024, 6, 27, 20, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.1946, 'pv_estimate10': 0.1634, 'pv_estimate90': 0.2043}, {'period_start': datetime.datetime(2024, 6, 27, 21, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.1044, 'pv_estimate10': 0.0643, 'pv_estimate90': 0.1096}, {'period_start': datetime.datetime(2024, 6, 27, 21, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0122, 'pv_estimate10': 0.0081, 'pv_estimate90': 0.0122}, {'period_start': datetime.datetime(2024, 6, 27, 22, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 22, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 23, 0, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0}, {'period_start': datetime.datetime(2024, 6, 27, 23, 30, tzinfo=zoneinfo.ZoneInfo(key='Europe/Berlin')), 'pv_estimate': 0.0, 'pv_estimate10': 0.0, 'pv_estimate90': 0.0})

If I pass the node data directly in the sensor state and in the attributes everything works fine.

Passing the node data via action variables results in sensor state unavailable …

The array passed via variable var is working fine:

{%- macro globalVariables() %}
    {{- {
            'inverter_rated_power': 6000,
            'inverter_maximum_active_power': 6600,
            'batteries_rated_capacity': 15,
            'batteries_rated_charge_energy': 17.250,
            'batteries_rated_discharge_power': 6600,
            'batteries_rated_charge_power': 7500,
            'batteries_charge_end_time_expected': '14:00',
            'batteries_soc_setpoint_low': 15,
            'epexspot_quantile_setpoint_low': 0.15,
            'epexspot_quantile_setpoint_neutral': 0.33,
            'epexspot_quantile_setpoint_high': 0.8
        }|to_json -}}
{%- endmacro %}

I really needs to avoid redundant sensor calls because I want’s to setup an Energy Monitoring System with nearly real time monitoring.

FPro · June 27, 2024, 11:31am

Try
node: "{{ state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') | from_json }}"
I think you are trying to pass a String. Not sure what the method exactly does with the variable.

petro · June 27, 2024, 12:08pm

the datetime objects are causing serialization issues. They don’t serialize into a basic object so the python resolver that runs after templates assigns the whole object to a string. You need to cast the datetime object in your list of objects as a string.

Keep in mind, this will affect how you use the objects later on. If you plan to run comparisons, make sure you compare string versions of your dt objects instead of dt objects. I.e. you’ll need to change your forecastPeriod period macro.

node: >
  {% set ns = namespace(items=[]) %}
  {% set target = 'period_start' %}
  {% for info in state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
    {% set dt = [(target, info[target].isoformat())] %}
    {% set remaining = info.items() | list | rejectattr('0','eq', target) | list %}
    {% set ns.items = ns.items + [ dict.from_keys(dt+remaining) ] %}
  {% endfor %}
  {{ ns.items() }}

The other option is to use forecast period in your variables (which is what I’d probably do).

As a sidebar, your ts_end_time calculations shows me that you aren’t utilizing datetime objects and timedelta objects properly. If you post forecastPeriod, I can help you simplify your calculations.

heinemannj · June 27, 2024, 1:47pm

node: array of ‘solcast’ data
dateField: string of ‘solcast’ attribute to filter on
valueField: string of ‘solcast’ attribute to filter on
valueFactor: int to convert between kWh an Wh
ts_end_time: int as a timestamp

{%- macro forecastPeriod(node, dateField, valueField, valueFactor, ts_end_time) %}
    {%- set var = namespace(total = 0) %}
    {%- set var.dateField = dateField %}
    {%- set var.valueField = valueField %}
    {%- set var.valueFactor = valueFactor %}
    {%- set var.ts_end_time = ts_end_time %}
    {%- set var.ts_now = ((as_timestamp(now())/1800)|round(0,'floor')|int * 1800) %}

    {%- if now().minute < 30 %}
        {%- set var.ts_actual_period = today_at(now().hour ~ ':00').timestamp() %}
        {%- set var.remaining_minutes_actual_period = 30 - now().minute %}
    {% else %}
        {%- set var.ts_actual_period = today_at(now().hour ~ ':30').timestamp() %}
        {%- set var.remaining_minutes_actual_period = 60 - now().minute %}
    {%- endif %}

    {%- for record in node %}
        {%- set ts = as_timestamp(record[var.dateField]) %}
        {%- if ts == var.ts_actual_period and ts < var.ts_end_time %}
            {%- set var.total = var.total + (record[var.valueField]|float(0) * 0.5 / var.valueFactor / 30 * var.remaining_minutes_actual_period) %}
        {%- endif %}
        {%- if ts > var.ts_now and ts < var.ts_end_time %}
            {%- set var.total = var.total + (record[var.valueField]|float(0) * 0.5 / var.valueFactor) %}
        {%- endif %}
    {%- endfor %}
    {{- var.total|round(3) -}}
{%- endmacro %}

heinemannj · June 27, 2024, 2:08pm

petro:

  {% set ns = namespace(items=[]) %}
  {% set target = 'period_start' %}
  {% for info in state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
    {% set dt = [(target, info[target].isoformat())] %}
    {% set remaining = info.items() | list | rejectattr('0','eq', target) | list %}
    {% set ns.items = ns.items + [ dict.from_keys(dt+remaining) ] %}
  {% endfor %}
  {{ ns.items() }}

TypeError: ‘list’ object is not callable

petro · June 27, 2024, 2:11pm

change last line to

{{ ns.items }}

heinemannj · June 27, 2024, 2:12pm

This is working in Template editor - Woohhhhhh

heinemannj · June 27, 2024, 2:24pm

petro:

node: >
  {% set ns = namespace(items=[]) %}
  {% set target = 'period_start' %}
  {% for info in state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
    {% set dt = [(target, info[target].isoformat())] %}
    {% set remaining = info.items() | list | rejectattr('0','eq', target) | list %}
    {% set ns.items = ns.items + [ dict.from_keys(dt+remaining) ] %}
  {% endfor %}
  {{ ns.items }}

WORKS!!! - You are the hero of our world #()

template:
  - trigger:
      - platform: time_pattern
        minutes: "/1"
      - platform: state
        entity_id: sensor.solcast_pv_forecast_prognose_heute
        not_from:
          - "unknown"
          - "unavailable"
          - "none"
        not_to:
          - "unknown"
          - "unavailable"
          - "none"
    action:
      - variables:
          var: >-
            {% from '032-emhass.jinja' import globalVariables %}
            {{ globalVariables()|from_json }}
          node: >
            {% set ns = namespace(items=[]) %}
            {% set target = 'period_start' %}
            {% for info in state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') %}
              {% set dt = [(target, info[target].isoformat())] %}
              {% set remaining = info.items() | list | rejectattr('0','eq', target) | list %}
              {% set ns.items = ns.items + [ dict.from_keys(dt+remaining) ] %}
            {% endfor %}
            {{ ns.items }}
    sensor:
      - name: solcast_forecast_remaining_today_every_minute
        unique_id: solcast_forecast_remaining_today_every_minute
        #friendly_name: "Solcast Forecast Remaining Today (Every Minute)"
        state_class: "total"
        device_class: energy
        unit_of_measurement: "kWh"
        icon: mdi:solar-power
        state: >-
          {# set node = state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') #}
          {%- set ts_end_time = ((as_timestamp(now().replace(day=now().day+1).replace(hour=0).replace(minute=0).replace(second=0).replace(microsecond=0))/1800)|round(0,'floor')|int * 1800) %}

          {% from '032-emhass.jinja' import forecastPeriod %}
          {{ forecastPeriod(node, 'period_start', 'pv_estimate', 1, ts_end_time)|float(0) }}
        attributes:
          solar_noon_energy: >-
            {# set node = state_attr('sensor.solcast_pv_forecast_prognose_heute', 'detailedForecast') #}
            {%- set ts_end_time = today_at(var.batteries_charge_end_time_expected).timestamp() %}

            {% from '032-emhass.jinja' import forecastPeriod %}
            {{ forecastPeriod(node, 'period_start', 'pv_estimate', 1, ts_end_time)|float(0) }}
          solar_noon_time: "{{ var.batteries_charge_end_time_expected }}"
          last_updated: "{{ now() }}"

PLEASE:
Every additional improvement on the code is highly appriciated - I’m a bloody newby on templating and jinja …

petro · June 27, 2024, 2:33pm

Ok, so your macro is fine for the most part, but there’s a lot of unnecessary things that just cause some overhead.

This should do the same thing, you might want to test it out.

{%- macro forecastPeriod(node, date, field, factor, ts_end_time) %}
    {%- set var = namespace(total = 0) %}
    {%- set t = now() %}
    {%- set ts_now = ((t.timestamp()/1800)|round(0,'floor')|int * 1800) %}
    {%- set remaining = ((t.timestamp() % 1800) / 60) | round(0, 'floor') %}

    {%- for record in node %}
        {%- set ts = as_timestamp(record[date]) %}
        {%- set value = record[field]|float(0) %}
        {%- if ts == ts_now and ts < ts_end_time %}
            {%- set var.total = var.total + (value * 0.5 / factor / 30 * remaining) %}
        {%- elif ts_now < ts < ts_end_time %}
            {%- set var.total = var.total + (value * 0.5 / factor) %}
        {%- endif %}
    {%- endfor %}
    {{- var.total|round(3) -}}
{%- endmacro %}

heinemannj · June 28, 2024, 5:35am

This is my final code:

{%- macro forecastPeriod(node, date, value, factor, ts_end_time) %}
    {%- set var = namespace(total = 0) %}
    {%- set t = now() %}
    {%- set ts_now = ((t.timestamp()/1800)|round(0,'floor')|int * 1800) %}

    {%- if t.minute < 30 %}
        {%- set remaining = 30 - t.minute %}
    {% else %}
        {%- set remaining = 60 - t.minute %}
    {%- endif %}

    {%- for record in node %}
        {%- set ts = as_timestamp(record[date]) %}
        {%- set value = record[value]|float(0) %}
        {%- if ts == ts_now and ts < ts_end_time %}
            {%- set var.total = var.total + (value * 0.5 / factor / 30 * remaining) %}
        {%- elif ts > ts_now and ts < ts_end_time %}
            {%- set var.total = var.total + (value * 0.5 / factor) %}
        {%- endif %}
    {%- endfor %}
    {{- var.total|round(3) -}}
{%- endmacro %}

Thank you so much for your patience and your spirit !!!