Jinja select not working with "in" test as expected

Hi,

I’m currently working on an automation to push schedule changes to my TRV’s.
I use a state_changed trigger combined with template conditions to monitor changes on an scheduler objects (third party integration “scheduler component”).

Problem

The states object contains a list of target entities, and could contain multiple entities, and my TRV’s follow a simple name scheme (trv_<floor>_<room>). Therefore, my first template condition looks like this (*):

{{ 'climate.trv_' | select('in',entities) }}

But: This doesn’t work. I started Testing in the dev tools template editor to check what wen’t wrong:

{{ ['abc','a','b'] | select('in','a') | list }}
{{ 'a' | select('in',['abc','a','b']) | list }}
{{ {"a":"a","b":"b","abc":"abc"} | select('in','a') | list }}
{{ 'a'  | select('in',{"a":"a","b":"b","abc":"abc"}) | list }}

All of them return ['a'], which I would expect using the eq (aka ==) test. Even more confusing to me, all the following will return ['a','b']:

{{ ['abc','a','b'] | select('in','abe') | list }}
{{ 'abe' | select('in',['abc','a','b']) | list }}
{{ {"a":"a","b":"b","abc":"abc"} | select('in','abe') | list }}
{{ 'abe'  | select('in',{"a":"a","b":"b","abc":"abc"}) | list }}

And I totally don’t understand what they compute there to get this result (if changing ‘abe’ to ‘babe’ ['b','a','b'] gets the result giving me atleast some hint what is done computationally but doesn’t make any sense to me).

Btw, using actuall data:

{{ 'climate.trv_' | select('in',['climate.trv_3_br']) | list }}
{{ ['climate.trv_3_br'] | select('in','climate.trv_') | list }}

result in an empty list.

Expectation

I would expect ['a','abc'] for the first example and [] for the second, as this is whats done in python

Python 3.10.8 (main, Nov  1 2022, 14:18:21) [GCC 12.2.0] on linux
...
>>> list(map(lambda i: "a" in i,["a","b","c","abc"]))
[True, False, False, True]
>>> list(filter(lambda i: "a" in i,["a","b","c","abc"]))
['a', 'abc']
>>> list(filter(lambda i: "ab" in i,["a","b","c","abc"]))
['abc']
>>> list(filter(lambda i: "abe" in i,["a","b","c","abc"]))
[]

Actually I’m expecting select as an jinja equivalent of python filter using (predefined) jinja tests as filter functions.

Question

Am I misusing/misunderstanding the filter or is this a bug in jinja?

Workaround

As a Workaround the list can be joined and the in test can be used:

{{ 'a' in ['abc','a','b'] | join(' ') }}
This seems fine to me as there shouldn't be an entity name containing an whitespace (I'm not sure about this so if someone knows better please correct me!). But mind, I'm only interessted if my entity name pattern is contained. Potential filterning for iterating would be done when iterating over the entities (currently not implemented, see below)!

As @123 pointed out the (HA custom) match test can be used as a replacement.
Othen then in his snipped the list needs to be first and the regex in the select statement.

{{ ['abc','a','b'] | select('match','.*a.*') | list | length > 0 }}

My actual automation

As someone might ask/would like to see:

alias: TRV schedule update
description: Programm schedule to be executed on the TRV it self
trigger:
  - platform: event
    event_type: state_changed
    context:
      user_id:
        - 036d3eeafc754998bc5a0b28848c1489
condition:
  - condition: template
    value_template: "{{ 'climate.trv_' in trigger.event.data.new_state.attributes.entities | join(' ') }}"
  - condition: template
    value_template: "{{ trigger.event.data.new_state.state == 'on' }}"
  - condition: template
    value_template: "{{ trigger.event.data.old_state.state == 'off' }}"
action:
  - service: switch.turn_off
    data:
      entity_id: "{{ trigger.event.data.entity_id }}"
  - alias: Populate variables
    variables:
      attrib: "{{ trigger.event.data.new_state.attributes }}"
      weekdaytranslate: {'mon':'monday','tue':'tuesday','wed':'wednesday','thu':'thursday','fri':'friday','sat':'saturday','sun':'sunday'}
  - variables:
      attrib_times: "{{ attrib.timeslots|map('truncate',5,end='')|list }}"
      attrib_temp: "{{ attrib.actions | map(attribute='data.temperature') | list }}"
  - alias: Publish message for-each entity
    repeat:
      for_each: "{{ attrib.entities }}"
      sequence:
        - alias: Preserve iter
          variables:
            outer: "{{ repeat }}"
        - alias: Publish message for-each day
          repeat:
            for_each: "{{ attrib.weekdays }}"
            sequence:
              - service: mqtt.publish
                data:
                  topic: testzigbee2mqtt/{{ outer.item[8:] | upper }}/set
                  payload: >-
                    { "{{ weekdaytranslate[repeat.item] }}": { {% for i in range(9) -%}
                       "{{ weekdaytranslate[repeat.item] }}_temp_{{i+1}}": "{{ attrib_temp[i] | default(min(attrib_temp)) }}","{{ weekdaytranslate[repeat.item] }}_hour_{{i+1}}": "{{ today_at(attrib_times[i] | default("00:00")).hour }}","{{ weekdaytranslate[repeat.item] }}_minute_{{i+1}}": "{{ today_at(attrib_times[i] | default("00:00")).minute }}"{{ ',' if not loop.last else '' }}
                    {%- endfor %}}}
mode: queued
max: 10

Please keep in mind, this is pretty much work in progress.

Best regards,
Markus

(*) Example is shortend for better readability

The following template assumes the value of the entities attribute is a list of strings.

  - condition: template 
    value_template: "{{ 'climate.trv_' | select('match', trigger.event.data.new_state.attributes.entities) | list | count > 0 }}"

BTW, why didn’t you choose to use a State Trigger?

Match works great. Will update my workaround, as match is much closer to the in test.
Sorry was not aware that match is available as a HA custom test. I’ve heavily worked with the Jinja documentation, and missed all the (to me) well known ansible filters and tests :frowning:
Is there a short Cheatsheet around with HA custom filters/tests, maybe something like jinja has is its documentation?

I’m not using a state trigger as it does not support wildcards as entities - as far as my research told me. I want to avoid static entity lists as they are prone to be forgotten when adding a new device.

The match filter is documented here: Templating

avoid static entry lists … prone to be forgotten when adding a new device

How often do you add new climate entities?


FWIW, using an Event Trigger to listen for state_changed events will make it a very busy
trigger. The inclusion of user_id makes it less busy but it’s still not nearly as focused as a State Trigger can be.

I’m not watching for state_changes on climate entities, I’m watching for state_changes on switch.schedule_* entities. The test for climate.trv_.* is to ensure that only schedules bound to TRV’s get consumed in this automation. So the correct entity type is validated by structure instead of a proper check for its name scheme. Schedule entities are also used for other purposes. And more important, every user will add / remove them as needed for what ever needed.
But devices may come and go without my notice too.

The user_id is currently in there to be able to see traces. Otherwise the 5 stored traces (as per default configuration) are not sufficient for debugging the automation.

If the automation gets to noisy I may need to check if I can customize the schedule entities to call a script instead of using a trigger. There is a possibility to do so, but I haven’t dig into this - yet.

In the Event Trigger you posted above (“My actual automation”), where’s the part that makes it listen exclusively for switch.schedule_* entities?

It is not, the listener listens for all state_changed events, the first condition tests for the entities attribute in the new state of the entity. Assuming that this structure in combination with the pattern for the entities identifies a schedule entity’s state_change (condition 1).

value_template: "{{ trigger.event.data.new_state.>>attributes<<.entities | select('match','climate.trv_.*') | list | length > 0 }}"

As is, each other entity providing an entities attribute within its state object is tested for containing climate.trv_.* and could trigger the automation on state change from off to on (condition 2 & 3).

That’s my original point; it listens for all state_changed events.

That’s why I was confused when you said “I’m watching for state_changes on switch.schedule_* entities”. That Event Trigger is not limited to listening to specific entities, it’s listening for that event produced by all entities.

It’s the automation’s Template Condition that checks if the state_change event was produced by an entity whose entities attribute contains the desired string.

How does the Template Condition behave when it attempts to check an entity that does not have an entities attribute? Does it fail silently or does it report it as an error in the log?

With context.user_id removed I get log messages - a lot of them:

Logger: homeassistant.helpers.template
Source: helpers/template.py:540
First occurred: 22:00:34 (267 occurrences)
Last logged: 22:11:27

Template variable warning: 'homeassistant.util.read_only_dict.ReadOnlyDict object' has no attribute 'entities' when rendering '{{ trigger.event.data.new_state.attributes.entities | select('match','climate.trv_.*') | list | length > 0 }}'

Now I see that’s a problem! Thanks for your patience in pointing this out!

To avoid the error, the template should first check if the entities attribute is defined before attempting to reference it.