Heads up! Upcoming breaking change in the Template integration

Configuration
141

Great !
ill check that out as soon as I get my HA instance back…

ive got this going on:

Invalid config for [automation]: 
  Integration 'sun' does not provide trigger support.

please see here:

what could I do to find that…

would it be an automation like this:

  - alias: Notify sun state change
    id: Notify sun state change
    trigger:
      platform: state
      entity_id: sun.sun
    condition:
      - condition: template
        value_template: >
          {{trigger.to_state.state is in ['above_horizon','below_horizon'] and
            trigger.to_state.state != trigger.from_state.state}}
    action:

which has worked just fine/hasnt caused havoc before?

142

The error you posted has nothing to do with the changes being discussed in this thread.

143

yeah, sorry, would have hoped it to be an erroneous error, pointing to a missing entity_id for sun in a template sensor somehow.
I’ve got no clue where to find elsewhere. if you have a suggestion, please don’t hold back in the other thread I opened for this?

144

I’m using this to monitor the sensors in error, how can I change this? it used to be updated using entity_id: sensor.time and being updated once per minute only, now it gets updated frantically…

  - platform: template
    sensors:
      issues:
        friendly_name: 'Sensors that need attention'
        value_template: >
          {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}
        attribute_templates:
          entities: >
            {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') }}
145

this is a new template problem:

2020-09-19 02:52:48 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template("{{states.sensor.birthday_marijn.attributes.type}}: {{states.sensor.birthday_marijn.attributes.original_date}}
")

which is quite impossible, since the package this sensor is configured in, is no longer in the config, and, along with quite some other packages, moved from my ‘put-aside’ folder on the HA card, to my Mac;s harddrive, so no longer physically there…

how can this still be referenced? Could it somehow be left in the DB, and the new templating engine, even after several ha host reboot s still want to read that??

146

The answer is at the bottom of the first post of this thread. Just add this line to the template:

{% set x = states('sensor.date') %}

In your case, use sensor.time instead of sensor.date.

The template starts with states which represents every entity in your system. That means Home Assistant has assigned a listener to every entity, so whenever any one of them changes state, the template is evaluated.

Minimally, change it to states.sensor so at least only all sensor entities are monitored as opposed to every entity.

2 Likes
147

Here is an attempt at a rework, using an automation with input_text and input_number helpers (you must create those in the UI).

This did reveal an issue because the input_text state might not be able to hold all the information you had in the attribute. To get around this I had to truncate the output.

automation:
  - alias: 'Update issues'
    trigger:
      - platform: time_pattern
        minutes: '*'
    action:
      - service: input_text.set_value
        data:
          entity_id: input_text.issues
          value: "{{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') | truncate(250) }}"
      - service: input_number.set_value
        data:
          entity_id: input_number.issues_count
          value: "{{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | length }}"
148

This is an optimization of your template, it iterates states once instead of twice. I think this is valid even though it references itself now.

  - platform: template
    sensors:
      issues:
        friendly_name: 'Sensors that need attention'
        value_template: >
          {{ (state_attr('sensor.issues', 'entities') or '') | wordcount // 2 }}
        attribute_templates:
          entities: >
            {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') }}
149

I just upgraded to HA 0.115.1 (from 0.114.4) and everything seems fine but I’m not too good with templates and fear that the below will need to be changed in order to keep working. Can someone please advise?

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day:
        entity_id: input_boolean.retic_program1_monday, input_boolean.retic_program1_tuesday, input_boolean.retic_program1_wednesday, input_boolean.retic_program1_thursday, input_boolean.retic_program1_friday, input_boolean.retic_program1_saturday, input_boolean.retic_program1_sunday, sensor.time
        value_template:  >
          {% set sensor_names = [ 'monday', 'tuesday', 'wednesday','thursday','friday','saturday','sunday'] %}
          {% set today_name = sensor_names[now().weekday()] %}
          {% set entity_id = 'input_boolean.retic_program1_'+today_name %}
          {{ is_state(entity_id, 'on') }}

I have a few like these… (I didn’t create them, they were from forum help)

…and also these:

- platform: template
    sensors:
      green_waste_bin_tomorrow:
        entity_id:
          - sensor.time
        friendly_name: "Green Waste Bin Tomorrow"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-09 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_tomorrow.png'}}
  - platform: template
    sensors:
      recycling_bin_today:
        entity_id:
          - sensor.time
        friendly_name: "Recycling Bin Today"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-03 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/recycling_bin_today.png'}}
  - platform: template
    sensors:
      green_waste_bin_today:
        entity_id:
          - sensor.time
        friendly_name: "Green Waste Bin Today"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-10 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_today.png'}}

Help would be awesome. Thanks

EDIT: just looked at my logs and I do indeed have 12 occurrences that need fixing

150

Dave,
To maximise perfortmance I’d write it like this : -

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day:
        value_template:  >
          {% set day =  (as_timestamp(as_local(states.sensor.date.last_changed)) | timestamp_custom ('%A')) | lower%}
          {% set ent_id = 'input_boolean.retic_program1_'~day %}
          {{ is_state(ent_id, 'on') }}

As this will only evaluate once per day. (Edit: AND whenever ‘today’s’ input_boolean changes )

IGNORE THIS : -
if you regularly (or even just occasionally) update your input_boolean’s then it might be wise to include them in the template (maybe @bdraco could advise, sorry to tag but …)

going to : -

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day: # rubbish version
        value_template:  >
          {% set x = input_boolean.retic_program1_monday or input_boolean.retic_program1_tuesday or input_boolean.retic_program1_wednesday or input_boolean.retic_program1_thursday or input_boolean.retic_program1_friday or input_boolean.retic_program1_saturday or input_boolean.retic_program1_sunday %}
          {% set day =  (as_timestamp(as_local(states.sensor.date.last_changed)) | timestamp_custom ('%A')) | lower%}
          {% set ent_id = 'input_boolean.retic_program1_'~day %}
          {{ is_state(ent_id, 'on') }}

The top one correctly determins which sensor to look at on which day, it’s forced to re-evaluate at 00:00 everyday, so just moves to the next sensor
[my god bdraco, that’s a REALLY neat bit of processing]

I’m sure there’s a way to list just the states on those sensors, or expand the group but I hate creating a group just to expand upon it and the group listing is just the same as listing in the template, so unless it’s repeated …

Everyday’s a school day
:rofl:

Edit: try to avoid ‘+’ as a string concatenator, use ‘~’ instead

1 Like
151

For the rest (i.e. your second set), (I’ve just done the first done as a sample) : -

- platform: template
    sensors:
      green_waste_bin_tomorrow:
        friendly_name: "Green Waste Bin Tomorrow"
        value_template: "{{ ((as_timestamp(states.sensor.date.last_changed) - as_timestamp('2019-01-09 00:00:00')) / 86400) | int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_tomorrow.png'}}

I was going to simplify the fixed date timestamp substraction but leaving it makes the template more readable as to what and why. Again this updates only once per day, so ‘optimum’.

EDIT: Important See Post 3 below this one

1 Like
152

This has been a great thread and really helped me understand the change in 0.115 (and also the way it worked before which seems a mystery to me!) – thank you all contributors!

However, I have the opposite of the challenge that it seems we’re trying to overcome. I have a couple of sensors for monitoring stuff, previously I updated them when needed using homeassistant.update_entity.

Following this super-duper change they’re flying off left, right and centre which proves the strength of the change – but its causing havoc with my objective.

Before I start amending my automations, was looking for some insight to see if there’s a way to suppress this behavior and update it when I needed.

Example: this sensor checks for devices on my set up which are offline, got some flaky WiFi switches which drop off and come back every so often – however I like to do a check once a day to see who’s still offline and give them a kick……yes yes, I know I could update the automations associated to this activity but was curious… :blush:

      unavailable_devices:
        friendly_name: Unavailable Devices
        unit_of_measurement: Devices
        icon_template: "{{ 'mdi:checkbox-marked-circle-outline' if states('sensor.unavailable_devices')|int == 0 else 'mdi:alert-circle-outline' }}"
        value_template: >
          {{ states | selectattr('state', 'eq', 'unavailable') | list | length }}
        attribute_templates:
          matched_devices: >
            {%- for item in states -%}
            {%- if item.state == 'unavailable' -%} 
             - {{ item.name }}|
            {%- endif -%}
            {%- endfor -%}
153

yep, expressing the exact same thing. Hope Bdraco and Amelchio will bring back some way of limiting updating templates, the way entity_id did before.

have a challenge myself, which before simply and only was updated by an automation/script (see the commented entity_id)

now, it won’t stop and kill the instance…

  - platform: template
    sensors:
      entities_unavailable:
#        entity_id:
#          - script.update_entities_uun
#          - automation.check_for_unavailable_entities
        friendly_name: Entities Unavailable
        value_template: >
          {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                'light.porch_floodlight','light.parking_light'] if
                                 is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
          {% set unavailable = states|selectattr('state','eq','unavailable')
                                     |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                     |rejectattr('entity_id','in',ignore_list)
                                     |rejectattr('domain','eq','media_player')
                                     |map(attribute='entity_id')
                                     |list %}
          {{unavailable|count}}

#          {% if states('sensor.entities_unavailable')|int > 0 %} mdi:thumb-down
#          {% else %} mdi:thumb-up
#          {% endif %}

      entities_uun:
#        entity_id:
#          - script.update_entities_uun
#          - automation.check_for_unavailable_entities
        friendly_name: Entities U/U/N
        value_template: >
          {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                'light.porch_floodlight','light.parking_light'] if
                                 is_state('binary_sensor.outside_daylight_sensor','on') else [] %}

          {{states|selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |rejectattr('domain','in',['group','media_player'])
                  |map(attribute='entity_id')
                  |list|length}}
        attribute_templates:
          Unknown: >
            {% set unknown = states|selectattr('state','eq','unknown')
                                   |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                   |rejectattr('domain','in',['group'])
                                   |map(attribute='entity_id')
                                   |list %}
            {% if unknown|count == 0 %} 0
            {% else %}
            {{unknown|count}}:
            {{'\n' + unknown|join(',\n')}}
            {% endif %}
          Unknown sensors: >
            {% set unknown = states.sensor|selectattr('state','eq','unknown')
                                          |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                          |map(attribute='entity_id')
                                          |list %}
            {% if unknown|count == 0 %} 0
            {% else %}
            {{unknown|count}}:
            {{'\n' + unknown|join(',\n')}}
            {% endif %}
          Unavailable: >
            {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                  'light.porch__floodlight','light.parking_light'] if
                                   is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
            {% set unavailable = states|selectattr('state','eq','unavailable')
                                       |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                       |rejectattr('entity_id','in',ignore_list)
                                       |rejectattr('domain','eq','media_player')
                                       |map(attribute='entity_id')
                                       |list %}
            {% if unavailable|count == 0 %} 0
            {% else %}
            {{unavailable|count}}:
            {{'\n'}}{{unavailable|join(',\n')}}
            {% endif %}
          None: >
            {% set none_ = states|selectattr('state','eq','none')
                                 |map(attribute='entity_id')
                                 |list %}
            {% if none_|count == 0 %} 0
            {% else %}
            {{none_|count}}:
            {{'\n' + none_|join(',\n')}}
            {% endif %}
          Full: >
            {% set full = states|selectattr('state','in',['unavailable','unknown','none'])
                                |map(attribute='entity_id')
                                |list %}
            {{full|count}}:
            {{'\n' + full|join(',\n')}}
154

Okay, I did some thinking.

Dave is comparing (and I re-phrased) as midnight of a fixed date in the past to the timestamp generated when sensor.date “last_changed” And I’m doing that when LOCAL midnight occurs as that is when the date changes.
Now depending on when the ‘fixed’ date was it could be in DST or not.
Dave currently resides in the UK (until we kick his immigrant ass out again ( :stuck_out_tongue: just Joking about our current 'isolationist' policies)) The point is, Dave is in UTC for 5 months/yr and DST (BST) 7 months/yr.
If it wasn’t in DST then Dave’s offset to midnight last night (which was DST) when divided by 86400 will not yield an integer.
scenairios for different settings mean that for different start dates and different comparison dates you will be sometimes slightly ahead or sometimes slightly behind (never both, for the same fixed date) and sometimes spot on.

So @bdraco how do we compare a fixed date in the past with now() bearing in mind that if you are in TZ +5 then ‘local’ is 5 or 6 hours (depending on DST) out ?
I admit it’s a special circumstance for time comparison

How do we resolve this ?

Doing an addition of a couiple of hours and then | int the / 84600 should do it, but it seems VERY messy - inelegant !

@123 you may have ideas on making this less “kludgy”

155

Any template that starts with states:

{{ states | etc ... }}}

makes Home Assistant assign a listener to every entity in your system. So not just sensor and binary_sensor but other domains as well. My guess is that’s a lot more entities than one normally needs to monitor for the template’s purpose. (EDIT: It means your template is evaluated when just about anything else in your system changes state).

If the need is to monitor only certain domains, then you can specify those domains with the expand() function. This will creates listeners for only sensor and binary_sensor entities:

{{ expand(states.sensor, states.binary_sensor) | etc ... }}

If you know precisely which entities you wish to monitor, create a group containing those entities (if the entities have some sort of commonality, the group can even be created dynamically at startup via an automation). Then your template only needs to expand the group (and listeners will be created exclusively for the group’s members).

{{ expand('group.my_entities') | etc ... }}
156

I had the same problem with my unavailable checks after upgrading and decided to completely restructure how I was handling it. Instead of a template sensor that did all the checks, I moved to a combination of two automations and an MQTT sensor.

The first automation checks to see what devices are in a bad states and publishes on an MQTT topic. Note that the variables are defined in the action section. I found if I did any higher in the script that it would expand and track states which put me right back where I started.:

- id: '1596733904625'
  alias: Unavailable entities sensor update
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /5
  condition:
  - above: '5'
    condition: numeric_state
    entity_id: sensor.hass_uptime
  action:
  - variables:
      entity_ids: |-
        {{ states   
          | selectattr('state','in',['unavailable','unknown','none'])
          | rejectattr('entity_id','in',state_attr('group.ignored_entities', 'entity_id'))
          | rejectattr('entity_id','in',state_attr('group.phone_sensors', 'entity_id'))   
          | rejectattr('domain','eq','group')
          | rejectattr('domain','eq','media_player')
          | map(attribute='entity_id')
          | list
          | join(',')
         }}
      names: |-
        {% set ns = namespace(names=[]) %}
        
        {% for dev in entity_ids.split(',') %}
          {% set ns.names = ns.names + [state_attr(dev, 'friendly_name')] %}
        {% endfor %}
        
        {{ ns.names | to_json }}
      num: '{{ entity_ids.split(",") | length }}'
  - data:    
      payload: |-
        {
          "num": {{ num }},
          "names": {{ names }},
          "entity_ids": {{ entity_ids.split(',') | to_json }}
        }
      topic: hass/unavailable_entities
    service: mqtt.publish

The second is a pretty simple action on when the sensor changes:

- id: '1600444528651'
  alias: Unavailable entities notification
  description: ''
  trigger:
  - platform: state
    entity_id: sensor.unavailable_entities
  condition: []
  action:
  - choose:
    - conditions:
      - condition: template
        value_template: '{{ trigger.to_state.state | int > 0 }}'
      sequence:
      - service: persistent_notification.create
        data:
          message: '{{ state_attr(trigger.to_state.entity_id, "names") | join("\n")
            }}'
          title: Entities Unavailable
          notification_id: entities_unavailable
    - conditions:
      - condition: template
        value_template: '{{ trigger.to_state.state | int == 0  }}'
      sequence:
      - service: persistent_notification.dismiss
        data:
          notification_id: entities_unavailable
    default: []
  mode: single

MQTT sensor for reference:

- platform: mqtt
  name: Unavailable Entities
  state_topic: hass/unavailable_entities
  value_template:  '{{ value_json.num }}'
  json_attributes_topic: hass/unavailable_entities
2 Likes
157

Not sure why people are panicking about their unavailable entities sensors reacting to all states, isn’t that the point of the sensor? To react to all state changes and update a counter if one or more changed to unavailable.

158

For example, instead of using a ‘wildcard’ like states:

          {{states|selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |rejectattr('domain','in',['group','media_player'])
                  |map(attribute='entity_id')
                  |list|length}}

specify the domains of the entities you actually want to monitor for state-changes:

          {{expand(states.sensor, states.binary_sensor, states.light, states.switch, states.lock, states.fan)
                  |selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |map(attribute='entity_id')
                  |list|length}}

The eagle-eyed may have noticed that the second version no longer needs this line:

                  |rejectattr('domain','in',['group','media_player'])

because those domains are not listed in the first line.

EDIT

Observations:

Listeners will be created for every entity produced by the template’s first line. That means listeners will be created for the entities in ignore_list and group.entity_blacklist. In other words, this template will be evaluated even when one of those ignored/blacklisted entities changes state. That includes the sensor containing this template (i.e. self-referential: it is evaluated even when its own state changes).

159

Well, yeah, but not every second on all states on state change. Panic because it halts the system

I had it work ‘on demand’ exactly because it is an expensive template.

We can’t do that any longer.

@123 thanks those are very fine suggestions, of course I posted my ‘old’ sensor not yet optimized , let alone for 115.

Do you have a suggestion for the ‘on demand’ issue?

Maybe rewrite it as python script and use the same automation for calling it .

Now that’s a challenge :wink:

160

My system is running at 5% cpu use, which is exactly what it was at before these changes. This is my unavailable sensor:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        friendly_name: Unavailable Entities
        value_template: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
            |map(attribute='entity_id')|list|length }}"
        attribute_templates:
          entities: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
              |map(attribute='entity_id')|list|join(', ') }}"
        unit_of_measurement: items


group:
  entity_ignore_list:
    entities:
      - THINGS YOU ARE HAPPY TO BE UNAVAILABLE LISTED HERE

Updates on every state change, instant acknowledgment of unavailable items. No issues :man_shrugging:

3 Likes