Heads up! Upcoming breaking change in the Template integration

Configuration
148

This is an optimization of your template, it iterates states once instead of twice. I think this is valid even though it references itself now.

  - platform: template
    sensors:
      issues:
        friendly_name: 'Sensors that need attention'
        value_template: >
          {{ (state_attr('sensor.issues', 'entities') or '') | wordcount // 2 }}
        attribute_templates:
          entities: >
            {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') }}
149

I just upgraded to HA 0.115.1 (from 0.114.4) and everything seems fine but I’m not too good with templates and fear that the below will need to be changed in order to keep working. Can someone please advise?

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day:
        entity_id: input_boolean.retic_program1_monday, input_boolean.retic_program1_tuesday, input_boolean.retic_program1_wednesday, input_boolean.retic_program1_thursday, input_boolean.retic_program1_friday, input_boolean.retic_program1_saturday, input_boolean.retic_program1_sunday, sensor.time
        value_template:  >
          {% set sensor_names = [ 'monday', 'tuesday', 'wednesday','thursday','friday','saturday','sunday'] %}
          {% set today_name = sensor_names[now().weekday()] %}
          {% set entity_id = 'input_boolean.retic_program1_'+today_name %}
          {{ is_state(entity_id, 'on') }}

I have a few like these… (I didn’t create them, they were from forum help)

…and also these:

- platform: template
    sensors:
      green_waste_bin_tomorrow:
        entity_id:
          - sensor.time
        friendly_name: "Green Waste Bin Tomorrow"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-09 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_tomorrow.png'}}
  - platform: template
    sensors:
      recycling_bin_today:
        entity_id:
          - sensor.time
        friendly_name: "Recycling Bin Today"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-03 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/recycling_bin_today.png'}}
  - platform: template
    sensors:
      green_waste_bin_today:
        entity_id:
          - sensor.time
        friendly_name: "Green Waste Bin Today"
        value_template: "{{ ((as_timestamp(now()) - as_timestamp('2019-01-10 00:00:00'))) / 86400 |int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_today.png'}}

Help would be awesome. Thanks

EDIT: just looked at my logs and I do indeed have 12 occurrences that need fixing

150

Dave,
To maximise perfortmance I’d write it like this : -

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day:
        value_template:  >
          {% set day =  (as_timestamp(as_local(states.sensor.date.last_changed)) | timestamp_custom ('%A')) | lower%}
          {% set ent_id = 'input_boolean.retic_program1_'~day %}
          {{ is_state(ent_id, 'on') }}

As this will only evaluate once per day. (Edit: AND whenever ‘today’s’ input_boolean changes )

IGNORE THIS : -
if you regularly (or even just occasionally) update your input_boolean’s then it might be wise to include them in the template (maybe @bdraco could advise, sorry to tag but …)

going to : -

sensor:
  - platform: template    # determine if today is selected as a watering day for program 1
    sensors:
      retic_program1_watering_day: # rubbish version
        value_template:  >
          {% set x = input_boolean.retic_program1_monday or input_boolean.retic_program1_tuesday or input_boolean.retic_program1_wednesday or input_boolean.retic_program1_thursday or input_boolean.retic_program1_friday or input_boolean.retic_program1_saturday or input_boolean.retic_program1_sunday %}
          {% set day =  (as_timestamp(as_local(states.sensor.date.last_changed)) | timestamp_custom ('%A')) | lower%}
          {% set ent_id = 'input_boolean.retic_program1_'~day %}
          {{ is_state(ent_id, 'on') }}

The top one correctly determins which sensor to look at on which day, it’s forced to re-evaluate at 00:00 everyday, so just moves to the next sensor
[my god bdraco, that’s a REALLY neat bit of processing]

I’m sure there’s a way to list just the states on those sensors, or expand the group but I hate creating a group just to expand upon it and the group listing is just the same as listing in the template, so unless it’s repeated …

Everyday’s a school day
:rofl:

Edit: try to avoid ‘+’ as a string concatenator, use ‘~’ instead

1 Like
151

For the rest (i.e. your second set), (I’ve just done the first done as a sample) : -

- platform: template
    sensors:
      green_waste_bin_tomorrow:
        friendly_name: "Green Waste Bin Tomorrow"
        value_template: "{{ ((as_timestamp(states.sensor.date.last_changed) - as_timestamp('2019-01-09 00:00:00')) / 86400) | int % 14 < 1 }}"
        entity_picture_template: >-
          {{ '/local/images/green_waste_bin_tomorrow.png'}}

I was going to simplify the fixed date timestamp substraction but leaving it makes the template more readable as to what and why. Again this updates only once per day, so ‘optimum’.

EDIT: Important See Post 3 below this one

1 Like
152

This has been a great thread and really helped me understand the change in 0.115 (and also the way it worked before which seems a mystery to me!) – thank you all contributors!

However, I have the opposite of the challenge that it seems we’re trying to overcome. I have a couple of sensors for monitoring stuff, previously I updated them when needed using homeassistant.update_entity.

Following this super-duper change they’re flying off left, right and centre which proves the strength of the change – but its causing havoc with my objective.

Before I start amending my automations, was looking for some insight to see if there’s a way to suppress this behavior and update it when I needed.

Example: this sensor checks for devices on my set up which are offline, got some flaky WiFi switches which drop off and come back every so often – however I like to do a check once a day to see who’s still offline and give them a kick……yes yes, I know I could update the automations associated to this activity but was curious… :blush:

      unavailable_devices:
        friendly_name: Unavailable Devices
        unit_of_measurement: Devices
        icon_template: "{{ 'mdi:checkbox-marked-circle-outline' if states('sensor.unavailable_devices')|int == 0 else 'mdi:alert-circle-outline' }}"
        value_template: >
          {{ states | selectattr('state', 'eq', 'unavailable') | list | length }}
        attribute_templates:
          matched_devices: >
            {%- for item in states -%}
            {%- if item.state == 'unavailable' -%} 
             - {{ item.name }}|
            {%- endif -%}
            {%- endfor -%}
153

yep, expressing the exact same thing. Hope Bdraco and Amelchio will bring back some way of limiting updating templates, the way entity_id did before.

have a challenge myself, which before simply and only was updated by an automation/script (see the commented entity_id)

now, it won’t stop and kill the instance…

  - platform: template
    sensors:
      entities_unavailable:
#        entity_id:
#          - script.update_entities_uun
#          - automation.check_for_unavailable_entities
        friendly_name: Entities Unavailable
        value_template: >
          {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                'light.porch_floodlight','light.parking_light'] if
                                 is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
          {% set unavailable = states|selectattr('state','eq','unavailable')
                                     |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                     |rejectattr('entity_id','in',ignore_list)
                                     |rejectattr('domain','eq','media_player')
                                     |map(attribute='entity_id')
                                     |list %}
          {{unavailable|count}}

#          {% if states('sensor.entities_unavailable')|int > 0 %} mdi:thumb-down
#          {% else %} mdi:thumb-up
#          {% endif %}

      entities_uun:
#        entity_id:
#          - script.update_entities_uun
#          - automation.check_for_unavailable_entities
        friendly_name: Entities U/U/N
        value_template: >
          {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                'light.porch_floodlight','light.parking_light'] if
                                 is_state('binary_sensor.outside_daylight_sensor','on') else [] %}

          {{states|selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |rejectattr('domain','in',['group','media_player'])
                  |map(attribute='entity_id')
                  |list|length}}
        attribute_templates:
          Unknown: >
            {% set unknown = states|selectattr('state','eq','unknown')
                                   |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                   |rejectattr('domain','in',['group'])
                                   |map(attribute='entity_id')
                                   |list %}
            {% if unknown|count == 0 %} 0
            {% else %}
            {{unknown|count}}:
            {{'\n' + unknown|join(',\n')}}
            {% endif %}
          Unknown sensors: >
            {% set unknown = states.sensor|selectattr('state','eq','unknown')
                                          |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                          |map(attribute='entity_id')
                                          |list %}
            {% if unknown|count == 0 %} 0
            {% else %}
            {{unknown|count}}:
            {{'\n' + unknown|join(',\n')}}
            {% endif %}
          Unavailable: >
            {% set ignore_list = ['light.driveway_floodlight','light.garden_backyard_floodlight','light.garden_terrace_floodlight',
                                  'light.porch__floodlight','light.parking_light'] if
                                   is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
            {% set unavailable = states|selectattr('state','eq','unavailable')
                                       |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                       |rejectattr('entity_id','in',ignore_list)
                                       |rejectattr('domain','eq','media_player')
                                       |map(attribute='entity_id')
                                       |list %}
            {% if unavailable|count == 0 %} 0
            {% else %}
            {{unavailable|count}}:
            {{'\n'}}{{unavailable|join(',\n')}}
            {% endif %}
          None: >
            {% set none_ = states|selectattr('state','eq','none')
                                 |map(attribute='entity_id')
                                 |list %}
            {% if none_|count == 0 %} 0
            {% else %}
            {{none_|count}}:
            {{'\n' + none_|join(',\n')}}
            {% endif %}
          Full: >
            {% set full = states|selectattr('state','in',['unavailable','unknown','none'])
                                |map(attribute='entity_id')
                                |list %}
            {{full|count}}:
            {{'\n' + full|join(',\n')}}
154

Okay, I did some thinking.

Dave is comparing (and I re-phrased) as midnight of a fixed date in the past to the timestamp generated when sensor.date “last_changed” And I’m doing that when LOCAL midnight occurs as that is when the date changes.
Now depending on when the ‘fixed’ date was it could be in DST or not.
Dave currently resides in the UK (until we kick his immigrant ass out again ( :stuck_out_tongue: just Joking about our current 'isolationist' policies)) The point is, Dave is in UTC for 5 months/yr and DST (BST) 7 months/yr.
If it wasn’t in DST then Dave’s offset to midnight last night (which was DST) when divided by 86400 will not yield an integer.
scenairios for different settings mean that for different start dates and different comparison dates you will be sometimes slightly ahead or sometimes slightly behind (never both, for the same fixed date) and sometimes spot on.

So @bdraco how do we compare a fixed date in the past with now() bearing in mind that if you are in TZ +5 then ‘local’ is 5 or 6 hours (depending on DST) out ?
I admit it’s a special circumstance for time comparison

How do we resolve this ?

Doing an addition of a couiple of hours and then | int the / 84600 should do it, but it seems VERY messy - inelegant !

@123 you may have ideas on making this less “kludgy”

155

Any template that starts with states:

{{ states | etc ... }}}

makes Home Assistant assign a listener to every entity in your system. So not just sensor and binary_sensor but other domains as well. My guess is that’s a lot more entities than one normally needs to monitor for the template’s purpose. (EDIT: It means your template is evaluated when just about anything else in your system changes state).

If the need is to monitor only certain domains, then you can specify those domains with the expand() function. This will creates listeners for only sensor and binary_sensor entities:

{{ expand(states.sensor, states.binary_sensor) | etc ... }}

If you know precisely which entities you wish to monitor, create a group containing those entities (if the entities have some sort of commonality, the group can even be created dynamically at startup via an automation). Then your template only needs to expand the group (and listeners will be created exclusively for the group’s members).

{{ expand('group.my_entities') | etc ... }}
156

I had the same problem with my unavailable checks after upgrading and decided to completely restructure how I was handling it. Instead of a template sensor that did all the checks, I moved to a combination of two automations and an MQTT sensor.

The first automation checks to see what devices are in a bad states and publishes on an MQTT topic. Note that the variables are defined in the action section. I found if I did any higher in the script that it would expand and track states which put me right back where I started.:

- id: '1596733904625'
  alias: Unavailable entities sensor update
  description: ''
  trigger:
  - platform: time_pattern
    minutes: /5
  condition:
  - above: '5'
    condition: numeric_state
    entity_id: sensor.hass_uptime
  action:
  - variables:
      entity_ids: |-
        {{ states   
          | selectattr('state','in',['unavailable','unknown','none'])
          | rejectattr('entity_id','in',state_attr('group.ignored_entities', 'entity_id'))
          | rejectattr('entity_id','in',state_attr('group.phone_sensors', 'entity_id'))   
          | rejectattr('domain','eq','group')
          | rejectattr('domain','eq','media_player')
          | map(attribute='entity_id')
          | list
          | join(',')
         }}
      names: |-
        {% set ns = namespace(names=[]) %}
        
        {% for dev in entity_ids.split(',') %}
          {% set ns.names = ns.names + [state_attr(dev, 'friendly_name')] %}
        {% endfor %}
        
        {{ ns.names | to_json }}
      num: '{{ entity_ids.split(",") | length }}'
  - data:    
      payload: |-
        {
          "num": {{ num }},
          "names": {{ names }},
          "entity_ids": {{ entity_ids.split(',') | to_json }}
        }
      topic: hass/unavailable_entities
    service: mqtt.publish

The second is a pretty simple action on when the sensor changes:

- id: '1600444528651'
  alias: Unavailable entities notification
  description: ''
  trigger:
  - platform: state
    entity_id: sensor.unavailable_entities
  condition: []
  action:
  - choose:
    - conditions:
      - condition: template
        value_template: '{{ trigger.to_state.state | int > 0 }}'
      sequence:
      - service: persistent_notification.create
        data:
          message: '{{ state_attr(trigger.to_state.entity_id, "names") | join("\n")
            }}'
          title: Entities Unavailable
          notification_id: entities_unavailable
    - conditions:
      - condition: template
        value_template: '{{ trigger.to_state.state | int == 0  }}'
      sequence:
      - service: persistent_notification.dismiss
        data:
          notification_id: entities_unavailable
    default: []
  mode: single

MQTT sensor for reference:

- platform: mqtt
  name: Unavailable Entities
  state_topic: hass/unavailable_entities
  value_template:  '{{ value_json.num }}'
  json_attributes_topic: hass/unavailable_entities
2 Likes
157

Not sure why people are panicking about their unavailable entities sensors reacting to all states, isn’t that the point of the sensor? To react to all state changes and update a counter if one or more changed to unavailable.

158

For example, instead of using a ‘wildcard’ like states:

          {{states|selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |rejectattr('domain','in',['group','media_player'])
                  |map(attribute='entity_id')
                  |list|length}}

specify the domains of the entities you actually want to monitor for state-changes:

          {{expand(states.sensor, states.binary_sensor, states.light, states.switch, states.lock, states.fan)
                  |selectattr('state','in',['unavailable','unknown','none'])
                  |rejectattr('entity_id','in',ignore_list)
                  |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                  |map(attribute='entity_id')
                  |list|length}}

The eagle-eyed may have noticed that the second version no longer needs this line:

                  |rejectattr('domain','in',['group','media_player'])

because those domains are not listed in the first line.

EDIT

Observations:

Listeners will be created for every entity produced by the template’s first line. That means listeners will be created for the entities in ignore_list and group.entity_blacklist. In other words, this template will be evaluated even when one of those ignored/blacklisted entities changes state. That includes the sensor containing this template (i.e. self-referential: it is evaluated even when its own state changes).

159

Well, yeah, but not every second on all states on state change. Panic because it halts the system

I had it work ‘on demand’ exactly because it is an expensive template.

We can’t do that any longer.

@123 thanks those are very fine suggestions, of course I posted my ‘old’ sensor not yet optimized , let alone for 115.

Do you have a suggestion for the ‘on demand’ issue?

Maybe rewrite it as python script and use the same automation for calling it .

Now that’s a challenge :wink:

160

My system is running at 5% cpu use, which is exactly what it was at before these changes. This is my unavailable sensor:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        friendly_name: Unavailable Entities
        value_template: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
            |map(attribute='entity_id')|list|length }}"
        attribute_templates:
          entities: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
              |map(attribute='entity_id')|list|join(', ') }}"
        unit_of_measurement: items


group:
  entity_ignore_list:
    entities:
      - THINGS YOU ARE HAPPY TO BE UNAVAILABLE LISTED HERE

Updates on every state change, instant acknowledgment of unavailable items. No issues :man_shrugging:

3 Likes
161

Thanks Marc, will check after the time trial (TdF) :wink:

Response time might differ depending on the count of states of course …

1 Like
162

I believe Mariusthvdb summarized the issue nicely when he described it as being “expensive”.

Screenshot from 2020-09-19 11-22-13
Screenshot from 2020-09-19 11-22-131065×191 21.4 KB

As shown in the screenshot, the use of states means it now listens to everything (entities in all domains; automation, scene, script, zone, etc).

That’s different from how it worked prior to 0.115 (which created no listeners for states). Therefore for the people who are experiencing performance issues, they may wish to start their debugging adventure by removing any “expensive” Template Sensors they may have and seeing if it improves anything.

Whether it proves to be ‘expensive’ or not, from a design perspective it seems overreaching to have a Template Sensor refreshed when anything else changes state.

163

If you want to limit updates to only a specific interval, something like this will work:

- id: '1600355905751'
  alias: Update Unavailable Entities
  description: ''
  trigger:
  - platform: time_pattern
    hours: '*'
    minutes: '15'
    seconds: '0'
  condition: []
  action:
  - service: input_text.set_value
    data:
       value: "{{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}"
    entity_id: input_text.unavailable_entities
  mode: single
164

But that is kinda the point of the sensor in this case is it not? Monitor all state changes and record the ones that change to unavailable.

165 
{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
            |list|length }}

Taking out the extra map will make this a bit faster

1 Like
166

I agree its purpose is to report when something is unavailable. It’s the “something” that I feel ought to have a narrower scope as opposed to its current scope which is “everything”.

For example, as a consequence of using states, the Template Sensor checks if any automation, zone, script, scene, etc becomes unavailable. Is that even a remote possibility? Nevertheless, listeners are created for all automations (and zones, scripts, etc) and the Template Sensor gets updated when any one of those ‘extraneous’ entities changes state. Seems like a very wide (global) net is being cast to catch just a few (local) fish.

167

the way I used it, was have a look a startup, see what got initiated and what not. was this expected? no? act. If expected, add the to the ignore list.

update the sensor. manually by clicking the update script.
all well? thumbs up. if not, have another look.

That did the trick to see for config errors, or integrations not being initiated at startup, mostly the weather integration.
For me, I don’t need the constant check of all states. Just a simple tool to reveal the issues in the current state of affairs.

update

even testing the above templates in the template field causes issues: my Hue integration has always been an indicator of internal issues, and becomes unavailable completely and remains like that, if I enter the template.
even after emptying the template editor, it wont come back.
so yeah, it is very expensive, and I am not yet very happy with the new way of handling these templates at all.
entering entity_id: was not so bad, and had the advantage of controlling the update process… now, the system keeps checking states, even if unnecessary and even worse, not desired.

at the moment. y system has more than 278 unavailable entities, so the list wouldn’t be anywhere near usefully listed using Nicks automation

seems any effort to control the evaluation of these templates (using an automation with an input_text, or maybe a python script) is futile if a single run of it already has such impact.

Cant help but feeling there is a bug somewhere, not giving back to the system, or staying in a loop somehow. I had a very swiftly responding system up to HA 114.4, even with all my many entities, without so much as a single Hue unavailability left.
This must be the muddiest iteration yet.