Heads up! Upcoming breaking change in the Template integration

well yeah, I am writing a few automations to get the next_alarm sensor up to speed with the new entity_id issue… so Im am conforming. no worries.

the counter template sensor above was not mine tbh, (I use a python script to do the works) but was posted because quite a few members here use that. If so, it is to be expected that they all see a rise in processor spikes.
What I am trying to say, is that I am not really speaking for myself, but for the community as a whole. Seeing the demise of a perfectly fine technique for limiting the updating of a sensor, and a global one at that (you must sympathize with that) feels just so unnecessary.

To be even more precise in describing my worries: it is not that I see these sensors break because they can’t find an entity_id any longer.
It is more of the opposite: I fear they will keep firing because they are constantly checking the state they are finding…

the few ones that need sensor.time are easily spotted and rewritten. It’s a more fundamental worry than that…

in fact it could be compared to the move custom button-card made some time ago. From updating on all state changes, to entering the triggers_update: configuration option. Don’t use that, and have the system update on all state changes mentioned. Do use it and have update only on that particular (set of) entity_id(s).

Best of both worlds. Would be great to have I core templates too…

We might be able put together an optimization for {{states|count}} , I’ll take a look later today.

I’ve helped a few people convert their templates, and by far the most common cases don’t actually need to iterate all states which is the better solution.

I’ve seen a lot of these

states | selectattr('entity_id', 'in', state_attr('group.family','entity_id'))

which can be replaced with:

expand('group.family')

Also for those who are struggling to find why an update happens, we have additional logging options coming to help track these down: https://github.com/home-assistant/core/pull/40180

Processing templates with states in them is already significantly faster in 0.115 after https://github.com/home-assistant/core/pull/39731

We have a change coming in 0.116 that will make iterating states twice as fast when you have a large number of states in the system https://github.com/home-assistant/core/pull/40012

I’ll see if we can make iterating states faster yet.

4 Likes

to understand fully: does this in fact change the burden on the backend, or is it simply simpler in the yaml config.
iow, does it limit the states checked for the evaluation of the template value?

I’m not 100% sure what you are asking so I’ll try to respond and hope I’ve hit your question.

The template developer tools will show you which state changes the template will listen for now:

3 Likes

thats a nice tool!

it also shows that for a mere counting of the states with:

        value_template: >
          {{states|count}}
        attribute_templates:
          headline: >
            {{states|count}} entities in {{states|groupby('domain')|count}} domains
          alert: >
            {{states.alert|count}}
          automation: >
            {{states.automation|count}}
          binary_sensor: >
            {{states.binary_sensor|count}}
          camera: >
            {{states.camera|count}}
          climate: >
            {{states.climate|count}}
          counter: >
            {{states.counter|count}}

it listens to all state changes. Which isn’t necessary at all…
which is where my worry kicks in: deprecating the option of the entity_id: sensor.time is very expensive.

zoomin in on a single counter:

{{states.sensor|count}}

gives

not exactly sure what that means… in terms of processor needs?

I recall in this post, where I was trying to understand how the improvements will work with expand, you stated:

Creates listeners for each entity in the group

{{ expand('group.my_lights')
  | selectattr('state', 'eq', 'on') | etc }}

Creates listeners for all light entities and group.lights

{{ states.lights
  | selectattr('entity_id', 'in', state_attr('group.lights', 'entity_id')) | etc }}

In 0.115 it appears to be slightly more inclusive than that.

In the first example, a listener is also created for group.my_lights.
In the second example, listeners are also created for all light entities.

My only purpose in mentioning this is to clarify its behavior, especially for people like me who still remember how its behavior was originally described. :wink:


To exercise the new handling of expand, I carried out a simple experiment consisting of nested groups:

  lights_downstairs:
    - light.family_mantle
    - light.family_ceiling
    - light.kitchen_ceiling

  lights_upstairs:
    - light.hallway_ceiling
    
  lights:
    - group.lights_downstairs
    - group.lights_upstairs

The new technique is now clever enough to identify the four light entities in group.lights.

An added bonus is that I can add/subtract a light from a group, execute Reload Groups, and the listeners are automatically revised. This fulfills my old Feature Request:

3 Likes

Nice test. Thanks for sharing.

I was able to improve the performance of counting states here https://github.com/home-assistant/core/pull/40250

4 Likes

@bdraco

Sweet. If this particular entity counting template is popular, we might even provide an integration for that rather than have everyone build it with templates.

@Mariusthvdb

Please note that I am giving these workarounds as a way for you to get things back to working now. I am not necessarily saying that this is the preferred way for the long term.

However, I don’t think this has been rushed. At one point we have to get it out there and collect feedback, we cannot wait for it to be perfect because then it never will be. I do understand where you are coming from. It is a big change that takes some getting used to and at first it can appear strange. And of course, fixing up a big configuration is always a drag.

You mention fear and worry. We should not let that drive development, that is called premature optimization. I think it is better to identify real issues and then we will think of solutions to those.

2 Likes

cool. heavily into checking the config, which is not so very nasty to do, given the new possibilities galore ! font get me wrong, I am very excited :wink:

please let me ask one specific config question:

        value_template: >
          {% set ns = namespace(total = 0) %}
          {% for state in states.sensor if '_actueel' in state.object_id %}
            {% set ns.total = ns.total + (state.state|int) %}
          {% endfor %}
          {{ns.total}}

can be written using the expand(group) function now to? there’s a group.energy_actueel which could be used… adding the values of all these sensors ?

You can rewrite it as

{{ expand('group.energy_actueel') | map(attribute='state') | map('int') | sum }}

1 Like

wow, Thanks!
that’s exactly what I was testing right now:

      sensors_actueel_verbruik_summed:
        friendly_name: Sensors actueel verbruik summed
#        entity_id: sensor.netto_verbruik
        unit_of_measurement: Watt
        value_template: >
          {{expand('group.sensors_actueel')
             |map(attribute='state')|map('int')|sum}}

I’ve made an automation to create the group:

  - alias: Create actueel group
    trigger:
      platform: homeassistant
      event: start
    action:
      service: group.set
      data_template:
        object_id: sensors_actueel
        entities: >-
          {%- for s in states.sensor
            if ('_actueel' in s.entity_id )%}
            {{s.entity_id}}{% if not loop.last %}, {% endif %}
          {%- endfor %}

beats writing the group verbosely.
Now only need to think of a way to get this restored. Or will the sensor be restored, even though the group is created at startup… ?

The sensor will update automatically as soon as the group is created because the group creation fires a state_changed event.

I’ve got a branch that will significantly reduce the number of re-renders when we are only counting states and not actually looking at what is in the state that builds on https://github.com/home-assistant/core/pull/40250

Edit: I’ve opened a draft PR for this https://github.com/home-assistant/core/pull/40272

2 Likes

Great !
ill check that out as soon as I get my HA instance back…

ive got this going on:

Invalid config for [automation]: 
  Integration 'sun' does not provide trigger support. 

please see here:

what could I do to find that…

would it be an automation like this:

  - alias: Notify sun state change
    id: Notify sun state change
    trigger:
      platform: state
      entity_id: sun.sun
    condition:
      - condition: template
        value_template: >
          {{trigger.to_state.state is in ['above_horizon','below_horizon'] and
            trigger.to_state.state != trigger.from_state.state}}
    action:

which has worked just fine/hasnt caused havoc before?

The error you posted has nothing to do with the changes being discussed in this thread.

yeah, sorry, would have hoped it to be an erroneous error, pointing to a missing entity_id for sun in a template sensor somehow.
I’ve got no clue where to find elsewhere. if you have a suggestion, please don’t hold back in the other thread I opened for this?

I’m using this to monitor the sensors in error, how can I change this? it used to be updated using entity_id: sensor.time and being updated once per minute only, now it gets updated frantically…

  - platform: template
    sensors:
      issues:
        friendly_name: 'Sensors that need attention'
        value_template: >
          {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}
        attribute_templates:
          entities: >
            {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | map(attribute='entity_id') | list | join('\n') }}

this is a new template problem:

2020-09-19 02:52:48 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template("{{states.sensor.birthday_marijn.attributes.type}}: {{states.sensor.birthday_marijn.attributes.original_date}}
")

which is quite impossible, since the package this sensor is configured in, is no longer in the config, and, along with quite some other packages, moved from my ‘put-aside’ folder on the HA card, to my Mac;s harddrive, so no longer physically there…

how can this still be referenced? Could it somehow be left in the DB, and the new templating engine, even after several ha host reboot s still want to read that??

The answer is at the bottom of the first post of this thread. Just add this line to the template:

{% set x = states('sensor.date') %} 

In your case, use sensor.time instead of sensor.date.

The template starts with states which represents every entity in your system. That means Home Assistant has assigned a listener to every entity, so whenever any one of them changes state, the template is evaluated.

Minimally, change it to states.sensor so at least only all sensor entities are monitored as opposed to every entity.

2 Likes