Heads up! Upcoming breaking change in the Template integration

would this also go for a template like:

      entities_domains_counter:
#        entity_id:
#          - sensor.date
#          - sensor.count_persistent_notifications
        friendly_name: Entities per domain
        value_template: >
          {{states|count}}
        attribute_templates:
          headline: >
            {{states|count}} entities in {{states|groupby('domain')|count}} domains
          alert: >
            {{states.alert|count}}
          automation: >
            {{states.automation|count}}
          binary_sensor: >
            {{states.binary_sensor|count}}
          camera: >
            {{states.camera|count}}
          climate: >
            {{states.climate|count}}
          counter: >
            {{states.counter|count}}
# continue to list all domains

which, by adding entity_id: sensor.day was only re-evaluated once a day. Doesnt this trigger now on each and every state change?

Fair enough, I just wanted to make sure I wasnā€™t going about it in an overly complex way :slight_smile: Thank you!

You are right that {{states|count}} will trigger on each state change. For this simple example that is obviously redundant but for a slightly more complex example:

{{ states | map(attribute="last_updated") | max }}

we can see that each state change indeed must be considered when state is accessed. Incidentally, it is jaw-dropping to put this example into the template developer tool and see it update in real-time!

If this is actually a performance issue, it might be possible to make a special optimization for {{states|count}} where only life-cycle events are considered. I am not quite sure whether that is possible (or even desired), @bdraco could probably answer that. A workaround is to put the template into an input_number and update it manually as desired.

If you want to drive updates manually, automations are able to do that with much more flexibility than the templates were ever able to do. For example, your hour_icon value template seems to only need changes every hour and you can do that with a time_pattern which would make it 60x more efficient!

Please note, I am not saying that the current implementation is perfect. We will probably have to tweak it for a few releases. But the concerns I have seen are about things that can still be done so I think this will turn out fine once we get comfortable with the new ways

HI Amelchio,

thanks for replying. I admire the hugeness of the 115 update. And thank everyone for that. Its just that the deprecation of this entity_id in templates seems to have been rushed in, and not all consequences are thought out carefully enough.
I am not sure what youā€™re saying here is acknowledging the issues introduced by deprecating entity_id:

I do understand for a majority of template sensors, this is very nice to see happening, dynamic and constant updating of relevant entity_idā€™s even when not explicitly mentioned in the template.

However, suggesting even more convoluted solutions like creating timed automations for updating input_numbers or other domains which hold the current state of those templates is quite baffling tbh.

Especially since we had a perfect trigger/limiter for those updates!

Wouldnā€™t it be a viable scenario that the new way of evaluating the templates simply co-exist with the functionality of entity_id:

That way, we would have it all:

  • Automatic and dynamic template evaluation for entities in the templates
  • option for a global entity to be valid for the complete template sensor that limits updating to changes of that global entity (or entities in a list)

I donā€™t want to update manually, I simply donā€™t want to update on all state changes, which, now they are found automatically, drastically increase in numberā€¦

(btw my hour icon needs updating every minute, hence the sensor.time)

1 Like

In fairness to the changes, all of my concerns were because I didnā€™t realise how far reaching the changes went. Iā€™ve removed entity_id from all of mine and patiently waited to see which broke and not a single one did :slightly_smiling_face:

Itā€™s no secret that your configuration is particularly complicated, which of course is your perogative, but I suspect if you went through all of your templates you can probably define a lot of them in different ways that work more effectively and wouldnā€™t require any workarounds, then the few that are left may just need the ā€˜cheatā€™ defined above.

The downside for you, with which I sympathise, is that because your setup is complicated, going through it to achieve this is probably a lot of work.

2 Likes

The change is awesome tbh. One of the most annoying support issues (as you know) was explaining the updates on template sensors

well yeah, I am writing a few automations to get the next_alarm sensor up to speed with the new entity_id issueā€¦ so Im am conforming. no worries.

the counter template sensor above was not mine tbh, (I use a python script to do the works) but was posted because quite a few members here use that. If so, it is to be expected that they all see a rise in processor spikes.
What I am trying to say, is that I am not really speaking for myself, but for the community as a whole. Seeing the demise of a perfectly fine technique for limiting the updating of a sensor, and a global one at that (you must sympathize with that) feels just so unnecessary.

To be even more precise in describing my worries: it is not that I see these sensors break because they canā€™t find an entity_id any longer.
It is more of the opposite: I fear they will keep firing because they are constantly checking the state they are findingā€¦

the few ones that need sensor.time are easily spotted and rewritten. Itā€™s a more fundamental worry than thatā€¦

in fact it could be compared to the move custom button-card made some time ago. From updating on all state changes, to entering the triggers_update: configuration option. Donā€™t use that, and have the system update on all state changes mentioned. Do use it and have update only on that particular (set of) entity_id(s).

Best of both worlds. Would be great to have I core templates tooā€¦

We might be able put together an optimization for {{states|count}} , Iā€™ll take a look later today.

Iā€™ve helped a few people convert their templates, and by far the most common cases donā€™t actually need to iterate all states which is the better solution.

Iā€™ve seen a lot of these

states | selectattr('entity_id', 'in', state_attr('group.family','entity_id'))

which can be replaced with:

expand('group.family')

Also for those who are struggling to find why an update happens, we have additional logging options coming to help track these down: https://github.com/home-assistant/core/pull/40180

Processing templates with states in them is already significantly faster in 0.115 after https://github.com/home-assistant/core/pull/39731

We have a change coming in 0.116 that will make iterating states twice as fast when you have a large number of states in the system https://github.com/home-assistant/core/pull/40012

Iā€™ll see if we can make iterating states faster yet.

4 Likes

to understand fully: does this in fact change the burden on the backend, or is it simply simpler in the yaml config.
iow, does it limit the states checked for the evaluation of the template value?

Iā€™m not 100% sure what you are asking so Iā€™ll try to respond and hope Iā€™ve hit your question.

The template developer tools will show you which state changes the template will listen for now:

3 Likes

thats a nice tool!

it also shows that for a mere counting of the states with:

        value_template: >
          {{states|count}}
        attribute_templates:
          headline: >
            {{states|count}} entities in {{states|groupby('domain')|count}} domains
          alert: >
            {{states.alert|count}}
          automation: >
            {{states.automation|count}}
          binary_sensor: >
            {{states.binary_sensor|count}}
          camera: >
            {{states.camera|count}}
          climate: >
            {{states.climate|count}}
          counter: >
            {{states.counter|count}}

it listens to all state changes. Which isnā€™t necessary at allā€¦
which is where my worry kicks in: deprecating the option of the entity_id: sensor.time is very expensive.

zoomin in on a single counter:

{{states.sensor|count}}

gives

not exactly sure what that meansā€¦ in terms of processor needs?

I recall in this post, where I was trying to understand how the improvements will work with expand, you stated:

Creates listeners for each entity in the group

{{ expand('group.my_lights')
  | selectattr('state', 'eq', 'on') | etc }}

Creates listeners for all light entities and group.lights

{{ states.lights
  | selectattr('entity_id', 'in', state_attr('group.lights', 'entity_id')) | etc }}

In 0.115 it appears to be slightly more inclusive than that.

In the first example, a listener is also created for group.my_lights.
In the second example, listeners are also created for all light entities.

My only purpose in mentioning this is to clarify its behavior, especially for people like me who still remember how its behavior was originally described. :wink:


To exercise the new handling of expand, I carried out a simple experiment consisting of nested groups:

  lights_downstairs:
    - light.family_mantle
    - light.family_ceiling
    - light.kitchen_ceiling

  lights_upstairs:
    - light.hallway_ceiling
    
  lights:
    - group.lights_downstairs
    - group.lights_upstairs

The new technique is now clever enough to identify the four light entities in group.lights.

An added bonus is that I can add/subtract a light from a group, execute Reload Groups, and the listeners are automatically revised. This fulfills my old Feature Request:

3 Likes

Nice test. Thanks for sharing.

I was able to improve the performance of counting states here https://github.com/home-assistant/core/pull/40250

4 Likes

@bdraco

Sweet. If this particular entity counting template is popular, we might even provide an integration for that rather than have everyone build it with templates.

@Mariusthvdb

Please note that I am giving these workarounds as a way for you to get things back to working now. I am not necessarily saying that this is the preferred way for the long term.

However, I donā€™t think this has been rushed. At one point we have to get it out there and collect feedback, we cannot wait for it to be perfect because then it never will be. I do understand where you are coming from. It is a big change that takes some getting used to and at first it can appear strange. And of course, fixing up a big configuration is always a drag.

You mention fear and worry. We should not let that drive development, that is called premature optimization. I think it is better to identify real issues and then we will think of solutions to those.

2 Likes

cool. heavily into checking the config, which is not so very nasty to do, given the new possibilities galore ! font get me wrong, I am very excited :wink:

please let me ask one specific config question:

        value_template: >
          {% set ns = namespace(total = 0) %}
          {% for state in states.sensor if '_actueel' in state.object_id %}
            {% set ns.total = ns.total + (state.state|int) %}
          {% endfor %}
          {{ns.total}}

can be written using the expand(group) function now to? thereā€™s a group.energy_actueel which could be usedā€¦ adding the values of all these sensors ?

You can rewrite it as

{{ expand('group.energy_actueel') | map(attribute='state') | map('int') | sum }}

1 Like

wow, Thanks!
thatā€™s exactly what I was testing right now:

      sensors_actueel_verbruik_summed:
        friendly_name: Sensors actueel verbruik summed
#        entity_id: sensor.netto_verbruik
        unit_of_measurement: Watt
        value_template: >
          {{expand('group.sensors_actueel')
             |map(attribute='state')|map('int')|sum}}

Iā€™ve made an automation to create the group:

  - alias: Create actueel group
    trigger:
      platform: homeassistant
      event: start
    action:
      service: group.set
      data_template:
        object_id: sensors_actueel
        entities: >-
          {%- for s in states.sensor
            if ('_actueel' in s.entity_id )%}
            {{s.entity_id}}{% if not loop.last %}, {% endif %}
          {%- endfor %}

beats writing the group verbosely.
Now only need to think of a way to get this restored. Or will the sensor be restored, even though the group is created at startupā€¦ ?

The sensor will update automatically as soon as the group is created because the group creation fires a state_changed event.

Iā€™ve got a branch that will significantly reduce the number of re-renders when we are only counting states and not actually looking at what is in the state that builds on https://github.com/home-assistant/core/pull/40250

Edit: Iā€™ve opened a draft PR for this https://github.com/home-assistant/core/pull/40272

2 Likes