Heads up! Upcoming breaking change in the Template integration

[123 (Taras)]

@amelchio

reaffirmed me that removing entity_id: is right. Even a power user like that gets it wrong sometimes and it is just bad to have such a blunt tool around.
…

So I think we could add something like auto_update: false to the template sensor for the rare cases where manual control is desired.

To summarize:

• Glad we removed entity_id.
• Think we need to replace it.

Sounds like politics:

My predecessor’s plan was flawed and I have cancelled it. I am replacing it with a new plan (proceeds to outline a nearly equivalent plan).

---

The decision to remove entity_id didn’t foresee one specific consequence, namely losing the ability to easily control/constrain the template’s evaluation (i.e. what refreshes it and how often). I’m pleased to see that this loss is now recognized and the discussion has turned to mitigation.

scan_interval

On the surface, it sounds like a more flexible replacement for entity_id because you have more choices of polling frequency. However, it lacks the ability of sensor.date to trigger at the beginning of every new day. That’s an important ability for many common date/time calculations that must reset daily. For that you will now be obliged to use an automation (thereby increasing the complexity of what was once possible by simply adding entity_id: sensor.date).

Question: if I specify a scan_interval will that prevent the assignment of listeners to the template?
That, of course, is the desired behavior because it’s how entity_id works.

automation

Using an automation with a Time Pattern Trigger (to periodically update a Template Sensor) was already a technique used in all versions prior to 0.115. It was typically suggested to users who needed a very specific updating frequency (i.e. not once a minute or once a day). Based on my impression, most users rarely needed to use this technique because most Template Sensors worked satisfactorily with a one-minute or once-a-day update interval (using sensor.time and sensor.date, respectively).

auto-update

In the past, simply adding entity_id was an implicit auto_update: false. In addition, entity_id was more compact because it also indicated the entity (or entities) that would serve to update the template. It appears it is now suggested to use two options to perform what was once achievable with a single option.

I still believe entity_id was a simpler way to get the job done. However, whatever combination of new options (scan_interval, auto_update, whatever) are used to replace entity_id would be appreciated because the current situation complicates certain Template Sensors.

[Mariusthvdb (Marius)]

@bdraco

Its probably because I have https://github.com/home-assistant/core/pull/40272 merged in and its fixed there.

hmm, it doesnt fail here either… and I most certainly dont have that fix merged yet.

[123 (Taras)]

Ditto; works with 0.115.0 in the Template Editor.

[klogg]

Another day another new thing.

@Mariusthvdb

    action:
      service: group.set

Are you saying there is a service that will dynamically create groups?
I had no idea. Although I don’t use many groups.

Before I posted this I tried to find it documented somewhere but I couldn’t so I just want to make sure I have this right?

---

Whilst this is barely on-topic it does relate directly to a solution offered to the very much on-topic topic.

@123

Assuming you create group.irrigation_zones containing all the sensors you had listed in entity_id then I believe this (untested) template should work.

I need to create a group of zones and some users of my package use more than the ‘standard’ now hard coded eight zones. this will alleviate a step in their customisation of my project.

[Mariusthvdb (Marius)]

@klogg

Before I posted this I tried to find it documented somewhere but I couldn’t so I just want to make sure I have this right?

yes, its a very nice tool to have in the box. check

especially nice to use now with the expand() functionality

it has been since forever, way before Lovelace

    action:
      service: group.set
      data_template:
        object_id: battery_sensors
        entities: >-
          {%- for s in states.sensor
            if ('battery_level' in s.entity_id or
                'motion_sensor_battery' in s.entity_id)%}
            {{s.entity_id}}{% if not loop.last %}, {% endif %}
          {%- endfor %}

or a bit more fun:

    action:
      service_template: >
        script.group_battery_sensors_low_{{'remove' if
          states('sensor.low_level_batteries')|int == 0 else 'create'}}

with

script:
  group_battery_sensors_low_create:
    alias: Group battery sensors low create
    sequence:
      service: group.set
      data_template:
        object_id: battery_sensors_low
        entities: >-
          {%- for s in states.sensor
            if ('battery_level' in s.entity_id or
                'motion_sensor_battery' in s.entity_id)
             and s.state|int < states('input_number.battery_alert_level')|int %}
            {{s.entity_id}}{% if not loop.last %}, {% endif %}
          {%- endfor %}

  group_battery_sensors_low_remove:
    sequence:
      service: group.remove
      data:
        object_id: battery_sensors_low

[123 (Taras)]

@klogg

tried to find it documented somewhere but I couldn’t

It’s definitely missing from its documentation.

It’s described in its services.yaml file but that’s not where I expect most users learn about an integration’s services.

You can use an automation to dynamically define a group at startup. Alternatively, vote for this WTH which requests the ability to template groups. It amounts to the same thing but with less coding.

Why the heck don't we have templates/glob for groups?

What if there were groups that you could template? It would convert a comma spaced output (or list) to a list. For example, a group that shows all people that are at home group: name: persons_at_home template_group: > {{ states.person | selectattr('state', 'eq', 'home') | join(', ', attribute="entity_id") }} or a group that contains all entities that have battery_level in their name group: name: battery_sensors template_group: > {{ states.sensor | selectattr('entity_id', 'co…

NOTE

I imagine there’s a point where if everything can be templated you may create a circular dependency. That results in problems at startup because it’s faced with a ‘chicken and egg’ situation.

[amelchio]

@123

To summarize:

• Glad we removed entity_id .
• Think we need to replace it.

I don’t think we need to replace entity_id:. I do think this issue is much smaller than it is made out to be in this thread. It seems like almost every concrete example turns out to not be an issue after all. It will be interesting to see where @Mariusthvdb ends after the rewrite of all the templates and with the code optimizations from bdraco.

However, though the examples are few, I think we might want a static template renderer. Since input_text has some shortcomings, reusing the template sensor is a simple way. I actually don’t like my own idea much, the very definition of a sensor is something that detects changes. But it is a manageable way to restore all the features we used to have and I think that has some merit.

@123

I still believe entity_id was a simpler way to get the job done.

I believe it was a simple way to shoot yourself in the foot

[123 (Taras)]

@amelchio

seems like almost every concrete example turns out to not be an issue after all

Possibly because you have not addressed the scenario I described nor answered my questions. You are certainly free to ignore all that but it doesn’t change the reality that the loss of entit_id cannot always be easily, or conveniently, mitigated.

@amelchio

interesting to see where @Mariusthvdb ends after the rewrite of all the templates

As I was saying, “cannot always be easily, or conveniently, mitigated”.

@amelchio

I don’t think we need to replace entity_id:

Not sure what to make of that statement given the suggestions to add options that effectively replace the functionality of entity_id (scan_interval and auto_update).

@amelchio

simple way to shoot yourself in the foot

Arguably it’s even easier to do that now, given the additional complexity of having to re-write a complex Template Sensor as an automation or python_script.

[Mariusthvdb (Marius)]

@123

Alternatively, vote for this WTH which requests the ability to template groups . It amounts to the same thing but with less coding.

exactly. long before this with, this was discussed and paused, because the demise of the all-groups at that time was still imminent…
Ive voted for the WHT, but just now gave your final summarization some extra love. Hope we would get this.
Having a template group would be even better than the startup automation. so: +1 to that.

[Mariusthvdb (Marius)]

@amelchio

I don’t think we need to replace entity_id:

well, personally I think this thread has indications enough to make you reconsider that. Or, at least mark you own words

the static renderer is one reason, limiting the triggers and limiting the frequency are 2 other fundamental reasons.

If you consider entity_id: solely as a tool to instruct HA which entities it should find in the template, you are right, that is no longer needed. It was however also capable of limiting the update frequency, and limit the triggering entity_id’s. So only 1 out of 3 reasons is no longer needed. And with that 115 took the other 2 from us…

I would hope you would still agree on this:

auto_update: true or, maybe auto_update: auto
being the default, and as is now, with nothing limiting the template engine, and updating on each state change of entities it finds.

auto_update: false or, maybe auto_update: manual
creating a one time snapshot of a template, which doesn’t keep track of any further state changes of the entities found. No further updates whatsoever. Unless by external trigger of an automation/script.

auto_update: minute/day, or, maybe even preferably, an auto_update: entity_id/list of entities
coming as close to the former entity_id as can be, limiting in update frequency, or on entity/ies listed. If entities listed, it wouldn’t update on any state change in the templates entities, but on changes of the listed entity/ies.

3 in 1!

@amelchio

will be interesting to see where @Mariusthvdb ends after the rewrite of all the templates and with the code optimizations from bdraco.

All my rewriting of templates as you mention is/was solely aimed at creating more efficient templates, and would be ranked under the 1 of 3 reasons I used entity_id: Not because the current template needs it, but because the end-user wants/needs control over updating. In an easy and straightforward way. Without the extra clutter of an automation.

Of course I am looking forward to all improvements bdraco has in store for us. Mostly they still are focussed on the same first reason though, and less or not on the other 2…

please give us back control, I think that is quite simply what this amounts to.

(again, in full awe for what the new code does for intelligence and auto-seeking aspects, which is quite simply amazing to have)

[jwelter (John Welter)]

This is interesting, but I have to ask why set these groups at all?

With Lovelace you can do this dynamically with the filter functionality to only list devices with low batteries. And if you want to send a notification on such a thing it can also be done completely dynamically.

As your scripts are a for loop over all the devices it is just as heavy as doing this dynamically, and not having to worry about the groups.

My goal is zero templates and zero groups. I know I will never achieve that as some templates are needed but with each release we get a bit more freedom from this. With 115 it was triggering from attributes - no need to use a template sensor or template trigger to get at an attribute now.

And every step in this path my system, which is busy, gets more responsive and less maintenance required - which is a good thing.

I feel that we have some that like to tinker and tune and those are the ones that are frustrated as that was removed with the latest template changes; but the changes were certainly needed to fix something that was broken as constant posts asking why templates not updating as expected. Now it will just work in many more cases, but without the ability to tune and tinker.

The good of many, versus the desires of the few, and it’s the right decision for the growth of the community.

[finity]

@jwelter

The good of many, versus the desires of the few,

That’s assuming that you can’t have both. It seems it’s a false (self-imposed?) dichotomy.

[anon43302295]

It also assumes that the desires of the few would not benefit the many, when actually it probably would as it would allow the many to have maximum flexibility.

[Mariusthvdb (Marius)]

You are comparing frontend Lovelace with backend functionality.

You might be able to do a dynamic Lovelace card, but it stays there, in that single card, and remember, in the frontend.

Creating a template sensor, or a group in this case, is done in the backend, which makes it available for the full system, including the Lovelace frontend. In any card, or any other integration.

@jwelter

why set these groups at all

well, if you don’t need it, you don’t.

Groups are a great way of, well, grouping entities and manipulate that group. (all motion detection switches) Or check its status (eg presence detection or critical devices, and batteries). Or lights (heck there’s even a special integration for that…)

Don’t think dynamic groups are for the Tinkerers. It’s core functionality

@jwelter

Now it will just work in many more cases, but without the ability to tune and tinker.

Now how’s that for Homeautomation: ‘without the ability to tune’. Don’t think that will benefit anyone the few nor the many.

And, it was unforeseen, so that’s why we are discussing various ways to keep the improvement and repair the loss.
Great community I would say.

[Mariusthvdb (Marius)]

@jwelter

but the changes were certainly needed to fix something that was broken as constant posts asking why templates not updating as expected

btw, nothing was broken. It all worked pretty well. ‘Constant posts asking’ don’t proof anything is broken.

[amelchio]

@finity

That’s assuming that you can’t have both. It seems it’s a false (self-imposed?) dichotomy.

We can’t have it for free. Every feature has a very real cost. For example in code complexity which means more implementation work and more bugs. So we might get both of these things but there will be other nice things that we won’t get because the effort is spent elsewhere. Alternative ways also complicate the documentation so it is more work writing it and more work reading it. It makes it harder to learn the product and harder to master it. It increases the risk of misunderstandings so the product is unknowingly used in some inferior way that hurts the user.

So that’s why I think it is perfectly fine that power users spend their power prowess in writing a few automations to update the state, rather than having support for some alternative mini automation system with shorthands for state triggers (entity_id) or time patterns (sync_interval). That’s full flexibility in what you can do but not in how you can do it and I think that can be a perfectly fine balance.

If arbitrary entity triggers were a frequent need the balance would be different and I would be more okay with the shorthands (like just introduced for attribute triggers). But I have asked for examples and barely seen any. Also, while it is a tricky metric, I do not notice a shitstorm about this change – even though any big change has a few vocal opponents.

I am on the fence about the now() templates that never update. I think those are fairly frequent and the sensor.time solutions are unsatisfactory to me, always were. I would like a better solution here, maybe even sync_interval which is known from other parts of the system (but is being phased out).

Finally, there is the entity_id: [] case which is kind of the opposite of the now() challenge. Here we want to stop the updates and we can no longer do that. Unlike what has been stated, this was not unforseen. It was realized in advance and the suggested workaround is to use an automation to update an input_text.

Now I have learned that input_text is a poor replacement for numerous reasons, some of them inherent. It is indeed unforseen that there is no real workaround. Something was actually taken away, not just changed. I would like to give back what was taken away and auto_update: false was an attempt at that. It is much more limited than the full entity_id support and thus the balance that I mentioned above is IMHO better.

@Mariusthvdb

btw, nothing was broken. It all worked pretty well. ‘Constant posts asking’ don’t proof anything is broken.

Wow, I think that explains why we have a hard time understanding each other’s point of view.

[Mariusthvdb (Marius)]

Hi Amelchio,
fully appreciative of what you wrote above. And I do understand your point of view.
Aamof, I am really glad you write you’re on the fence of the now() updating, and will add the auto_update: false. The latter being fundamentally more important (because we can mitigate the now() already, but cant stop a template from updating as it is)

Those 2 additions would be great to have. Sure we can write a few automations, that’s what we do now for python scripts. so that’s not an issue. (would have loved it to be possible via a template sensors option, but if that will be too costly that’s understood)

anticipated or unforeseen, well, let’s not mince words, and focus on going forward. Glad to help, test, run into issues and suggest possible thoughts for workarounds/solutions if I can. As are we all here, no doubt.

I for one would welcome that new balance.

[jwelter (John Welter)]

Comes down to resources. Sure, the ideal would be we get both. Or get the static templates proposed that updates either via the existing update_entity call, or perhaps also timed update like scan_interal or something.

But someone has to code it, test it, document it, and maintain all that. So with resources being finite it should be to satisfy the many first.

[jwelter (John Welter)]

Go back and read my post as you selectively ignored parts of it.

I said clearly that for the GUI Lovelace can handle it and for back end notifications and such it can be generated on the fly.

So what is the benefit of a group, particularly one that is populated dynamically the same way you would generate on the fly anyhow? You take the same CPU hit to build it - so no savings - but also consume some system resources to maintain this group. Why not just create it on the fly when needed?

I really suspect most of you having performance issues have systems cluttered with groups, templates, and who knows what else (in your case I know the old custom-ui stuff) that all eats performance particularly on a limited system.

But again I will point out with every release lately it’s brought features that have allowed me to de-complex my setup and this has paid off in performance, maintenance, etc.

This whole thread reminds me of the Don Quixote story over battling windmills… Good luck with that.

[Mariusthvdb (Marius)]

@jwelter

for back end notifications and such it can be generated on the fly.

So what is the benefit of a group, particularly one that is populated dynamically the same way you would generate on the fly anyhow

not really sure what the difference is between these?

@jwelter

(in your case I know the old custom-ui stuff) that all eats performance particularly on a limited system.

Here, you are confusing frontend and backend. Custom-ui is done in the frontend browser side computation, just as Lovelace is doing. Jinja is backend server side computation so yes, that’s seen in the processor usage.

for the sake of experiment Ive just taken out my full package of customizations, which leads to exactly no change at all in processor usage%. So that’s not it.

But, this thread is about breaking changes in the template integration. and Amelchio has stated what he is doing and why.
We are all eager to see them live.