Heads up! Upcoming breaking change in the Template integration

[Mariusthvdb (Marius)]

Thanks Marc, will check after the time trial (TdF)

Response time might differ depending on the count of states of course …

[123 (Taras)]

I believe Mariusthvdb summarized the issue nicely when he described it as being “expensive”.

As shown in the screenshot, the use of states means it now listens to everything (entities in all domains; automation, scene, script, zone, etc).

That’s different from how it worked prior to 0.115 (which created no listeners for states). Therefore for the people who are experiencing performance issues, they may wish to start their debugging adventure by removing any “expensive” Template Sensors they may have and seeing if it improves anything.

Whether it proves to be ‘expensive’ or not, from a design perspective it seems overreaching to have a Template Sensor refreshed when anything else changes state.

[bdraco (J. Nick Koston)]

If you want to limit updates to only a specific interval, something like this will work:

- id: '1600355905751'
  alias: Update Unavailable Entities
  description: ''
  trigger:
  - platform: time_pattern
    hours: '*'
    minutes: '15'
    seconds: '0'
  condition: []
  action:
  - service: input_text.set_value
    data:
       value: "{{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}"
    entity_id: input_text.unavailable_entities
  mode: single

[anon43302295]

@123

Whether it proves to be ‘expensive’ or not, from a design perspective it seems overreaching to have a Template Sensor refreshed when anything else changes state.

But that is kinda the point of the sensor in this case is it not? Monitor all state changes and record the ones that change to unavailable.

[bdraco (J. Nick Koston)]

@anon43302295

{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
            |map(attribute='entity_id')|list|length }}

{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_ignore_list', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_ignore_list')
            |list|length }}

Taking out the extra map will make this a bit faster

[123 (Taras)]

@anon43302295

point of the sensor in this case is it not? Monitor all state changes and record the ones that change to unavailable.

I agree its purpose is to report when something is unavailable. It’s the “something” that I feel ought to have a narrower scope as opposed to its current scope which is “everything”.

For example, as a consequence of using states, the Template Sensor checks if any automation, zone, script, scene, etc becomes unavailable. Is that even a remote possibility? Nevertheless, listeners are created for all automations (and zones, scripts, etc) and the Template Sensor gets updated when any one of those ‘extraneous’ entities changes state. Seems like a very wide (global) net is being cast to catch just a few (local) fish.

[Mariusthvdb (Marius)]

the way I used it, was have a look a startup, see what got initiated and what not. was this expected? no? act. If expected, add the to the ignore list.

update the sensor. manually by clicking the update script.
all well? thumbs up. if not, have another look.

That did the trick to see for config errors, or integrations not being initiated at startup, mostly the weather integration.
For me, I don’t need the constant check of all states. Just a simple tool to reveal the issues in the current state of affairs.

update

even testing the above templates in the template field causes issues: my Hue integration has always been an indicator of internal issues, and becomes unavailable completely and remains like that, if I enter the template.
even after emptying the template editor, it wont come back.
so yeah, it is very expensive, and I am not yet very happy with the new way of handling these templates at all.
entering entity_id: was not so bad, and had the advantage of controlling the update process… now, the system keeps checking states, even if unnecessary and even worse, not desired.

at the moment. y system has more than 278 unavailable entities, so the list wouldn’t be anywhere near usefully listed using Nicks automation

seems any effort to control the evaluation of these templates (using an automation with an input_text, or maybe a python script) is futile if a single run of it already has such impact.

Cant help but feeling there is a bug somewhere, not giving back to the system, or staying in a loop somehow. I had a very swiftly responding system up to HA 114.4, even with all my many entities, without so much as a single Hue unavailability left.
This must be the muddiest iteration yet.

[anon43302295]

@Mariusthvdb

at the moment. y system has more than 278 unavailable entities

I think your system has a lot more going wrong than this sensor

[123 (Taras)]

What would you say to the suggestion to undo the deprecation and permit entity_id to supercede the template?

As you and I know, that’s the way it works in versions prior to 0.115. If you specify entity_id you’re telling the system to "create listeners for these entities and not for any in value_template".

I have no idea what are the technical implications of allowing for this behavior now that there’s an entirely different way that the system identifies entities in a template.

[Mariusthvdb (Marius)]

That’s what I suggested before. Would be best to have both options available.

So I would say: yes please!

[pollinolas]

hi guys how could i fix this template to work with sensor.date:

{{ ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'][now().weekday()] }}

i tried this:

{% set day = states('sensor.date') %}
{{ ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'][now().weekday()] }}

In the template editor it validates it well, but then when checking the configuration it throws an error

greetings and thanks

[bdraco (J. Nick Koston)]

@pollinolas

{% set day = states('sensor.date') %}
{{ ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'][now().weekday()] }}

Do you have the time & date integration enabled?

Time & Date

[pollinolas]

@pollinolas

'

Oops, my mistake, I was about to delete the post, I forgot to add ‘>’
Thanks for the quick reply

it might be a good suggestion for next month WTH

[123 (Taras)]

You can also include sensor.date in your template like this:

{{ ['Monday','Tuesday','Wednesday','Thursday','Friday','Saturday','Sunday'][as_local(states.sensor.date.last_updated).weekday()] }}

EDIT

Simply adding {% set day = states('sensor.date') %} to the template will work in the current (0.115) and previous versions of Home Assistant but adds another line to the template containing a seemingly unused variable called day (in your template). Nevertheless, it’s a quick way to compensate for the deprecation of entity_id.

The alternative way (shown above) will only work in 0.115+ because it uses a new function called as_local(). However, it doesn’t require adding another line and there’s no question that it’s an integral part of the template.

[pollinolas]

Thank you very much for your wisdom, after having been reading the thread I had tried “as_local (states.sensor.date.last_updated)” but I have not been able to get it.
I actually had no idea about jinja until I started with home assistant, based on reading you and other gurus and doing my tests in the editor I am learning.
I love seeing its complex templates and I am really enjoying learning and understanding them, I am not satisfied with copy-paste, thank you very much to all gentlemen.

[123 (Taras)]

How do feel about the idea of re-instating the use of entity_id in a Template Sensor?

Its behavior would remain the same as in previous versions (i.e. it supersedes any entities within the template).

Would resistance to the idea be due to:

1. Technical challenges given the significantly different way that entities are now identified and listeners are assigned.
2. Philosophical disagreement about how it should work.
3. A bit of both.
4. None of the above.

[bdraco (J. Nick Koston)]

I implemented a proof of concept a few weeks ago. However, after discussion, it was decided against moving it forward as there are better ways (ones that require less maintenance over time and that don’t require manually maintaining lists of entities) to accomplish what has been discussed in this thread. I certainly don’t feel comfortable maintaining two unique template implementations in the same integration, and it seems unlikely we would find someone who would sign up to that.

I rebased the commit to fix the conflicts with 0.115.1 if you want to play with it https://github.com/bdraco/home-assistant/commit/d30293e92609e222999b369b3454a1638c499997

I can’t speak to how well it works as it doesn’t have sufficient test coverage, and I haven’t tested it in production since I first wrote it a few weeks ago.

There is going to be some transition time as inefficient templates are adjusted. After that, we end up with a system that requires less manual configuration for the majority of cases.

[amelchio]

@123

Whether it proves to be ‘expensive’ or not, from a design perspective it seems overreaching to have a Template Sensor refreshed when anything else changes state.

Do you really mean that? If it could be made very fast you would still prefer to have a delayed update? From a design perspective I think the new implementation is beautiful because it is always correct. But it is obviously a problem that it is causing systems to break down.

It is not yet clear to me, though, whether this problem is so fundamental that we ought to implement a way to disable automatic updates. For example, a feedback loop detector might be able to find (and maybe even fix) the templates that are causing systems to break down.

I have tried to reproduce the performance issue by installing the unavailability sensor that @anon43302295 showed above but so far I am not noticing anything wrong.

[123 (Taras)]

OK, thanks.

Although the release notes for 0.115 are top-notch and mention the deprecation as a Breaking Change (under the heading of Templates), I have the feeling that its impact is proving to be greater than anticipated.

I agree that the cases involving sensor.time and such are easily mitigated and don’t merit the resurrection of entity_id. However, there are examples in this thread where the lack of entity_id makes it far more challenging (if not impossible) to control what gets assigned a listener.

It’s my understanding that entity_id was ostensibly implemented as a workaround for the fact the old way of assigning listeners couldn’t always identify all the entities. However, a side-benefit of this ‘workaround’ was that it could override the entities in value_template.

For example, I may have a complex value_template (i.e. see examples above that report unavailable entities) but I do not want any entity within it to be automatically assigned a listener. I only want listeners for whatever entities I specify in entity_id.

In 0.115, that side-benefit is no longer available. If I only want the template to update exclusively according to (for example) sensor.time, and not according to any entity listed in the template, I know of no way to do that.

[Mutt (Muttley)]

From 0.114.4 to 0.115.1 my processor use has fallen from 3-4% (varies during the day) to a fairly rock solid 1%.
I’m very happy with bdraco 's work on this so far and though I’m sure we’ll find a few upsets yet to bite us in the ass, we also need to fine tune some of the templates we’ve used a hammer to fix in this first round. I am confident that with bdraco’s work things will improve even further
@anon43302295 said his processor use was a steady 5%, I think he was referring to somebody else with expensive templates