Heads up! Upcoming breaking change in the Template integration

templates or automations, some fundamental decision being made here…
lead me to few related questions:

having several ‘old’ automations using the

    trigger:
      platform: event
      event_type: state_changed

I was wondering if these create listeners for all states, or, as I thought to have made them, only for the states described in the condition:

      value_template: >
        {{trigger.event.data.entity_id in state_attr('group.all_lights_only','entity_id')}}

they worked before, but I no longer use them, so turned off the automations. If turned off, do they in effect NOT create listeners for these states either, or should I completely comment them from the system to prevent them form doing so?

trying to release the system from as much pressure as possible…

mine has gone up from about 8/9% to generic 16/20% with regular spikes of 25 up to even 40 or more and that’s with all counters and state changed automations and sensors taken out. So much less functionality.

There’s no denying in my setup 115’s advantage of always knowing the state is costly.

Heck, take the above unavailable template sensor. Before, we needed to hit the enter key to refresh, right now, even testing a template like that chokes the system. (That should really not be possible should it, what’s the test environment for, just a thought. Test a bad template and the system goes down …)

I would really love the ‘superseding’ thought of @123. Act as per design now by default. But also allow for some extra control for those templates (and automations) that need that. Have those update only on certain triggers, be it a named entity_id, or by script (which would be possible if an entity_id was set)

Maybe call it trigger_only: to indicate this is not the default behavior? Or, as said before, triggers_update: like the custom button-card that had the exact same impact of templates being updated constantly without cause.

So ‘if’ you need even one of these intensive (expensive) templates, do as bdraco suggests, assign an input text for each, have 3 automations (one every 10 mins, one every 30 mins and one every hour) and assign each of the templates to one of these automations (depending on your required update frequency) to populate their respective text/number (whatever) and then have the sensor display the text.
It’s a bit of work but you picked the route of having so many of these AND it’s better than running your CPU choked.
I don’t think ‘griping’ about it is going to make bdraco’s further optimisations come any quicker

1 Like

This will listen to all state changes and then render a template to see if it should run the action. So it is almost as expensive as the unavailability template.

If you still have a performance degradation with everything taken out, maybe you actually are having an unrelated issue with 0.115 :thinking:

Right. This should work and it works for me. It seems there is something else in your system that is causing the slowdown, in combination with this template.

Again, personally I am not 100% against the entity_id: but I want to be 100% sure that it returns because it is needed, not because we didn’t get to the bottom of the problems.

2 Likes

that’s great to hear, and yes I understand your words on the ‘something else in the system’. Of course I admit having a rather extensive system, running on a Pi4. Still, it does its work, very well I might add, and it has been sort of a sports to me to make it as efficient as possible, given its complexity.

Well, of course I didnt take everything out. Only the counting sensors and the unavailability sensor, which proved to have beaten the system, because now all is running. Slower than before, but it is running.

the entity_id option did, (and hopefully will any time soon in an incarnation you feel good about) offer the end-user the ultimate control he/she might need in certain cases. HAs been explained above, no need to repeat (I certainly don’t want to ‘gripe’ eh Mutt :wink: )

You see, it is rather easy for fellows here to tell me I have a complex system, and because of that should not complain because my processor hits 10%. I could counter that with the simplicity of several setups I see shared in the GitHub, which of course wouldn’t compare with more worked-through setups.

Point I think I am trying to make is if at all possible, consider both sides of the user community (and the lot in between).
The progress being made now is huge, many people will only feel the positive sides of it.
I certainly don’t want to take that away. But it has its downsides in efficiency.

I did love the control option that has been taken out now. If that could be brought back: wow. 1+1 would be 4!
Having a single or continuous evaluation in the template editor would be nice too. Single to check if a template is syntactically correct, and continuous to check it in operation.

btw, what is your assessment on the state_changed automations being ‘off’. Do they still create listeners (and should we comment them mout) or are they truly inactive in the system (and can we safely leave them, being ‘off’)

as another example I have this:

  - alias: GitHub boolean updates sensor
    id: GitHub boolean updates sensor
    trigger:
      platform: event
      event_type: state_changed
    condition:
      condition: template
      value_template: >
        {{trigger.event.data.entity_id in
            state_attr('group.github_repo_updates','entity_id')}}

can we already use the expand(‘group.github_repo_updates’) in a trigger_template (if yes, how, because it would have to evaluate to true?) so it sees to only those. I think that would be an improvement?

Ideally I would need to do something like:

  - alias: GitHub boolean updates sensor
    id: GitHub boolean updates sensor
    trigger:
      platform: state
      entity_id: >
        {{expand('group.github_repo_updates')|map(attribute='entity_id')|join(',\n')}}

thanks!

1 Like

I’ve come to the end of the road on the optimizations as I’ve run out of data, and I don’t want to start optimizing prematurely. If you still have high cpu usage and its not covered by once of the PRs I posted above, please post a py-spy. Instructions linked below:

2 Likes

Okay I ran some tests for optimisations (on @sparkydave 's original (a-), (which was to produce a slug name, so had to be lower case, but also just used now() (so wouldn’t update) )
I then tried my initial offereing (b-)
Then @bdraco 's (c- )
And Then @123 's (d-)
I then cheated and removed the “! lower” from the end of mine (e-) (as you lot didn’t so … ) it makes quite a difference.


BUT they are ALL sub 1 ms by a large factor.

And ALL of these are triggered off sensor.date so that’s 0.5 ms processor time per DAY
(So generally, if you take care, @anon43302295 has a damn good point about concentrating your efforts and letting the rest just get on with it.
@Mariusthvdb, see what you can do to optimise and bung the rest into bdracos’s automations (if you just can’t live without some expensive sensors)
cheers !
:rofl:
Values change per run but remain relative to each other (scale wise)

Edit: I see Taras is replying so I’ll beat him to the punch and say :-
" Yes, You won the Trophy for the fastest ! "
( I’ll send you one of Marc’s Tee-Shirts )

1 Like

Ah, another perfect caption for a tee-shirt…

3 Likes

My point wasn’t about processing time but limiting the number of listeners to only the ones your application truly needs.

Loose analogy: let’s say you are interested in all RSS feeds concerning climate change. However, it’s difficult to subscribe to just those feeds so you subscribe to all RSS feeds in existence and then filter them for just climate change. Far more efficient would be to subscribe to whatever subset is available (all science-related feeds), subscribe to those, and then filter that shorter list.

That’s the point I was making about a template that starts with states. Unlike previous versions, 0.115 will detect states and create listeners for all of your entities. Aside from casting a needlessly wide net, your template will be evaluated every time any entity changes state. This seems like a wasteful use of resources.

All that’s needed is to narrow states down to something smaller and more relevant to the template’s purpose. For example, {{ expand(states.sensor, states.binary_sensor) | etc ...}} constrains it to just two domains.

I think toilet paper is a better fit :rofl:

2 Likes

No, I 100% get that.

I was just persuing my own curiosity about the 4 different approaches to essentially the same question and 4 of the solutions given to achieve it.
I’ll (try to when I can) to optimise any template (and i know because I’m lazy I’ll only test a sub-set of the possible situational combinations).

I wasn’t commenting on your proposal which intelectually has merit (it just doesn’t seem to apply to my current templates, so I’m not invested in it. (but will study the outcome regardless).

Edit: and also to show people how they can test their own templates to compare speed

I didn’t know you wanted to get that intimate with Marc’s face !!!

:thinking:

No no no, not his face, just the quote!

1 Like

He loves me, it’s no secret, he’s only human…

2 Likes

Hi all, I have an interesting outcome I hope to get some helpful insight on.
The Alexa Media Player has an attribute (last_called) that allows one to track which Alexa/Echo device was last spoken to, and many people make a template sensor to track this.
I changed my template sensor to the following (commented out the entity_id(s)):

  - platform: template
    sensors:
      last_alexa:
#       entity_id:
#         - media_player.kitchen_echo
#         - media_player.alex_room_echo
#         - media_player.gina_s_fire_tv_stick
        value_template: >
          {% set alexa_players = [ states.media_player.kitchen_echo,states.media_player.alex_room_echo, states.media_player.gina_s_fire_tv_stick ] %}
          {{ alexa_players  | selectattr('attributes.last_called','eq',True) | map(attribute='entity_id') | first }}

In my testing, this template sensor works. However I noticed that during HA startup, it throws an error:

....skipped beginning stuff.......
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 74, in attrgetter
    item = environment.getitem(item, part)
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/sandbox.py", line 384, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: 'None' has no attribute 'attributes'

Again after the error is thrown, in my testing the template sensor seemingly works.

Could it be that these particular media players are not yet setup within HA at the time when HA first looks at this template?

FYI, I ran it through the template editor and it says:
This template listens for the following state changed events:

  • Entity: media_player.alex_room_echo
  • Entity: media_player.gina_s_fire_tv_stick
  • Entity: media_player.kitchen_echo

In the template editor, the template’s resulting value does indeed change on the fly when I speak to different Alexa/Echo devices. So again, it seems to work, just not sure why the error.

EDIT:
I just tried the following to see if there was any difference:

        value_template: >
          {{ expand(states.media_player.alex_room_echo, states.media_player.kitchen_echo, states.media_player.gina_s_fire_tv_stick) | selectattr('attributes.last_called','eq',True) | map(attribute='entity_id') | first }}

And it works, and this time, there is no error is thrown.

After looking at a few more py-spy results, its apparent that there is a performance issue with fetching the logbook data on the more info popup that is making it harder to distinguish between what is caused by the template changes, and what is caused by the logbook changes.

I’ve gone ahead and optimized the logbook to be at least an order of magnitude faster for single entity lookups: https://github.com/home-assistant/core/pull/40075

If you are still experiencing a performance issue, please try this PR in addition to the ones I posted above.

1 Like

That is likely what is going on here. Can you post the whole traceback without skipping the beginning stuff?

Here is the full log output:

2020-09-20 18:17:29 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template("{% set alexa_players = [ states.media_player.alex_room_echo, states.media_player.kitchen_echo, states.media_player.gina_s_fire_tv_stick ] %}  {{ alexa_players  | selectattr('attributes.last_called','eq',True) | map(attribute='entity_id') | first }}
")
Traceback (most recent call last):
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/homeassistant/helpers/template.py", line 285, in async_render
    return compiled.render(kwargs).strip()
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/environment.py", line 1090, in render
    self.environment.handle_exception()
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/environment.py", line 832, in handle_exception
    reraise(*rewrite_traceback_stack(source=source))
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/_compat.py", line 28, in reraise
    raise value.with_traceback(tb)
  File "<template>", line 1, in top-level template code
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 510, in do_first
    return next(iter(seq))
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 1117, in do_map
    for item in seq:
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 1324, in select_or_reject
    if func(item):
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 1317, in <lambda>
    return seq, lambda item: modfunc(func(transfunc(item)))
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/filters.py", line 74, in attrgetter
    item = environment.getitem(item, part)
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/jinja2/sandbox.py", line 384, in getitem
    return obj[argument]
jinja2.exceptions.UndefinedError: 'None' has no attribute 'attributes'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/homeassistant/helpers/template.py", line 306, in async_render_to_info
    render_info._result = self.async_render(variables, **kwargs)
  File "/opt/homeassistant/venv_3.7/lib/python3.7/site-packages/homeassistant/helpers/template.py", line 287, in async_render
    raise TemplateError(err) from err
homeassistant.exceptions.TemplateError: UndefinedError: 'None' has no attribute 'attributes'

Thanks. We should be able to safely defer setup of the attribute templates until startup without causing any unexpected side effects.

Please open a github issue and I’ll work on a PR to make the adjustment.

OK Thanks!