This is precisely why I posted here and not directly to a new GH issue. If i can figure out what condition(s) are required to reproduce then i’ll open an issue there.
This is precisely why i used the word “schedules”. At the end of the day, some code path is triggered; that path ends up calling out to the jinja2 library for evaluation. I do not have an issue with the result of the template… the issue is that the result is not being computed in a timely or consistent manner… hence ‘schedule’.
While i am very certain that the ephemeral template expression via the webUI and template entities use the same evaluation engine, I am less certain that the triggers that lead to that eval are the same… which is why ephemeral expression seems to do what I want, but the template entity does not appear to be evaluated when i’d expect it to.
I don’t understand what you’re asking for?
The template sensor in question is detailed a few posts up in this thread:
Hopefully this is enough to explain things / satisfy your ask?
me@some-host # grep -C2 "template" configuration.yaml
# Template sensors are now a specific integration
# See: https://www.home-assistant.io/integrations/template/
##
template: !include_dir_merge_list devices/template/
me@some-host # cat devices/template/notifications.yaml
# get a count of the number of persistent notifications
##
- sensor:
- name: "Active Persistent Notifications"
# Unique ID is required for mgmt through the web UI
unique_id: tmpl-active-persistent-notifications
icon: >-
{% if states.persistent_notification|length > 0 %}
mdi:bell-badge
{% else %}
mdi:bell
{% endif %}
state: >
{{states.persistent_notification|length}}
What you call “schedule” is part of the same code. Listeners are created and the listeners drive the updates. It’s literally identical top to bottom. Theres no difference. Not sure how else to explain it. 
Yes, It is a massive codebase.
As is common with large code bases, a given function will often will have several different paths to it. Each path presents an opportunity for a different behavior.
One path may immediately try to render the template and return a value, another path may make a note that the template should be re-evaluated and put that note on a queue where some other process may eventually get the note. While the code that evaluates the template may be identical in both paths, the act of evaluating the template is scheduled to take place at different times in the two examples. One is scheduled “as soon as possible” and the other is scheduled “after the other $things that were already in line are dealt with”.
In the case of the dev tools / template page, the code that actually parses the template string and tries to return something useful lives here: core/homeassistant/helpers/template.py at 4c51299dcc7b690e4e6789cee826e2d67b50eed2 · home-assistant/core · GitHub
The Dev Tools page triggers a template evaluation every time client/browser detects a change in the forms’ content; when the text changes, a message is sent over the websocket. There are some route/validate layers i’ll skip over, but eventually the message lands here:
The message looks something like this:
{"type":"render_template","template":"this is an example template string","timeout":3,"id":48}
In this case, the template evaluation is scheduled for the next event loop (call_soon_threadsafe()) as soon as the message is received.
Since the “there is a new notification to show” and the “this notification was dismissed” messages are exchanged over the same WS, it makes sense that the template value is always updated in near real time. On the next loop, it’ll be evaluated and the result pushed back to the browser via WS if the result has changed.
But here’s the kicker: the “state of this $thing has just changed” messages are also sent to the browser over that same WS. This is how the dev tools/states page can show the state of a contact sensor - for example - in near real time based on weather or not that door where the sensor is attached is open or closed.
But this begs the question: why is the devtools template result practically instantaneous but the template
sensor is clearly not?
Either:
- the template sensor is not being scheduled for timely re-evaluation
or - the template sensor is being evaluated in a timely manner… but the “the state has changed” message is not propagating to the web interface and the other callback function(s) where the automation for the status light is dealt with.
or - some other possibility that eludes me
By definition, i can’t speak to the 3rd option. If you have a plausible explanation that better explains the symptoms, then please share!
The second option seems unlikely just because it’s a less simple failure. Reloading the template entities immediately gets the dev tools/states view to show a correct value and the automation that turns the blue light off responds to the state change and turns the light off. This indicates that there is no issue communicating “the state of the sensor has changed” to the browser and to the other call backs that deal with the particular automation.
Which leads me to the first option. I am not a heavy user of async python and there’s some non-trivial use of it in the HA code base so i’m struggling to identify precisely what about this code path results in a sensor whos template does not reliably get re-evaluated.
I would recommend that you submit an issue containing all of the info that you posted in posd #18 above.
If that doesn’t get someone’s attention to at least start looking into it then there’s nothing more you can do.
I guess you could (and I’m surprised I am even suggesting it given my dislike for the medium) try to get some attention for this on the HA Discord.
It looks like you’ve done your homework on this (at least as much as you could) so hopefully given that effort on your part someone might take up the issue.
Please for the love of god stop trying to mansplan the code I helped with. Thanks.
The real area that you should be looking is this class, which sets up the listeners for templates.
If there was a bug, it would be here. Specifically in this area:
But as said earlier, I don’t think there is a bug in this code. As stated above, your system for some reason renders domain templates as None, which will update/remove the listeners for the template. That’s where you should be focusing and something is wrong with your system.
EDIT: To show you exactly the problem:
Notice how an invalid domain state still renders to a template DomainState object? This is what drives the update. Yours was rendering unknown, which is actually None because the template editor code changes None to unknown when rendering.
You’re the only one who has that issue. If you can replicate it, that’s what you should be writing up. Not the mess above because the mess above is a result of the underlying issue on your system.
Apologies if that’s how that came across. I was trying to communicate that irrespective of how the template is evaluated, the when is not consistent. It sounds like we’re on the same page.
The unknown issue hasn’t happened since the one time I was able to observe it. I wish i had the mental clarity at the time to screenshot it. Since i have not been able to observe it before or since, I am treating it like an outlier; something that is possibly resulted from some sort of user error in administering the test or observing the results… the most likely source of most outliers.
In zero of the screenshots i posted yesterday did that particular unknown or None behavior re-surface. If it is all the same code underneath, then it is incredibly difficult to explain why the template editor was accurate but the template sensor was not. Since they both consist of the same expression {{states.persistent_notification|length}}, and the expression in the template editor was always the correct integer, it is safe to conclude that the template sensor also would have never evaluated to None.
If, somehow, the template entity had evaluated to something that the length function could not be applied to, then either the template editor would have surfaced a TypeError or similar exception… right?
Neither home-assistant.log or the logs viewer through web UI indicate any such problem, though. If i had seen any, of course they’d be included in a post!
As a quick test:
me@some-host# cat devices/template/notifications.yaml
# get a count of the number of persistent notifications
##
- sensor:
- name: "Active Persistent Notifications"
# Unique ID is required for mgmt through the web UI
unique_id: tmpl-active-persistent-notifications
icon: >-
{% if states.persistent_notification|length > 0 %}
mdi:bell-badge
{% else %}
mdi:bell
{% endif %}
state: >
{{states.persistent_notification|length}}
# quick test
- sensor:
- name: "TEST PNOT CT"
state: >
{{None|length}}
me@some-host# grep -i "error.*template.*$" home-assistant.log
2021-07-23 15:37:01 ERROR (MainThread) [homeassistant.helpers.event] Error while processing template: Template("{{None|length}}")
2021-07-23 15:37:01 ERROR (MainThread) [homeassistant.components.template.template_entity] TemplateError('TypeError: object of type 'NoneType' has no len()') while processing template 'Template("{{None|length}}")' for attribute '_attr_state' in entity 'sensor.test_pnot_ct'
This is good to know and starts to sound plausible. If the callbacks are removed from the template sensor, is it safe to assume that the cached value for the template sensor would be persisted? If that assumption holds, and something went wrong w/ the template sensor evaluation (but not the dev tools template eval?! and nothing in logs!?) between notifications being fired off and dismissed, it would explain why the value does not decrement as expected but usually does increment as expected. Calling reload on the template service ‘fixing’ the issue also lends some credibility that callbacks to re-evaluate are not firing (assuming because they no longer exist).
I posted because it does appear that I’m the only one w/ the issue and the time to document it publicly.
The original intent of this thread was to identify where the cause of theses symptoms might exist. This whole time i have been trying to precisely communicate what the issue is and, with some difficulty, when it occurs… so that it becomes a narrower and narrower problem description that can eventually be replicated in an isolated environment.
I am grateful for most of the posts in this thread; they have been helpful in moving me towards the end goal of understanding the problem with the intent of solving it.
I’ll turn up the logging to see if anything else pops up:
me@some-host # grep -A3 -B1 "logger" configuration.yaml
# You can turn specific components to a specific level
# See: https://www.home-assistant.io/integrations/logger/#log-levels
#
# critical, fatal, error, warning, warn, info, debug, notset
##
logger:
default: info
logs:
homeassistant.components.template.template_entity: debug
It appears that way, but I’d expect an error when it fails.
Yes, if your listeners are removed it would stay static, forever. Only a reload (which re-evals the template) would cause the listener to be recreated.
That’s why this is troubling:
If that simple template resolves the domain as None, then the listeners will go away and the template will “freeze”.
Either way, turning on debug is a good place to start and hopefully the messages can lead to a cause. Because right now, what’s happening doesn’t make sense. I was hoping that you had more to your configuration that would point to a simple solution but that does not seem to be the case. All in all, the path forward is to tabulate this information inside an issue on github.