Unavailable / Unknown Entity Monitoring - Template Sensor

{{ expand(states.sensor, states.light) | etc ... }}
2 Likes

of course, not yet used to that completelyā€¦ thanks!

        {%- for s in expand(states.binary_sensor, states.sensor)
            if 'diskstation' in s.entity_id  %}
            {{s.entity_id}}{% if not loop.last %}, {% endif %}
          {%- endfor %} 

though, it still would be (much) more efficient to create a true group and have that tracked.
This is a very fine way for us yaml users to easily select all entities to create the group with.

nice.

Devs are aware of the issues and it looks like they are already working on a fix. It isnā€™t just this sensor causing problems, itā€™s all states based templates. I had dozens of them scatted throughout my config. Iā€™ve temporarily disabled them all and the performance difference is night and day. My UI is snappy even on mobile and it never was before.

I think maybe these templates were always causing an issue but most of them were being throttled by sensor.time or the other monitored entities so it wasnā€™t as apparent. Probably not as pointed out by @123

Really hope they get this worked out, the states based templates are very useful!

No. Prior to 0.115, states and domain-based variations like states.sensor, states.light, etc were never assigned listeners. As a result a template like this:

{{ states |  list }}

would get no listeners and so the template would only be evaluated at startup and never again (until the next startup). Thatā€™s why you supplied:

entity_id: sensor.time

so that the template would be evaluated periodically (once a minute, corresponding to when sensor.time changes state).

In 0.115, the entity-identification technique is far more clever. Now it understands states means all entities so it listens to all entities. In other words, when any entity in your system changes state, the template get evaluated. For most users, thatā€™s far more frequently than just once a minute.

Notice what the last line says for this template:
Screenshot from 2020-10-02 13-30-29


EDIT

There are a few PRs in the pipeline designed to provide the user with more control over how the template is evaluated. For example, one PR introduces a means to control the frequency of evaluations (with far more flexibility than merely once a minute or once a day).

1 Like

Thanks for clarifying. I havenā€™t had time to dig into the nuts and bolts of the changes. I guess the other improvements in v115 are the cause of all my giddiness right now. :grin:

Seriously, for me the performance difference in the last couple versions has been nothing short of amazing. Iā€™m still chugging along with a fairly extensive config and an RPi3 and an SD card so needless to say, I notice it!

I recall one post where bdraco calculated how many evaluations a userā€™s template was undergoing (based on the number of entities they had which was around 2500) and the result was several billion times a day. Clearly, thatā€™s enough workload to become, to put it mildly, noticeable.

For the moment, the best solution is to, wherever possible, avoid using states in a Template Sensor. The proposed new controls will make it easier in the near future but for now, the simplest mitigation technique is to re-design the template.

Just want to confirm we should still hold off on using this? Iā€™ve read through the past bunch of messagesa nd the comments on the gist. It seems we are waiting for a fix from HA and should not use this until then?

I am curiousā€¦

If I use this in the dev tools:

{{ states.sensor | selectattr('state', 'eq', 'unavailable') 
                 | map(attribute='name')
                 | list }}

It tells me it creates listeners for all my entities but what happens when I close out of the dev tools? Do those listerners get destroyed? Or is even running that code in the template tester a problem? (havent noticed a slow down yet )

In 0.116, templates containing states or states.domain (like states.sensor, states.light, etc) are handled differently than in 0.115. The template is evaluated periodically (every minute) as opposed to when any entity identified within the template changes its state. This applies to templates youā€™re testing in the Template Editor, in Template Sensors (and other Template entities) and in Template Triggers.

So even if the Template Editor claims it listens to all entities when it sees states in your template, it only evaluates it once a minute (or when you change the code in the Template Editor, which is the usual behavior).

That is good news, I am still on 0.115 but I will get upgraded to 0.116 before I install this sensor. I assume that resolves all the issues with this sensor?

Theoretically, it makes it works like it did in versions prior to 0.115 where if you had

entity_id: sensor.time

the template was evaluated once a minute. Full Disclosure: Although I collaborated on this Template Sensor, I donā€™t use it so I cannot confirm its behavior in 0.116.

Keep in mind that this new ā€˜periodic evaluationā€™ behavior in 0.116 only applies to templates containing states (or a derivative of it).

Iā€™ve updated the gist for this sensor. Have tested on an RPi3 on v0.117.0 and all is well in the state of Denmark.

2 Likes

would your solution help me with this code as well?

sensor:
  - platform: template
    sensors:
      energy_consumption_total:
        friendly_name: Energy Consumption Total
        icon_template: mdi:flash
        unit_of_measurement: kWh
        value_template: "{{ states('sensor.shelly_shem_3_40f52001972c_1_total_consumption')|float + states('sensor.shelly_shem_3_40f52001972c_2_total_consumption')|float + states('sensor.shelly_shem_3_40f52001972c_3_total_consumption')|float}}"

Iā€™m not sure I understand what you mean or what youā€™re asking for. What problem are you having with the sensor you posted and how do you think the unavailable entities sensor can help you?

I get the following error with the automation in unavail_sensor.yaml

Logger: homeassistant.components.automation.unavailable_entities_notification
Source: helpers/service.py:190
Integration: Automation (documentation, issues)
First occurred: 2:28:27 PM (12 occurrences)
Last logged: 4:23:22 PM

  • Unavailable Entities Notification: Choose at step 1: default: Error executing script. Unexpected error for call_service at pos 1: Error rendering data template: UndefinedError: ā€˜Noneā€™ has no attribute ā€˜splitā€™
  • Unavailable Entities Notification: Error executing script. Unexpected error for choose at pos 1: Error rendering data template: UndefinedError: ā€˜Noneā€™ has no attribute ā€˜splitā€™
  • While executing automation automation.unavailable_entities_notification

What do I need to change to avoid that error? Here is the automation portion of unavail_sensor.yaml

## OPTIONAL Example automation to demonstrate how you can utilize this sensor
automation:
  - id: unavailable_entities_notification
    alias: "Unavailable Entities Notification"
    description: "Create persistent notification if there are unavailable entities, dismiss if none."
    mode: restart

    trigger:
      - platform: state
        entity_id: sensor.unavailable_entities

    action:
      - choose:
          conditions:
            - condition: numeric_state
              entity_id: sensor.unavailable_entities
              below: 1

          sequence:
            - service: persistent_notification.dismiss
              data:
                notification_id: unavailable_entities

        default:
          - service: persistent_notification.create
            data:
              title: Unavailable Entities
              message: "- {{ state_attr('sensor.unavailable_entities','entities').split(',')|join('\n- ') }}"
              notification_id: unavailable_entities

Iā€™m assuming you used he package as is and didnā€™t rename anything. Did the sensor work previously without this error or is are you just setting it up for the first time? Does the sensor work sometimes or does it not work at all? Is there a pattern to when youā€™re getting that error? How often are you getting it? What is the state of the sensor when the error happens?

It looks like the entities attribute doesnā€™t exist for some reason. Does it only happen when there are no unavailable entities (the sensor state is 0)? That is the only thing I can think of that might cause that error. Maybe a timing issue where the state is going to 0 for a wee bit before the notification is dismissed and that is causing the error?

When you see the error, can you put the following in the developer template tool and post the results?

{{ states(ā€™ā€˜sensor.unavailable_entitiesā€™) }}
{{ state_attr(ā€˜sensor.unavailable_entitiesā€™,ā€˜entitiesā€™) }}

You could also maybe try replacing this template in the sensor definition:

            {{ states|selectattr('state','in',['unavailable','unknown','none'])|rejectattr('domain','eq','group')
                |rejectattr('entity_id','in',state_attr('group.ignored_entities','entity_id'))|map(attribute='entity_id')|join(',') }}

With this.

            {{ states|selectattr('state','in',['unavailable','unknown','none'])|rejectattr('domain','eq','group')
                |rejectattr('entity_id','in',state_attr('group.ignored_entities','entity_id'))|map(attribute='entity_id')|join(',')
                  if states('sensor.unavailable_entities')|int > 0 else 'No unavailable entities' }}

Actually, yes try this - please let me know if it solves the issue so I can update the gist. Donā€™t forget to reload your template entities!

It happens every time after I reload my template entities even after making the change that you suggested. I had no unavailable entities because I had some of them listed in the ignored_entities. I removed them from the list and restarted. The error still shows up each time I reload my template entities.

Now that you mention it, I think Iā€™ve seen the same error reloading my template entities at some point but I havenā€™t seen it lately. I guess during the template reload the dictionary object is null so the split function throws an error. It has no bearing on the sensor operation after the templates have been reloaded. I think you can safely ignore this warning. Thereā€™s probably a check I can put in there to avoid it but I canā€™t replicate the error at the moment to test it out. If I do, Iā€™ll post the solution here.

1 Like

Just curious if anyone has made this into a blueprint

I havenā€™t seen any, but I personally donā€™t really have a need for blueprints so I havenā€™t paid much attention. I donā€™t think there is really much to blueprint here, just drop the package in your config and youā€™re all set.

Unless of course you have some cool automations being driven by this sensor that others would find useful. I suppose you could always blueprint those.

1 Like

I have created a group of entities that I want to ingore but the entities are still triggering a notification after restart:

group:
  ignored_entities:
    entities:
      - binary_sensor.updater
      - group.entities_dont_care

I have verified that group.entities_dont_care is correctly defined in developer-tools