Unavailable / Unknown Entity Monitoring - Template Sensor

anon43302295 · December 16, 2019, 11:23am

Blacklist is entities that you are expecting to be unavailable so should not appear in the list. For example - the Alexa Media Player component creates several sensors that are unavailable a lot of the time, and I don’t want them to be monitored.

Whitelist is for entities that might not be in the states list on startup, for example - homeassistant reboots and the Hue hub isn’t ready when it does. Since hue became configured via ‘integration’ this means that my lights would not appear in the states list until they are dynamically added when the Hue hub becomes ready. If that isn’t for 10 minutes, then the ‘normal’ bit of the sensor that goes through the states list won’t know that I should have multiple lights and sensors from the Hue hub so will return 0 unavailable entities when in fact there are around 20. Whitelist makes the sensor look for devices that should be in the states list even if they’re not.

Entities that are not on either list should be monitored as well.

It works perfectly, except for the duplication.

Possibly a more concise way to put it:

Sensor should ignore anything on the blacklist.

Should report anything in states that is unavailable.

Should also report anything that is in the Whitelist but not in the states list yet.

123 · December 16, 2019, 1:27pm

I see; there’s another layer to this onion.

Moments after startup, there are:

Entities whose state might be unavailable (we can check their state)
Entities that haven’t even been created yet (we cannot check their state)

Whitelist contains entities from the second category. Entities from the first category will be monitored by default (i.e. no need to add them to the whitelist), unless they are in the black list. Is that correct?

anon43302295 · December 16, 2019, 1:40pm

Exactly that

123 · December 16, 2019, 2:15pm

Would it be fair to say that entities comprising the second category exist because of checking for their presence too soon after startup?

In other words, if I allowed a full minute to pass (after startup) I’d encounter fewer “2nd category entities” than if I checked a mere 15 seconds after startup?

This situation reminds me of the guidance provided for the Homekit integration (Disable auto-start). It suggests there may be a need to delay the start of the HomeKit integration until all dependent entities become available. The instructions suggest disabling the integration’s auto-start option, then creating an automation to enable it at the appropriate time. The automation’s trigger can be an event (if the dependent integration or entities can issue one) or simply after a fixed delay (a full five minutes in the documentation’s example!). So the simplest strategy is to give dependent entities sufficient time to establish themselves before checking on them.

FWIW, in my case, I use an automation with a fixed delay of merely 5 seconds since all dependent entities are MQTT-based. They’re discovered very quickly because it’s only a matter of subscribing to the MQTT broker and receiving all the retained values. In contrast, a longer delay may be needed for integrations that have to poll all related devices.

anon43302295 · December 16, 2019, 3:25pm

The problem there being - what if they never come online?

So imagine the reason my homeassistant is being restarted is because of a power cut. Unbeknown to me this event has also killed my esphome device that controls my heating (seen as switch.heater in my whitelist). Homeassistant boots, waits one minute for everything to settle down and then does the check. Switch.heater has never come online, so it never makes it in to the states list, but I still need to know its not there so I can take action to fix the problem.

jazzyisj · December 16, 2019, 5:27pm

I haven’t had a chance read the rest of the thread yet but this is exactly why I did my offline device sensor the way I did with the group. Now I’m going to read the rest of the thread and realize you know this and you’re talking about something else.

123 · December 16, 2019, 9:52pm

OK, thanks for the explanation, it makes sense to me now. What I had not considered are entities that exist only after they are (re)discovered by their associated integration. So if the integration fails to instantiate them, they don’t exist as far the state machine and templates are concerned. So the whitelist serves as a ‘master entity reference’.

klogg · January 4, 2020, 10:39am

Without wishing to appear lazy (no point reinventing the wheel ) did anyone come up with the ‘definitive solution’ to this?

anon43302295 · January 4, 2020, 11:06am

Aside from duplicates between whitelist and states list I have a one that works well. It’s refined a little bit from the one I posted up there somewhere, but @123 's idea to reduce the code by using expand doesn’t work for whitelisted entities that don’t yet exist, so it’s still fairly big.

Still interested in knowing (and hoping!) if there’s a way to remove duplicates from the final number and list.

Will post my latest version of the sensor when I’m at the computer.

klogg · January 6, 2020, 4:13pm

_{really small nudge}

anon43302295 · January 6, 2020, 4:25pm

Ha, yeah, I haven’t picked up my computer in days - tomorrow morning though I’m all over it

anon43302295 · January 7, 2020, 4:04pm

OK - the sensor -

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        entity_id: sensor.time
        friendly_name: Unavailable Entities
        value_template: >
          {% set count = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
            |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
            |map(attribute='entity_id')|list|length %}
          {%- set devices = state_attr('group.entity_whitelist', 'entity_id') -%}
          {%- set dev = namespace(value=0) -%}
            {%- for entity_id in devices -%}
              {%- if states(entity_id) in ['unavailable', 'unknown', 'none'] -%}
              {%- set dev.value = dev.value + 1 -%}
              {%- endif -%}
            {%- endfor -%}
          {{ count + dev.value }}
        attribute_templates:
          entities: >
            {% set entities = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
              |map(attribute='entity_id')|list|join(', ') %}
            {%- set whitelisted = state_attr('group.entity_whitelist', 'entity_id') -%}
            {%- set dev = namespace(value=0) -%}
              {%- for entity_id in whitelisted -%}
                {% if states(entity_id) in ['unavailable', 'unknown', 'none'] %}
                  {%- if not dev.value == 0 %}{{- ', ' -}}
                {% endif %}
                {%- set dev.value = dev.value + 1 -%}
                {{- entity_id -}}
              {% endif -%}
            {% endfor %}
            {%- if not dev.value == 0 %}{{- ', ' -}}{% endif %}
            {{- entities -}}

It currently updates every minute (with sensor.time) but I’m thinking of refreshing it with an automation that runs every second instead just to be sure it captures straight away when things go offline.

Obviously you’ll want

group:
  entity_whitelist:
    # Entities you want to check are there that might not be in the states list at startrup
  entity_blacklist:
    - sensor.unavailable_entities
    # any other entities that you don't mind being unavailable/unknown/none

As I say, aside from the duplication of whitelisted items that get in to the states list and then go offline, it’s pretty ticketyboo.

finity · January 7, 2020, 6:05pm

I have no idea what that means (but I can make an educated guess…) but for some reason I found that hilarious.

anon43302295 · January 7, 2020, 6:31pm

Lol - hopefully you guessed right…

finity · January 7, 2020, 7:08pm

I did!

123 · January 11, 2020, 2:16pm

There is ~~something new in 0.104~~ that might influence the way you detect unavailable sensors:

Entity handling
Draft: One major change: we’re restoring all entity registry entities on Home Assistant startup this allows Homekit/Google/Alexa to always find all entities to expose and Lovelace will no longer show yellow/red boxes. All entities are being restored as “unavailable” they can be differentiated from normal unavailable entities by the “restored” state attribute

Now officially part of release version 0.104.
Improved entity handling on startup.

jazzyisj · January 16, 2020, 8:45pm

After going through this thread and then looking at the changes for 0.104 I think this will render the whole whitelist concept unnecessary? Am I interpreting that correctly? I’ve updated to 0.104 now and my original sensors were not affected by the change. They function as they did before but I never utilized the whitelist concept that @anon43302295 came up with.

I was able to scoop a couple techniques from this thread to simplify my own templates and make them a little more robust so thanks guys!

anon43302295 · January 16, 2020, 8:57pm

Can confirm that whitelist is unnecessary in 104 - will post my revised sensor when I get chance.

jazzyisj · January 16, 2020, 10:28pm

I think it basically boils down to this…

    unavailable_entities:
      entity_id: sensor.time
      friendly_name: Unavailable Entities
      value_template: >
        {% set count = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
          |rejectattr('entity_id', 'in', state_attr('group.ignored_sensors', 'entity_id'))
          |rejectattr('domain', 'eq', 'group')
          |map(attribute='entity_id')|list|length %}
        {{ count }}
      attribute_templates:
        entities: >
          {% set entities = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.ignored_sensors', 'entity_id'))
            |rejectattr('domain', 'eq', 'group')
            |map(attribute='entity_id')|list|join(', ') %}
          {{- entities -}}

This is how unavailable entities show up in lovelace now!

anon43302295 · January 17, 2020, 12:07pm

Indeed - I ended up with:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        friendly_name: Unavailable Entities
        value_template: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
            |map(attribute='entity_id')|list|length }}"
        attribute_templates:
          entities: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |map(attribute='entity_id')|list|join(', ') }}"
        unit_of_measurement: items

…And using an automation to update the sensor every second. Works well