Sensor - Unavailable/Offline Detection

Tags: #<Tag:0x00007f7c75040580>

I see; there’s another layer to this onion.

Moments after startup, there are:

  1. Entities whose state might be unavailable (we can check their state)
  2. Entities that haven’t even been created yet (we cannot check their state)

Whitelist contains entities from the second category. Entities from the first category will be monitored by default (i.e. no need to add them to the whitelist), unless they are in the black list. Is that correct?

Exactly that :slightly_smiling_face:

Would it be fair to say that entities comprising the second category exist because of checking for their presence too soon after startup?

In other words, if I allowed a full minute to pass (after startup) I’d encounter fewer “2nd category entities” than if I checked a mere 15 seconds after startup?


This situation reminds me of the guidance provided for the Homekit integration (Disable auto-start). It suggests there may be a need to delay the start of the HomeKit integration until all dependent entities become available. The instructions suggest disabling the integration’s auto-start option, then creating an automation to enable it at the appropriate time. The automation’s trigger can be an event (if the dependent integration or entities can issue one) or simply after a fixed delay (a full five minutes in the documentation’s example!). So the simplest strategy is to give dependent entities sufficient time to establish themselves before checking on them.

FWIW, in my case, I use an automation with a fixed delay of merely 5 seconds since all dependent entities are MQTT-based. They’re discovered very quickly because it’s only a matter of subscribing to the MQTT broker and receiving all the retained values. In contrast, a longer delay may be needed for integrations that have to poll all related devices.

The problem there being - what if they never come online?

So imagine the reason my homeassistant is being restarted is because of a power cut. Unbeknown to me this event has also killed my esphome device that controls my heating (seen as switch.heater in my whitelist). Homeassistant boots, waits one minute for everything to settle down and then does the check. Switch.heater has never come online, so it never makes it in to the states list, but I still need to know its not there so I can take action to fix the problem.

1 Like

I haven’t had a chance read the rest of the thread yet but this is exactly why I did my offline device sensor the way I did with the group. Now I’m going to read the rest of the thread and realize you know this and you’re talking about something else. :slightly_smiling_face:

1 Like

OK, thanks for the explanation, it makes sense to me now. What I had not considered are entities that exist only after they are (re)discovered by their associated integration. So if the integration fails to instantiate them, they don’t exist as far the state machine and templates are concerned. So the whitelist serves as a ‘master entity reference’.

1 Like

Without wishing to appear lazy (no point reinventing the wheel :wink: ) did anyone come up with the ‘definitive solution’ to this?

Aside from duplicates between whitelist and states list I have a one that works well. It’s refined a little bit from the one I posted up there :arrow_double_up: somewhere, but @123 's idea to reduce the code by using expand doesn’t work for whitelisted entities that don’t yet exist, so it’s still fairly big.

Still interested in knowing (and hoping!) if there’s a way to remove duplicates from the final number and list.

Will post my latest version of the sensor when I’m at the computer.

1 Like

really small nudge

Ha, yeah, I haven’t picked up my computer in days - tomorrow morning though I’m all over it :slightly_smiling_face:

1 Like

OK - the sensor -

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        entity_id: sensor.time
        friendly_name: Unavailable Entities
        value_template: >
          {% set count = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
            |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
            |map(attribute='entity_id')|list|length %}
          {%- set devices = state_attr('group.entity_whitelist', 'entity_id') -%}
          {%- set dev = namespace(value=0) -%}
            {%- for entity_id in devices -%}
              {%- if states(entity_id) in ['unavailable', 'unknown', 'none'] -%}
              {%- set dev.value = dev.value + 1 -%}
              {%- endif -%}
            {%- endfor -%}
          {{ count + dev.value }}
        attribute_templates:
          entities: >
            {% set entities = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
              |map(attribute='entity_id')|list|join(', ') %}
            {%- set whitelisted = state_attr('group.entity_whitelist', 'entity_id') -%}
            {%- set dev = namespace(value=0) -%}
              {%- for entity_id in whitelisted -%}
                {% if states(entity_id) in ['unavailable', 'unknown', 'none'] %}
                  {%- if not dev.value == 0 %}{{- ', ' -}}
                {% endif %}
                {%- set dev.value = dev.value + 1 -%}
                {{- entity_id -}}
              {% endif -%}
            {% endfor %}
            {%- if not dev.value == 0 %}{{- ', ' -}}{% endif %}
            {{- entities -}}

It currently updates every minute (with sensor.time) but I’m thinking of refreshing it with an automation that runs every second instead just to be sure it captures straight away when things go offline.

Obviously you’ll want

group:
  entity_whitelist:
    # Entities you want to check are there that might not be in the states list at startrup
  entity_blacklist:
    - sensor.unavailable_entities
    # any other entities that you don't mind being unavailable/unknown/none

As I say, aside from the duplication of whitelisted items that get in to the states list and then go offline, it’s pretty ticketyboo.

2 Likes

:rofl:

I have no idea what that means (but I can make an educated guess…) but for some reason I found that hilarious.

1 Like

Lol - hopefully you guessed right…

:slightly_smiling_face:

I did! :+1:

:slightly_smiling_face:

1 Like

There is something new in 0.104 that might influence the way you detect unavailable sensors:

Entity handling
Draft: One major change: we’re restoring all entity registry entities on Home Assistant startup this allows Homekit/Google/Alexa to always find all entities to expose and Lovelace will no longer show yellow/red boxes. All entities are being restored as “unavailable” they can be differentiated from normal unavailable entities by the “restored” state attribute


Now officially part of release version 0.104.
Improved entity handling on startup.

2 Likes

After going through this thread and then looking at the changes for 0.104 I think this will render the whole whitelist concept unnecessary? Am I interpreting that correctly? I’ve updated to 0.104 now and my original sensors were not affected by the change. They function as they did before but I never utilized the whitelist concept that @mf_social came up with.

I was able to scoop a couple techniques from this thread to simplify my own templates and make them a little more robust so thanks guys!

1 Like

Can confirm that whitelist is unnecessary in 104 - will post my revised sensor when I get chance.

I think it basically boils down to this…

    unavailable_entities:
      entity_id: sensor.time
      friendly_name: Unavailable Entities
      value_template: >
        {% set count = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
          |rejectattr('entity_id', 'in', state_attr('group.ignored_sensors', 'entity_id'))
          |rejectattr('domain', 'eq', 'group')
          |map(attribute='entity_id')|list|length %}
        {{ count }}
      attribute_templates:
        entities: >
          {% set entities = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.ignored_sensors', 'entity_id'))
            |rejectattr('domain', 'eq', 'group')
            |map(attribute='entity_id')|list|join(', ') %}
          {{- entities -}} 

This is how unavailable entities show up in lovelace now!

image

Indeed - I ended up with:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        friendly_name: Unavailable Entities
        value_template: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
            |map(attribute='entity_id')|list|length }}"
        attribute_templates:
          entities: "{{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |map(attribute='entity_id')|list|join(', ') }}"
        unit_of_measurement: items

…And using an automation to update the sensor every second. Works well :slight_smile:

I never did get around to resolving the duplicates. However, it looks like procrastination paid off because 0.104 has obviated the need for it. Nevertheless, I don’t wish to appear as a slacker so I offer you some minor streamlining.

According to my tests, you can replace this:

            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')

with this:

            |reject('in', expand('group.entity_blacklist'))
            |reject('eq', states.group.entity_blacklist)

You can also remove this filter from the value_template:

map(attribute='entity_id')

Here’s the streamlined version:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        friendly_name: Unavailable Entities
        value_template: >
          {{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |reject('in', expand('group.entity_blacklist'))
            |reject('eq', states.group.entity_blacklist)
            |list|length}}
        attribute_templates:
          entities: >
            {{states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |reject('in', expand('group.entity_blacklist'))
              |reject('eq' , states.group.entity_blacklist)
              |map(attribute='entity_id')|list|join(', ')}}
        unit_of_measurement: items
1 Like