Unavailable / Unknown Entity Monitoring - Template Sensor

Share your Projects!
1

This template sensor will iterate the states object and return a count and list of entities that have a state of unknown, unavailable. This is a new version that incorporates a couple of new features such as ignoring entities for a time period and filtering entities based on a partial string match in addition to the entities declared in the ignored_entities group.

The package has been moved to it’s own repository with updated documentation.

Update 2023-9-22: Template and documentation have been updated with new syntax and to exclude new stateless domains (button, scene, event etc.) by default.

22 Likes
2

I love this and have used some of the code to create a sensor for unavailable entities - but a question if I may…

value_template: >
  {% set ignored_sensors = state_attr('group.ignored_sensors', 'entity_id') %}
  {% set unavail =  states.sensor | selectattr('state', 'eq', 'unavailable')
                              | rejectattr('entity_id', 'in', ignored_sensors)
                              | map(attribute='name')
                              | list
                              | length  %}

The ignored sensors is brilliant! But can we do the opposite too?

So like sometimes I have a problem where on startup my Hue network of bulbs, sensors etc is not ready, or sometimes one of my tp-link plugs won’t be ready. Because these entities are in the new style ‘integrations’ their entity_id is not in the states list until the integration is ready.

Obviously once they’ve come online, if they THEN go unavailable the sensor triggers, but is there a way to ‘whitelist’ entity_id’s that should be there, but might not be on startup?

Thanks for your toughts :slight_smile:

2 Likes
3

This is how I dealt with that issue. Basically all the devices I want to monitor are listed in a group and using a similar sensor template. Probably not the slickest way this can be done but it works for me.

Happy to hear input from anyone else!

#################################################################################
## Offline Devices
## state: number of offline devices
## attribute: entity_ids: - comma separated list of unavailable device entity id's
## - updates every minute (sensor.time)
## - device list is group.devices_connected
#################################################################################
sensor:
  - platform: template
    sensors: 
        offline_devices:
        entity_id: sensor.time
        friendly_name_template: >
          {% if states('sensor.offline_devices') | int == 0 %}
            All Devices Online
          {% else %}
            Device Offline
          {% endif %}
        icon_template: >
          {% if states('sensor.offline_devices') | int == 0 %}
            mdi:server-network
          {% else %}
            mdi:server-network-off
          {% endif %}
        value_template: >
          {% set devices = state_attr('group.devices_connected', 'entity_id') %}
          {% set dev = namespace(value=0) %}
          {% for entity_id in devices -%}
            {% if states(entity_id) == 'unknown' %}{% set dev.value = dev.value + 1 %}
            {% elif states(entity_id) == 'unavailable' %}{% set dev.value = dev.value + 1 %}
            {% endif %}
          {%- endfor %}
          {% if dev.value | int > 0 %}
            {{ dev.value }}
          {% else %}
            None
          {% endif %}
        attribute_templates:
          entity_ids: >
            {%- set devices = state_attr('group.devices_connected', 'entity_id') -%}
            {%- set dev = namespace(value=0) -%}
            {%- for entity_id in devices -%}
              {% if states(entity_id) == 'unknown' or states(entity_id) == 'unavailable' %}
              {%- if not dev.value == 0 -%}{{- ',' -}}{%- endif -%}
              {% set dev.value = dev.value + 1 %}
              {{- entity_id -}}
              {%- endif -%}
            {%- endfor -%}
##################################################################################
## System - Devices Connected
## - used to display offline devices in alert
## - used for binary_sensor.offline_devices state
##################################################################################
group:
  devices_connected:
    control: hidden
    entities:
      # media players
      - media_player.all_speakers
      - media_player.broadcast_speakers
      - media_player.music_speakers
      - media_player.dining_room_speaker
      - media_player.bedroom_display
      - media_player.bathroom_speaker
      - media_player.garage_speaker
      - media_player.living_room_tv
      - media_player.bedroom_tv
      - media_player.deck_tv

      # alarm
      - alarm_control_panel.house

      # lights
      - light.living_room_pot_lights

      # fans
      - fan.upstairs_bedroom_fan
      - fan.office_fan
      - fan.upstairs_bathroom_fan

      # covers
      - cover.garage_door_opener

      # climate
      - binary_sensor.upstairs_thermostat_connected

      # nest protects
      - binary_sensor.upstairs_nest_protect_online
      - binary_sensor.downstairs_nest_protect_online

      # cameras
      - camera.side_drive
      - camera.side_door
      - camera.front_drive
      - camera.front_door
      - camera.front_yard
      - camera.side_gate_front
      - camera.side_gate_back
      - camera.back_yard
      - camera.back_door
      - camera.patio_door
      - camera.back_house
      - camera.garage_inside
      - camera.nws_radar

etc. etc.

binary_sensor:
  - platform: template
    sensors:
      device_offline:
        friendly_name: Device Offline
        device_class: problem
        icon_template: mdi:server-network-off
        value_template: "{{ states('sensor.offline_devices') | int > 0 }}"

alert:
  device_offline:
    name: "Device Offline"
    title: "Device Offline"
    entity_id: binary_sensor.device_offline
    state: 'on'
    repeat:
      - 5
      - 60
    can_acknowledge: true
    skip_first: true # to avoid triggering on restart
    message: "- {{ state_attr('sensor.offline_devices','entity_ids').split(',') | join('\n- ') }}"
    done_message: "All devices are now online."
    data:
      actions:
        - action: pause_device_offline_alert
          title: "Pause Alerts"
          icon: !secret PAUSE_BUTTON
      tag: device_offline
      timestamp: "{{ as_timestamp(now()) }}"
      renotify: true
      ttl: 43200 # 12 hours
      priority: high
      requireInteraction: true
      silent: false
      url: /lovelace/system
      icon: !secret OFFLINE_ICON
      image: !secret OFFLINE_IMAGE
      badge: !secret OFFLINE_BADGE
    notifiers: jason
2 Likes
4

Ace, thanks :+1:

5

OK - I’ve messed about with it and simplified it all down and I’ve come up with the following:

sensor:
  - platform: template
    sensors:
      unavailable_entities:
        entity_id: sensor.time
        friendly_name: Unavailable Entities
        value_template: >
          {% set count = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
            |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
            |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
            |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
            |map(attribute='name')|list|length %}
          {%- set devices = state_attr('group.entity_whitelist', 'entity_id') -%}
          {%- set dev = namespace(value=0) -%}
            {%- for entity_id in devices -%}
              {%- if states(entity_id) in ['unavailable', 'unknown', 'none'] -%}
              {%- set dev.value = dev.value + 1 -%}
              {%- endif -%}
            {%- endfor -%}
          {{ count + dev.value }}
        attribute_templates:
          entities: >
            {% set entities = states|selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
              |rejectattr('entity_id', 'in', state_attr('group.entity_blacklist', 'entity_id'))
              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
              |map(attribute='name')|list|join(', ') %}
            {%- set whitelisted = state_attr('group.entity_whitelist', 'entity_id') -%}
            {%- set dev = namespace(value=0) -%}
              {%- for entity_id in whitelisted -%}
                {% if states(entity_id) in ['unavailable', 'unknown', 'none'] %}
                  {%- if not dev.value == 0 %}{{- ', ' -}}
                {% endif %}
                {%- set dev.value = dev.value + 1 -%}
                {{- entity_id -}}
              {% endif -%}
            {% endfor %}
            {%- if not dev.value == 0 %}{{- ', ' -}}{% endif %}
            {{- entities -}}


group:
# Whitelist contains entities that you expect to be there, that may not appear if the integration isn't ready on startup
  entity_whitelist:
    - light.attic
    - light.bath
    - light.bedroom
    - light.boys
    - light.hall
    - light.landing
    - light.kitchen
    - light.led1
    - light.led2
    - light.stairs
    - binary_sensor.attic_motion
    - binary_sensor.bathroom_motion
    - binary_sensor.hall_motion
    - binary_sensor.kitchen_motion
    - binary_sensor.landing_motion
    - switch.heater
    - switch.xmas_lights
    - switch.thermostat

# Blacklist contains sensors that you don't care if they're unavailable - should contain  the sensor itself!!
  entity_blacklist:
    - sensor.unavailable_entities
    - sensor.hacs
    - sensor.bedroom_next_alarm
    - sensor.bedroom_next_reminder
    - sensor.bedroom_next_timer
    - sensor.dark_sky_precip
    - sensor.livingroom_next_alarm
    - sensor.livingroom_next_reminder
    - sensor.livingroom_next_timer
    - sensor.this_device_next_reminder
    - switch.bedroom_repeat_switch
    - switch.bedroom_shuffle_switch
    - switch.everywhere_repeat_switch
    - switch.everywhere_shuffle_switch
    - switch.livingroom_repeat_switch
    - switch.livingroom_shuffle_switch
    - switch.this_device_repeat_switch
    - switch.this_device_shuffle_switch

This state of sensor.unavailable_entities is a count of all the unavailable entities that ARE in the whitelist, that AREN’T in the blacklist, and that are not on either list but are showing as unavailable (or unknown/none) on your system. The attribute ‘entities’ lists them in a comma separated string - whitelisted entities first, followed by any others.

:smiley:

edit - maybe not quite working 100%, as I just lost both my tp-link switches - it registered the 2 devices going offline but it just put the same one in the attributes twice, so a bit of working out still to do - but it’s pretty close!!

6

I’m at work so I can’t really play with it right now. What are these lines supposed to do?

              |rejectattr('entity_id', 'eq' , 'group.entity_blacklist')
              |rejectattr('entity_id', 'eq' , 'group.entity_whitelist')
7

They stop the two groups appearing in the list.

8

thanks for the link to here Marc, there was a parallel thread yesterday:

as you can see, the same base template is used:

 value_template: >
        {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}
      attribute_templates:
        entities: >
          {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none'])| map(attribute='entity_id')  | list | join(',\n ') }}

As asked in the other thread: would you know of a way to group the sensors listed by the state? (‘unavailable’, ‘unknown’, ‘none’).

Petro suggested the |groupby('state') but that doesn’t work unless I take out the map(attribute='entity_id'), and even then renders the list complete with all attributes… not what I need.

9

No idea if there’s an ‘easy way’, but you could loop through the entities three times, one for each state, and add thus effectively add the entities to three different lists.

10

ok thanks,
starting to sound like a new python script…

or like this:

value_template: >
   {{ states | selectattr('state', 'in', ['unavailable', 'unknown', 'none']) | list | length }}
attribute_templates:
  unknown: >
    {{ states | selectattr('state', 'eq','unknown')| map(attribute='entity_id')  | list | join(',\n ') }}
  unavailable: >
    {{ states | selectattr('state', 'eq', 'unavailable')| map(attribute='entity_id')  | list | join(',\n ') }}
  none: >
    {{ states | selectattr('state', 'eq', 'none')| map(attribute='entity_id')  | list | join(',\n ') }}

or maybe even this:

      sensors_uun:
        friendly_name: Sensors uun
        value_template: >
          {{ states|selectattr('state','in',['unavailable','unknown','none'])|list|length }}
        attribute_templates:
          unknown: >
            {% set unknown = states|selectattr('state','eq','unknown')|map(attribute='entity_id')|list %}
            ({{ unknown|count }})
             {{ unknown|join(',\n') }}
          unavailable: >
            {% set unavailable = states|selectattr('state','eq','unavailable')|map(attribute='entity_id')|list %}
            ({{ unavailable|count }})
             {{ unavailable|join(',\n') }}
          none: >
            {% set none_ = states|selectattr('state','eq','none')|map(attribute='entity_id')|list %}
            ({{ none_|count }})
             {{ none_|join(',\n') }}

had to find a trick to assign in the last template for none…

1 Like
11

tweaked a bit more.
this uses an ignore list (which is conditional) set in the sensor config, and has the advantage (I think) you can easily add entities without having to use predefined groups. Of course, if you need these in more than 1 sensor, a group would again come in handy.

also, I noticed a lot of groups are always unknown, so I rejected the domain group, as to cleanup the unknown entities list… same for ‘media_player’ in unavailable

        attribute_templates:
          Unknown: >
            {% set unknown = states|selectattr('state','eq','unknown')
                                   |rejectattr('domain','eq','group')
                                   |map(attribute='entity_id')
                                   |list %}
            {{ unknown|count }}:
            {{'\n'}}{{ unknown|join(',\n') }}

          Unavailable: >
            {% set ignore_list =  ['light.driveway','light.garden_backyard','light.garden_terrace',
                                  'light.porch_outdoors'] if 
                                   is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
            {% set unavailable = states|selectattr('state','eq','unavailable')
                                       |rejectattr('entity_id','in',ignore_list)
                                       |rejectattr('domain','eq','media_player')
                                       |map(attribute='entity_id')
                                       |list %}
1 Like
12

The other bonus of using groups is that you can add/remove the entity from the group and then reload groups and update the entity without having to restart homeassistant.

13

a yes, that Is a big bonus indeed… might reconsider the setup once more… options options…
best of both worlds:

          Unavailable: >
            {% set ignore_list =  ['light.driveway','light.garden_backyard','light.garden_terrace',
                                  'light.porch_outdoors'] if 
                                   is_state('binary_sensor.outside_daylight_sensor','on') else [] %}
            {% set unavailable = states|selectattr('state','eq','unavailable')
                                       |rejectattr('entity_id','in',state_attr('group.entity_blacklist','entity_id'))
                                       |rejectattr('entity_id','in',ignore_list)
                                       |rejectattr('domain','eq','media_player')
                                       |map(attribute='entity_id')
                                       |list %}
            {{ unavailable|count }}:
            {{'\n'}}{{ unavailable|join(',\n') }}
2 Likes
14

I believe this for-loop:

          {%- set devices = state_attr('group.entity_whitelist', 'entity_id') -%}
          {%- set dev = namespace(value=0) -%}
            {%- for entity_id in devices -%}
              {%- if states(entity_id) in ['unavailable', 'unknown', 'none'] -%}
              {%- set dev.value = dev.value + 1 -%}
              {%- endif -%}
            {%- endfor -%}
          {{ count + dev.value }}

can be reduced to this:

          {% set devices = expand('group.entity_whitelist')
             | selectattr('state', 'in', ['unavailable', 'unknown', 'none'])
             | list | count %}
          {{ count + devices }}
1 Like
15

Nice, I’ll try it.

Whilst you’re on format, can you think of a solution to this…

The whitelist is important because it checks for devices that are not online yet, therefore don’t have the entity_id registered in homeassistant yet.

Problem being the say 3 hours later if they go unavailable they now are in the entity list, so the sensor gets the same entity from the whitelist and the states loop causing duplicates.

Any ‘easy’ way to the final count number and entity list to ignore the duplicates?

16

I thought you might ask that and … I’ll be honest … I only focused on that one for-loop and didn’t take a holistic view of its operation. In other words, I dunno.

However, your invention intrigues me so I’ll have a look at in depth when I have more time to experiment with it.

1 Like
17

Excellent, I’ll await your thoughts :slightly_smiling_face:

18

in the meantime… :wink:

please could you explain this:

    # only continue if current number of sensors is equal or more than the number when triggered
      - condition: template
        value_template: >
          {{ states('sensor.entities_uun')|int >= trigger.to_state.state|int }}

I am having a hard time understanding why these aren’t equal… I mean, the current state is also the to_state? hence we always check for the from_state < to_state ?

iow, why would this template be necessary, if you already use

      - condition: template
        value_template: >
          {{ trigger.to_state.state|int > trigger.from_state.state|int }}

in the automation?

if Id change it to:

  - alias: 'Unavailable Notification'
    id: 'Unavailable Notification'
    description: 'Send notification when entities go.'
    trigger:
    # run whenever unavailable sensors sensor state changes
      platform: state
      entity_id: sensor.entities_uun
    condition: []
    action:
    # wait 30 seconds before rechecking sensor state
      - delay:
         seconds: 30
    # make sure the sensor is updated before we check the state
      - service: script.update_entities_uun
    # only continue if current number of sensors is equal or more than the number when triggered = 
    # only run if the number of unavailable sensors had gone up
      - condition: template
        value_template: >
          {{ trigger.to_state.state|int > trigger.from_state.state|int }}    # create a persistent notification
      - service: persistent_notification.create
        data_template:
          title: 'Entities Unavailable!'
          message: >
            ### Unavailable entities: {{ '\n' + state_attr('sensor.entities_uun','Unavailable').split(', ') 
                                         | join('\n') }}
          notification_id: 'entities_uun_alert'

would this do the following:

upon state change, update the sensor,
if to_state<from_state, do nothing else,
if to_state>from_state, create notification?

added this to keep checking when needed:

  - alias: 'Check for unavailabe entities'
    id: 'Check for unavailabe entities'
    trigger:
      platform: time_pattern
      minutes: "/5"
    condition:
      condition: template
      value_template: >
        {{is_state('persistent_notification.entities_uun_alert','notifying')}}
    action:
      service: script.update_entities_uun
19

I have a question: why is there a need for both a whitelist and a blacklist?

I would have thought only a whitelist is needed; just monitor the availability of desired entities and all others are ignored.

Or is having white and black lists meant to be a convenience? If I need to monitor 97 out of 100 entities, then it’s more convenient to blacklist 3 of them than whitelist 97 entities. But that implies using one or the other but I see in your example that you are using both. So if an entity is not whitelisted or blacklisted how is it handled?

  • It’s not on the whitelist so it should be rejected.
  • It’s not on the blacklist so it should be accepted.

How is this conflicting situation handled?

20

yes, my thoughts too.

I’m only using the blacklist option, (and an ignore list inside the sensors) and playing with the results, end up adding more to the blacklist. Seems no need for a whitelist is necessary.

I guess it could depend on the amount of entities one has and needs to check for state. Still, one of either would suffice?

have an additional request:
can we somehow add * to the ex/inclusions?

explain: I noticed the SolarEdge sensor is quite unreliable at the moment, and seems to go offline every now and then.
No need for the sensors to monitor, so Id like to exclude the set of solardedge sensors completely.

I could add them individually to the blacklist option. They are not of a dedicated domain, so can’t use that technique.

checking the available options on the states, I’d need something in the lines of
|rejectattr('object_id','startswith','solaredge')

doesnt work in Jinja though…

{{ trigger.event.data.object_id.startswith(‘solaredge.’)}}
{{ trigger.event.data.entity_id.split(’.’)[1].startswith(‘solaredge’)}}

would probably do it in an automation, but I need it in the sensor…