A sensor to list unavailable devices

Sorry, yes I did, although I am not aware of any such devices in my installation. If you scan the entity lists above, it is easy to see which one is the main one. I can find all my devices in the Devices page of the UI.

The code I am now using gives the result I want at the moment – the device name give by the user.

My only concern with this code is that there is a theoretical possibility that one of the sub-entities of a device will go unavailable while the main entity (an the device itself) is really still available. That would put the device on the list erroneously. Whether that actually happens in practice remains to be seen! I have not tested it by turning all devices off yet.

You’re welcome.

About that last line in your template, doing multiple replaces, what’s that all about?

None of my devices have a name_by_user attribute (it’s not one of the usual device attributes) so I can’t replicate the result that required you to do all of that replacement. I’m guessing you got a list, or list of lists, or something like that, and you are trying to ‘flatten’ it.

Without that line the output is a raw list…

[
  "Back door closure sensor",
  "Conservatory inner door",
  "Guest bedroom TRV",
  "Landing socket",
  "Sophias bedroom east TRV",
  "Studio motion sensor",
  "Studio temperature and humidity"
]

That line just tidies it up by removing the unwanted characters, so we get:

Back door closure sensor
Conservatory inner door
Guest bedroom TRV
Landing socket
Sophias bedroom east TRV
Studio motion sensor
Studio temperature and humidity

Of course, you can change name_by_user back to name, if that is what you want to report. You then get the system name for the devices, which in my case is:

_TZ3000_msl6wxk9 TS0202
_TZ3000_xr3htd96 TS0201
eWeLink WB01
IKEA of Sweden TRADFRI control outlet
LUMI lumi.airrtc.agl001
LUMI lumi.sensor_magnet.aq2

In my case the list is intended for a householder with little technical knowledge. If I tell him lumi.airrtc.agl001 is offline and needs attention, that will mean nothing to him. He cannot find it easily by searching Devices in the UI, either. If, however I say that Guest bedroom TRV is offline and needs attention, he can go straight to it, check the batteries and reconnect it to ZigBee if necessary. It is easy to find in the UI under that name.

Note too that the system name is not unique. In my case, Back door closure sensor, Conservatory inner door and about 20 similar devices have the name LUMI lumi.sensor_magnet.aq2, so reporting that that ‘device’ is unavailable is not very helpful, even if you recognise it as a closure sensor.

Use join to convert the list of strings to strings delimited by newlines.

{{ states
  | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
  | selectattr('state', 'in', ['unavailable'])
  | map(attribute='entity_id')
  | map('device_attr', 'name_by_user') 
  | reject('match', 'None')
  | unique
  | list 
  | sort
  | join('\n')
}}

Yep! That’s another improvement! Thanks again :bouquet:

So, in case anyone else wants it, here is the final full code – tested, tidied up and annotated.

Thanks to all of you for your help and to others on the forum and elsewhere on the internet who made contributions to this subject.

##----------------------------------------------------------------------------------------------------------------------
##
## Unavailable Entities Sensor
##
## 05-Feb-2024 | Andy Symons | created
##
## Credit: based loosely on https://github.com/jazzyisj/unavailable-entities-sensor/blob/main/README.md
##
## The sensor provides lists related to real devices, not internal entities, helpers, automations etc.,
## Entities with state 'unknown' are not counted, because it is possible for a device to have a sub-entity that is
##    unknown while the device itself is available.
##
## The STATE simply gives the count of unavailable entities.
## The long results have to be attributes because the state cannot contain more than 255 characters:
##   ATTRIBUTE 'entity_id_list' contains a list of unavailable entities using their entity ids, which may or may not have been set by the user.
##   ATTRIBUTE 'entity_name_list' contains a list of unavailable entities using their friendly names as assigned by the user.
##   ATTRIBUTE 'device_name_list contains a list of the devices that are unavailable, which is to say having one or more entities that are unavailable,
##      using their friendly names as assigned by the user.
##
##----------------------------------------------------------------------------------------------------------------------


template:
  - sensor:
      name: "Unavailable Entities"
      unique_id: unavailable_entities
      icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
      state_class: measurement
      unit_of_measurement: entities

      # The entity state is the count of unavailable entites
      state: >
        {{ states
        | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
        | selectattr('state', 'in', ['unavailable'])
        | map(attribute='entity_id')
        | unique
        | list 
        | count
        }}

      # The long results have to be attributes because the state cannot contain more than 255 characters.
      attributes:
        ## A list of unavailable entities using their entity ids (which mnay or may not have been set by the user).
        entity_id_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | reject('match', 'None')
          | list 
          | sort 
          | join('\n')
          }}

        ## A list of unavailable entities using their friendly names as assigned by the user.
        entity_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch']) 
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('state_attr', 'friendly_name') 
          | reject('match', 'None')
          | list 
          | sort 
          | join('\n')
          }}

        ## A list of the devices that are unavailable, using their friendly names as assigned by the user.
        device_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('device_attr', 'name_by_user') 
          | reject('match', 'None')
          | unique
          | list 
          | sort 
          | join('\n')
          }}

# HOW THE ATTRIBUTE TEMPLATES WORK
#  -- Taking device_name_list as an example...
#
# {{ states                                                        -- all the states (entities) in the system
#    | selectattr('domain','in',['binary_sensor', 'climate', etc.  -- filter only the entities for real devices
#    | selectattr('state', 'in', ['unavailable'])                  -- filter only entities that are unavailable
#    | map(attribute='entity_id')                                  -- get the entity id from the record
#    | map('device_attr', 'name_by_user')                          -- map the entity id onto the device name
#    | reject('match', 'None')                                     -- take out names 'None' (meaning there is no name, so not a device)
#    | unique                                                      -- take out duplicates (devices usually have several entities)
#    | list                                                        -- make into a list (in the template sense)
#    | sort                                                        -- put them in alphabetical order
#    | join('\n')                                                  -- take out extraneous punctuation for a tidy output
#  }}

This web page is colouring the last few lines incorrectly; it’s just comment and will be fine when you paste it into a YAML file.

If you use this sensor code, please …

6 Likes

FWIW, there are one or two other templating shortcuts that are possible but, seeing that you have already have your own Solution, it’s good enough.

I would like to see them anyway!
I readily confess that my solution was achieved more by trial-and-error than by a deep understanding of how templates work :person_shrugging:

I’m running into an issue using this because I have a separate sensors.yaml and when I paste this in, I think the indentation is wrong. Below are the non-critical errors that only exist when I paste the code in to my YAML file and attempt to check the HA config.

  - platform: template
    sensors:
        unavailable_entities:
            name: "Unavailable Entities"
            unique_id: unavailable_entities
            icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
            state_class: measurement
            unit_of_measurement: entities

      # The entity state is the count of unavailable entites
    state: >
        {{ states
        | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
        | selectattr('state', 'in', ['unavailable'])
        | map(attribute='entity_id')
        | unique
        | list 
        | count
        }}

      # The long results have to be attributes because the state cannot contain more than 255 characters.
    attributes:
        ## A list of unavailable entities using their entity ids (which mnay or may not have been set by the user).
        entity_id_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | reject('match', 'None')
          | list 
          | sort 
          | join('\n')
          }}

        ## A list of unavailable entities using their friendly names as assigned by the user.
    entity_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch']) 
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('state_attr', 'friendly_name') 
          | reject('match', 'None')
          | list 
          | sort 
          | join('\n')
          }}

        ## A list of the devices that are unavailable, using their friendly names as assigned by the user.
    device_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('device_attr', 'name_by_user') 
          | reject('match', 'None')
          | unique
          | list 
          | sort 
          | join('\n')
          }}


I don’t see your log with errors — only YAML config.

state and attributes should be indented more, but if you learn to interpret the errors and check the docs, you can fix it yourself.

If you have a separate sensors.yaml, then the key sensor is already in your configurations.yaml, presumably (as sensor: !include sensor.yaml) so you should not have the sensors keyword again in sensors.yaml

I have labeled mine, so it is written like this:

sensor sun2: !include sun2.yaml
sensor climate: !include climate.yaml
...

I managed to get it down to one error.

Invalid config for 'template' from integration 'sensor' at sensors.yaml, line 394: 'attributes' is an invalid option for 'sensor.template', check: sensors->unavailable_entities->attributes

I had to replace a bunch of the items like “state” with “value_template”, etc. I’m not sure what to do with the attributes section. I tried various indents. See below for updated YAML. I know the beginning is correct because all the other items in my sensors.yaml are formatted the same way.

  - platform: template
    sensors:
        unavailable_entities:
            friendly_name: "Unavailable Entities"
            unique_id: unavailable_entities
            icon_template: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
            unit_of_measurement: entities

# The entity state is the count of unavailable entites
            value_template: >-
                {{ states
                | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                | selectattr('state', 'in', ['unavailable'])
                | map(attribute='entity_id')
                | unique
                | list 
                | count
                }}

# The long results have to be attributes because the state cannot contain more than 255 characters.
            attributes:
# A list of unavailable entities using their entity ids (which mnay or may not have been set by the user).
                entity_id_list: >-
                    {{ states
                    | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | reject('match', 'None')
                    | list 
                    | sort 
                    | join('\n')
                    }}

# A list of unavailable entities using their friendly names as assigned by the user.
                entity_name_list: >-
                    {{ states
                    | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch']) 
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | map('state_attr', 'friendly_name') 
                    | reject('match', 'None')
                    | list 
                    | sort 
                    | join('\n')
                    }}

# A list of the devices that are unavailable, using their friendly names as assigned by the user.
                device_name_list: >-
                    {{ states
                    | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | map('device_attr', 'name_by_user') 
                    | reject('match', 'None')
                    | unique
                    | list 
                    | sort 
                    | join('\n')
                    }}

Thanks a million for sharing your final code! Really helpful!

This is for those who are having the same problem with an empty device_name_list:
Both attributes entity_id_list and entity_name_list work fine at my end.
However, I had a problem that the attribute device_name_list produced just an empty value. Reason was that the list is based on name_by_user which needs to be manually defined for each device which I never did.

My workaround was to replace name_by_user by just name.
Accordingly, my corresponding line in the device_name_list section is the following:
map('device_attr', 'name') instead of map('device_attr', 'name_by_user')

I edited @AndySymons sensor to output unavailable device count rather than entities.

It also inverts the order of the sensor details so that the list of unavailable devices is at the top and the entity IDs are at the bottom.

##----------------------------------------------------------------------------------------------------------------------
##
## Unavailable Devices Sensor
##
## 05-Feb-2024 | Andy Symons | created
## 16-May-2024 | mr_roboto | updated to track unavailable device count rather than entities
##
## Credit: based loosely on https://github.com/jazzyisj/unavailable-entities-sensor/blob/main/README.md
##
## The sensor provides lists related to real devices, not internal entities, helpers, automations etc.,
## Entities with state 'unknown' are not counted, because it is possible for a device to have a sub-entity that is
##    unknown while the device itself is available.
##
## The STATE simply gives the count of unavailable entities.
## The long results have to be attributes because the state cannot contain more than 255 characters:
##   ATTRIBUTE 'entity_id_list' contains a list of unavailable entities using their entity ids, which may or may not have been set by the user.
##   ATTRIBUTE 'entity_name_list' contains a list of unavailable entities using their friendly names as assigned by the user.
##   ATTRIBUTE 'device_name_list contains a list of the devices that are unavailable, which is to say having one or more entities that are unavailable,
##      using their friendly names as assigned by the user.
##
##----------------------------------------------------------------------------------------------------------------------


template:
  - sensor:
      name: "Unavailable Devices"
      unique_id: unavailable_devices
      icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
      state_class: measurement
      unit_of_measurement: devices

      # The entity state is the count of unavailable devices.
      state: >
        {{ states
        | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
        | selectattr('state', 'in', ['unavailable'])
        | map(attribute='entity_id')
        | map('device_attr', 'name_by_user')
        | reject('match', 'None')
        | unique
        | list
        | count
        }}

      # The long results have to be attributes because the state cannot contain more than 255 characters.
      attributes:
        ## A list of the devices that are unavailable, using their friendly names as assigned by the user.
        device_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('device_attr', 'name_by_user')
          | reject('match', 'None')
          | unique
          | list
          | sort
          | join('\n')
          }}

        ## A list of unavailable entities using their friendly names as assigned by the user.
        entity_name_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | map('state_attr', 'friendly_name')
          | reject('match', 'None')
          | list
          | sort
          | join('\n')
          }}

        ## A list of unavailable entities using their entity ids (which may or may not have been set by the user).
        entity_id_list: >-
          {{ states
          | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
          | selectattr('state', 'in', ['unavailable'])
          | map(attribute='entity_id')
          | reject('match', 'None')
          | list
          | sort
          | join('\n')
          }}


# HOW THE ATTRIBUTE TEMPLATES WORK
#  -- Taking device_name_list as an example...
#
# {{ states                                                        -- all the states (entities) in the system
#    | selectattr('domain','in',['binary_sensor', 'climate', etc.  -- filter only the entities for real devices
#    | selectattr('state', 'in', ['unavailable'])                  -- filter only entities that are unavailable
#    | map(attribute='entity_id')                                  -- get the entity id from the record
#    | map('device_attr', 'name_by_user')                          -- map the entity id onto the device name
#    | reject('match', 'None')                                     -- take out names 'None' (meaning there is no name, so not a device)
#    | unique                                                      -- take out duplicates (devices usually have several entities)
#    | list                                                        -- make into a list (in the template sense)
#    | sort                                                        -- put them in alphabetical order
#    | join('\n')                                                  -- take out extraneous punctuation for a tidy output
#  }}

Great work :ok_hand: I learned a lot.

Some thoughts to share…
Now you have a sensor, but what doing with it?
Showing on a dashboard?

This may lead to the situation, that you (or whoever) have to look regularly on it.
And in the most cases you hopefully see: Everything okay.

But are you really interested in the information that everything is okay? Isn’t it a „waste of time“ to check again and again a status which is fine?

I would guess you are interested in the situation where something is wrong.
Personally I don’t like to get bothered with informations I don’t need or which does not trigger any kind of action.

So for me the logical next step would be to use the sensor you created to trigger an automation which informs me about a problem.
And in case there is no problem… why should I care for a „no-problem“? :wink:

I originally wrote it for a ‘service’ dashboard on which I also display devices with low battery level, the system status and the status of Zigbee repeaters. It is intended for general maintenance and first place to look when funny things happen.

Hi Andy,

Great work, I’ve got a similar one. However, for one sensor state HA has to iterate through all entities three times.

Few options for process optimization:

  • consider adding a trigger to the sensor, default update interval is 5 seconds, I believe, do you really need on-the-fly info, or is once per minute quick enough
  • look into the option of making it a template sensor with an action, then HA only has to iterate once and then put the data in different attributes.
  • I only list a device as unavailable when all device entities are unavailable

Finally, you could change the base to integration_entities, for instance integration_entities('mqtt'), if you’re only interested in mqtt devices.

Kind regards,
- Ingrid

1 Like

Hi @AndySymons,
I took a bit of mine and a bit of yours, and this is the result.
The sensor action also creates a custom event. Which you can use in an automation :slight_smile: .

template:
  - sensor:
      - name: Unavailable devices
        unique_id: s1716371752
        state: >
          {{ attr.device_count }}
        attributes:
          # ========= Entity attributes =========
          # If you don't use the entity_id_list, I would not include them as attributes
          #entity_id_list: >-
          #  {{ attr.entity_id_list }}
          #entity_name_list: >-
          #  {{ attr.entity_name_list }}
          #entity_count: >-
          #  {{ attr.entity_count }}
          # ========= Device attributes =========
          device_id_list: >-
            {{ attr.device_id_list }}
          device_name_list: >-
            {{ attr.device_name_list }}
    trigger:
      - platform: time_pattern
        minutes: "/1"
      - platform: event
        event_type: event_template_reloaded
    action:
      - alias: Generate output
        variables:
          # ========= Global settings ========= #
          entity_id_of_sensor: >-
            {#- in actions you cannot reference this, so you need to set the entity_id -#}
            {{ 'sensor.unavailable_devices' }}
          output: >-
            {#- ========= Settings ========= -#}
            {%- set included_domains = [
              states.binary_sensor, 
              states.button, 
              states.climate, 
              states.light, 
              states.sensor, 
              states.switch
              ] 
            -%}
            {%- set reject_entity_ids = [
              "sensor.sun_next_rising", 
              "sensor.sun_next_noon"
              ] +
              area_entities('system')
            -%}
            {#- When all device entities are either unavailable or unknown, the device will be marked as offline. -#}
            {%- set watched_states = ['unavailable', 'unknown'] -%}
            {#- ========= Find Entites ========= -#}
            {%- set ns = namespace(entity_id_list=[], entity_name_list=[], entity_count=0, device_id_list=[], device_name_list=[], device_count=0) -%}
            {%- set watched_entities = 
              included_domains
              | expand
              | rejectattr('entity_id', 'in', reject_entity_ids)
              | selectattr('state', 'in', watched_states)
            -%}
            {%- for entity in watched_entities -%}
              {%- set ns.entity_id_list = ns.entity_id_list + [entity.entity_id] -%}
              {%- set ns.entity_name_list = ns.entity_name_list + [state_attr(entity.entity_id, 'friendly_name')] -%}
            {%- endfor -%}
            {%- set ns.entity_count = ns.entity_id_list | count -%}
            {#- ========= Find Devices ========= -#}
            {%- set device_ids = 
              ns.entity_id_list
              | map('device_id') 
              | select('ne', None)
              | unique
              | sort
            -%}
            {%- for device_id in device_ids -%}
              {%- if device_entities (device_id) | list | count == expand(device_entities (device_id)) | selectattr('state', 'in', watched_states) | list | count -%}
                {%- set ns.device_id_list = ns.device_id_list + [device_id] -%}
                {%- set ns.device_name_list = ns.device_name_list + [device_attr(device_id, 'name')] -%}
              {%- endif -%}
            {%- endfor -%}
            {%- set ns.device_count = ns.device_id_list | count -%}
            {{ { 'entity_id_list': ns.entity_id_list | list, 'entity_name_list': ns.entity_name_list | list, 'entity_count': ns.entity_count, 'device_id_list': ns.device_id_list | list, 'device_name_list': ns.device_name_list | list, 'device_count': ns.device_count } }}
      - alias: Devices down
        if:
          - condition: template
            value_template: >
              {{ output.device_id_list | reject('in', state_attr(entity_id_of_sensor, 'device_id_list')) | list | count > 0 }}
        then:
          - repeat:
              for_each: "{{ output.device_id_list | reject('in', state_attr(entity_id_of_sensor, 'device_id_list')) | list }}"
              sequence:
                - event: device_availability_changed
                  event_data:
                    device_id: "{{ repeat.item }}"
                    device_name: "{{ device_attr(repeat.item, 'name') }}"
                    new_state: down
      - alias: Devices up
        if:
          - condition: template
            value_template: >
              {{ state_attr(entity_id_of_sensor, 'device_id_list') | reject('in', output.device_id_list) | list | count > 0 }}
        then:
          - repeat:
              for_each: "{{ state_attr(entity_id_of_sensor, 'device_id_list') | reject('in', output.device_id_list) | list }}"
              sequence:
                - event: device_availability_changed
                  event_data:
                    device_id: "{{ repeat.item }}"
                    device_name: "{{ device_attr(repeat.item, 'name') }}"
                    new_state: up
      - alias: Make the new state available for the sensor
        variables:
          attr: >
            {{ output }}

And an example of the automation.

automation:
  - id: "1716378990"
    alias: Device availability notification
    description: Get notifications when a device goes down or back up agian.
    trigger:
      - platform: event
        event_type: device_availability_changed
    condition:
      - alias: If HA startup is more than 3 minutes ago
        condition: template
        value_template: >-
          {{ states('sensor.ha_uptime') | as_datetime | as_local < now() - timedelta(minutes=3) }}
    action:
      - alias: If new state is down
        if:
          - condition: template
            value_template: "{{ trigger.event.data.new_state == 'down' }}"
        then:
          - service: persistent_notification.create
            metadata: {}
            data:
              title: Device offline
              message: >-
                {{ 'Device "' ~ trigger.event.data.device_name ~ '" is no longer available. <br><br><a href="/devices/device/' ~ trigger.event.data.device_id ~ '">More information.</a>' }}
              notification_id: "{{ trigger.event.data.device_id }}"
      - alias: If new state is up
        if:
          - condition: template
            value_template: "{{ trigger.event.data.new_state == 'up' }}"
        then:
          - service: persistent_notification.dismiss
            metadata: {}
            data:
              notification_id: "{{ trigger.event.data.device_id }}"
          - service: notify.persistent_notification
            metadata: {}
            data:
              title: Device was offline for a period of time.
              message: >-
                {{ 'Device "' ~ trigger.event.data.device_name ~ '" was unavailable, but is has been reactivated.' }}
    mode: parallel
    max: 50

Kind regards,
- Ingrid

1 Like
  1. I agree!
  2. I mainly use it for a dashboard, as mentioned above, so the polling approach (1) is aproriate (and could be even less frequent). If I wanted an action, then I would trigger an automation by a change in the state (number ıf devices unavailable) – so a ‘pull’ rather than a ‘push’ approach. For some crucial devices I already have automations with notifications, for example Heating X2: Schedule Thermostats with Calendars notifies when a TRV is unavailable or fails to repond to a setting.
  3. Good idea. I did find the picture was clouded by devices that had unused entities marked as unknown or unavailable, probably due to a faulty ZHA quirk. I got around that by manually disabling the entties is question.
  4. (MQTT) yes, of course, but not applicable in my case

Thanks for sharing! :grinning: