A sensor to list unavailable devices

I want to make a user-friendly list of devices that are offline so that a householder who is not particularly technically minded can find them and check that they are plugged in and connected to WiFi or ZigBee as applicable.

I found several postings on this forum and elsewhere, mostly focused on entities that are unavailable. That is not what I want because
a) Each device typically has several entities and I don’t want the duplicates
b) Sometimes some entities are unavailable for a device that is actually working, probably due to an obscure device configuration (quirk) error.
c) Entities can include items like automations that are not devices at all. I only want the real devices (what the householder can see).

I don’t really know what I am doing (!) but by experimentation and borrowing from all over (thanks) I have cobbled together a sensor that delivers in attributes (because the state cannot exceed 255 characters) a list of entity ids, a list of entity friendly names and a list of devices by user-defined device name.

All three work to the extent that they produce a list using the correct name. However, if I then look at the listed devices or entities in the UI, I find that they are not all unavailable. Apparently the filtering is not yet correct.

template:
  - sensor:
      - name: "Unavailable Entities"
        unique_id: unavailable_entities
        icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
        state_class: measurement
        unit_of_measurement: entities
        state: >
          {% set entities_list = state_attr(this.entity_id,'entity_id_list') %}
          {{ entities_list | count if entities_list != none else none }} 

        # these results have to be attributes because the state cannot contain more than 255 characters 

        attributes:

          entity_id_list: >-
            {% set entities_list = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) %}
            {% set entities_list = entities_list | rejectattr('entity_id','in',ignored) if ignored != none else entities_list %}
            {% set entities_list = entities_list | map(attribute='entity_id')|reject('has_value')|list|sort %}
            {{ entities_list }}

          entity_name_list: >-
            {% set entities_list = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) %}
            {% set entities_list = entities_list | rejectattr('entity_id','in',ignored) if ignored != none else entities_list %}
            {% set entities_list = entities_list | map(attribute='entity_id')|reject('has_value')|list|sort %}
            
            {% set data = namespace(enamelist=[]) %}
            {% for entity_id in entities_list %}
            {% set entity_name = state_attr(entity_id,'friendly_name') | string %}
              {% if not entity_name in data.enamelist %}
                {% set data.enamelist = data.enamelist + [ entity_name ] %}
              {% endif %}
            {% endfor %}
            {{ data.enamelist | sort | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") }} 

          device_name_list: >-
            {% set device_list = states
              | selectattr('state', 'in', ['unavailable', 'unknown'])
              | map(attribute='entity_id')
              | map("device_id")
              | map("string")
              | unique
              | list
              | sort
            %}
            {% set data = namespace(dnamelist=[]) %}
            {% for device_id in device_list %}
              {% set device_name = device_attr(device_id, 'name_by_user') | string %}
              {% if device_name == "None" %}  
              {% elif device_name in data.dnamelist %}
              {% else %}
                {% set data.dnamelist = data.dnamelist + [ device_name ] %}
              {% endif %}
            {% endfor %}
            {{ data.dnamelist | sort | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") }} 

I would like to know

  1. How to distinguish the main entity for a device (the device itself)?
  2. How to list all the DEVICES on the system?
  3. How to filter on unavailability of the device itself, not just some of its entities?
  4. What exactly the ‘map’ function does?

Not all entities are related to a device.

Basically it lets you avoid having to use an explicit loop and namespace to perform two basic object manipulation tasks. The filter map() can be used to apply a filter on a sequence of objects, or look up a given attribute from a sequence of objects.

image

Jinja Docs - map() filter

1 Like

This is a similar idea:
https://community.home-assistant.io/t/detecting-unresponsive-devices/658030

Maybe it‘s helpful.

1 Like

Hi,

You could also use a custom card called auto-entities. See here.

I use this card config in my system.

type: custom:auto-entities
card:
  type: entities
  show_header_toggle: false
  state_color: true
  title: Unavailable Entities
filter:
  include:
    - domain: light
      state: unavailable
    - domain: switch
      state: unavailable
    - domain: fan
      state: unavailable
    - domain: cover
      state: unavailable
    - domain: binary_sensor
      state: unavailable
    - domain: sensor
      state: unavailable
  exclude:
    - entity_id: '*_update_available'
sort:
  method: friendly_name
1 Like

That is precisely the problem (questions 1 and 2). How do I filter on the entities that are devices – i.e. the MAIN entity for the device?

I read the developer doc but cannot quite yet make the link to a filter

I have tried that but it lists entities rather than devices. It does have the possibility to create the entity list using a template, so if only I knew how to write that filter, then using this card might be simpler than using a template sensor.

For example, part of the list is

Guest bedroom TRV
Guest bedroom TRV Away preset temperature
Guest bedroom TRV Battery
Guest bedroom TRV Calibrated
Guest bedroom TRV Child lock
Guest bedroom TRV External sensor
Guest bedroom TRV HVAC action
Guest bedroom TRV Preset
Guest bedroom TRV Valve alarm
Guest bedroom TRV Valve detection
Guest bedroom TRV Window detection
Guest bedroom TRV Window open

I want it to just say …

Guest bedroom TRV

… because that is what the householder will understand. Probably the TRV battery is flat or it has lost contact with the ZigBee network.

BTW, making a list of unavailable entities, getting the device name then filtering the duplicates is what I do in the example code above. That seems to be not quite correct because it reports a device as unavailable if any one of its entities is unavailable, and that is not always the correct. I need to get the list of entities that are actual devices, then select those that are unavailable.

I made an updated version of the code today that is an improvement, though not quite perfect!

I still do not know how to filter the ‘main’ device entities from a list of entities and perhaps it is not even possible? If I could do that, I would follow my preferred algorithm:

  1. List all entities
  2. Select only those that are (definitely) the main one for the devices
  3. Select from these those that are unavailable (or possibly, unknown)
  4. Retrieve the device user-assigned names for the entities on this list

However, in the meantime, I am working with the algorithm:

  1. List all entities
  2. Select those entities that are ‘unavailable’
  3. Select those that are (probably) connected with real devices by using the domain (binary_sensor, climate etc.)
  4. Retrieve the device user-assigned name from the device attributes associated with the entity_id
  5. If it is not “None”, add it to the list. I guess that eliminates devices that have a default name, but in my case they are all named by hand when they are on-boarded.

An important change for me was to only list ‘unavailable’ entities and not ‘unknown’ ones. That is because even available and working devices can have entities that are ‘unknown’ and with the present algorithm they would appear incorrectly on the unavailable devices list. For example, I have some TRV’s with the entity ‘HVAC action’ = ‘unknown’. I don’t know why, perhaps a hardware fault, perhaps a fault in a quirk, but the TRV is online and working so I want to ignore it for the purposes of this sensor.

##----------------------------------------------------------------------------------------------------------------------
##
## Unavailable Devices Sensor
## 
## 05-Feb-2024 | Andy Symons | based loosely on https://github.com/jazzyisj/unavailable-entities-sensor/blob/main/README.md
## 
##----------------------------------------------------------------------------------------------------------------------

template:
    - sensor:
        name: "Unavailable Entities"
        unique_id: unavailable_entities
        icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
        state_class: measurement
        unit_of_measurement: entities
        state: >
            {% set entities_list = states
                | selectattr ('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                | selectattr('state', 'in', ['unavailable'])
                | map(attribute='entity_id')
                | list
            %}
            {{ entities_list | count }}

        # these results have to be attributes because the state cannot contain more than 255 characters 

        attributes:

            entity_id_list: >-
                {% set entities_list = states
                    | selectattr ('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | list
                %}
                {% set data = namespace(eidlist=[]) %}
                {% for entity_id in entities_list %}
                    {% set data.eidlist = data.eidlist + [ entity_id | string ] %}
                {% endfor %}
                {{ data.eidlist | sort | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") }} 

            entity_name_list: >-
                {% set entities_list = states
                    | selectattr ('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | list
                %} 
                {% set data = namespace(enamelist=[]) %}
                {% for entity_id in entities_list %}
                    {% set data.enamelist = data.enamelist + [ state_attr(entity_id,'friendly_name')  ] %}
                {% endfor %}
                {{ data.enamelist | sort | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") }} 

            device_name_list: >-
                {% set entities_list = states
                    | selectattr ('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
                    | selectattr('state', 'in', ['unavailable'])
                    | map(attribute='entity_id')
                    | list
                %} 
                {% set data = namespace(dnamelist=[]) %}
                {% for entity_id in entities_list %}
                    {% set device_name = device_attr(entity_id, 'name_by_user') | string %}
                        {% if not device_name == "None" %} 
                        {% set data.dnamelist = data.dnamelist + [ device_name  ] %}
                    {% endif %}
                {% endfor %}
                {{ data.dnamelist | unique | sort | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") }} 



I display it on a dashboard using a markdown card…

type: vertical-stack
cards:
  - type: custom:mushroom-title-card
    title: Devices that are currently unavailable
    subtitle: They have at least one entity that is unavailable
  - type: markdown
    content: '{{ state_attr( ''sensor.unavailable_entities'' , ''device_name_list'' )  }}'

It looks like this.

Screenshot 2024-02-05 at 13.14.11

I checked the output by finding each device in the Devices page of the UI and confirmed that they really are unavailable.

For comparison (and test purposes) these are the first part of the much longer listings from the other two attributes, entity_id_list and entity_name_list.

The number of unavailable entities is the State of the template sensor.
Attributes are used for the main output lists because in Home Assistant the State cannot be longer than 255 characters.

type: vertical-stack
cards:
  - type: custom:mushroom-title-card
    title: Entities that are curently offline
    subtitle: (by name)
  - type: entities
    entities:
      - entity: sensor.unavailable_entities
  - type: markdown
    content: '{{ state_attr( ''sensor.unavailable_entities'' , ''entity_name_list'' )  }}'

type: vertical-stack
cards:
  - type: custom:mushroom-title-card
    title: Entities that are curently offline
    subtitle: (by entity_id)
  - type: entities
    entities:
      - entity: sensor.unavailable_entities
  - type: markdown
    content: '{{ state_attr( ''sensor.unavailable_entities'' , ''entity_id_list'' )  }}'

Let me know what you think!

1 Like

What I do is for each device I create a binary_sensor that tells me if it is available. This allows tailoring by device - for example require it is unavailable for 30 minutes. I can then send alerts when they are unavailable for a while. It’s a simple, low tech way to accomplish.

Use a naming convention and it’s very easy to use auto-entities to create the list.

1 Like

Add this line to your existing template:

  | map('device_attr', 'name') | unique

Example

{{ states
  | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
  | selectattr('state', 'in', ['unavailable'])
  | map(attribute='entity_id')
  | map('device_attr', 'name') | unique
  | list }}

Reference

From Templating - Devices:

1 Like

Intriguing. Can you explain a bit what ‘device_attr’ is? Is it an attribute of State that is only set to a name when the entity is the main one for the device??

I am aware of the document you quoted but there is nothing there to say, and nothing in my experiments to show, that device attributes are not available from any entity associated with the device. Filtering with them does not therefore reduce the list. I need a test that identifies exactly one instance of States (an entity) for each physical device that shows the availability of the device as a whole. It is equivalent to searching for Devices rather than Entities in the UI.

See my previous post.

Updated in parallel to your post! :grinning:

I am aware of the document you quoted but there is nothing there to say, and nothing in my experiments to show, that device attributes are not available from any entity associated with the device. Filtering with them does not therefore reduce the list. I need a test that identifies exactly one instance of States (an entity) for each physical device that shows the availability of the device as a whole. It is equivalent to searching for Devices rather than Entities in the UI.

Many thanks. I think I get it now! You have given me a more elegant way of programming my template. With the modification of using the ‘name_by_user’ and adding a filter to remove the None entries, the following produces the same result as my original code above for the device name list

{{ states
  | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
  | selectattr('state', 'in', ['unavailable'])
  | map(attribute='entity_id')
  | map('device_attr', 'name_by_user') 
  | reject('match', 'None')
  | unique
  | list 
  | sort 
  | replace("'","") | replace("[","") | replace("]","") | replace(", ","\n") 
}}

Many thanks :+1:

You miss understood me.
There are physical devices that do not have any devices in HA, but just a list of entities.

I do not think it is possible to get the Device name programmatically.
In the Spook documentation it has this advice to find a device ID:

Not sure what the device_id of an your device is? There are a few ways to find it:

Use this service in the developer tools, in the UI select the device you want to add and
select the **Go to YAML mode** button. This will show you the device ID in the YAML code.

Alternatively, you can visit the device page in the UI and look at the URL. The device ID is
the last part of the URL, and will look something like this: dc23e666e6100f184e642a0ac345d3eb.

Sorry, yes I did, although I am not aware of any such devices in my installation. If you scan the entity lists above, it is easy to see which one is the main one. I can find all my devices in the Devices page of the UI.

The code I am now using gives the result I want at the moment – the device name give by the user.

My only concern with this code is that there is a theoretical possibility that one of the sub-entities of a device will go unavailable while the main entity (an the device itself) is really still available. That would put the device on the list erroneously. Whether that actually happens in practice remains to be seen! I have not tested it by turning all devices off yet.

You’re welcome.

About that last line in your template, doing multiple replaces, what’s that all about?

None of my devices have a name_by_user attribute (it’s not one of the usual device attributes) so I can’t replicate the result that required you to do all of that replacement. I’m guessing you got a list, or list of lists, or something like that, and you are trying to ‘flatten’ it.

Without that line the output is a raw list…

[
  "Back door closure sensor",
  "Conservatory inner door",
  "Guest bedroom TRV",
  "Landing socket",
  "Sophias bedroom east TRV",
  "Studio motion sensor",
  "Studio temperature and humidity"
]

That line just tidies it up by removing the unwanted characters, so we get:

Back door closure sensor
Conservatory inner door
Guest bedroom TRV
Landing socket
Sophias bedroom east TRV
Studio motion sensor
Studio temperature and humidity

Of course, you can change name_by_user back to name, if that is what you want to report. You then get the system name for the devices, which in my case is:

_TZ3000_msl6wxk9 TS0202
_TZ3000_xr3htd96 TS0201
eWeLink WB01
IKEA of Sweden TRADFRI control outlet
LUMI lumi.airrtc.agl001
LUMI lumi.sensor_magnet.aq2

In my case the list is intended for a householder with little technical knowledge. If I tell him lumi.airrtc.agl001 is offline and needs attention, that will mean nothing to him. He cannot find it easily by searching Devices in the UI, either. If, however I say that Guest bedroom TRV is offline and needs attention, he can go straight to it, check the batteries and reconnect it to ZigBee if necessary. It is easy to find in the UI under that name.

Note too that the system name is not unique. In my case, Back door closure sensor, Conservatory inner door and about 20 similar devices have the name LUMI lumi.sensor_magnet.aq2, so reporting that that ‘device’ is unavailable is not very helpful, even if you recognise it as a closure sensor.

Use join to convert the list of strings to strings delimited by newlines.

{{ states
  | selectattr('domain','in',['binary_sensor', 'climate', 'light', 'sensor', 'switch'])
  | selectattr('state', 'in', ['unavailable'])
  | map(attribute='entity_id')
  | map('device_attr', 'name_by_user') 
  | reject('match', 'None')
  | unique
  | list 
  | sort
  | join('\n')
}}

Yep! That’s another improvement! Thanks again :bouquet: