A sensor to list unavailable devices

Great work :ok_hand: I learned a lot.

Some thoughts to share…
Now you have a sensor, but what doing with it?
Showing on a dashboard?

This may lead to the situation, that you (or whoever) have to look regularly on it.
And in the most cases you hopefully see: Everything okay.

But are you really interested in the information that everything is okay? Isn’t it a „waste of time“ to check again and again a status which is fine?

I would guess you are interested in the situation where something is wrong.
Personally I don’t like to get bothered with informations I don’t need or which does not trigger any kind of action.

So for me the logical next step would be to use the sensor you created to trigger an automation which informs me about a problem.
And in case there is no problem… why should I care for a „no-problem“? :wink:

I originally wrote it for a ‘service’ dashboard on which I also display devices with low battery level, the system status and the status of Zigbee repeaters. It is intended for general maintenance and first place to look when funny things happen.

Hi Andy,

Great work, I’ve got a similar one. However, for one sensor state HA has to iterate through all entities three times.

Few options for process optimization:

  • consider adding a trigger to the sensor, default update interval is 5 seconds, I believe, do you really need on-the-fly info, or is once per minute quick enough
  • look into the option of making it a template sensor with an action, then HA only has to iterate once and then put the data in different attributes.
  • I only list a device as unavailable when all device entities are unavailable

Finally, you could change the base to integration_entities, for instance integration_entities('mqtt'), if you’re only interested in mqtt devices.

Kind regards,
- Ingrid

1 Like

Hi @AndySymons,
I took a bit of mine and a bit of yours, and this is the result.
The sensor action also creates a custom event. Which you can use in an automation :slight_smile: .

template:
  - sensor:
      - name: Unavailable devices
        unique_id: s1716371752
        state: >
          {{ attr.device_count }}
        attributes:
          # ========= Entity attributes =========
          # If you don't use the entity_id_list, I would not include them as attributes
          #entity_id_list: >-
          #  {{ attr.entity_id_list }}
          #entity_name_list: >-
          #  {{ attr.entity_name_list }}
          #entity_count: >-
          #  {{ attr.entity_count }}
          # ========= Device attributes =========
          device_id_list: >-
            {{ attr.device_id_list }}
          device_name_list: >-
            {{ attr.device_name_list }}
    trigger:
      - platform: time_pattern
        minutes: "/1"
      - platform: event
        event_type: event_template_reloaded
    action:
      - alias: Generate output
        variables:
          # ========= Global settings ========= #
          entity_id_of_sensor: >-
            {#- in actions you cannot reference this, so you need to set the entity_id -#}
            {{ 'sensor.unavailable_devices' }}
          output: >-
            {#- ========= Settings ========= -#}
            {%- set included_domains = [
              states.binary_sensor, 
              states.button, 
              states.climate, 
              states.light, 
              states.sensor, 
              states.switch
              ] 
            -%}
            {%- set reject_entity_ids = [
              "sensor.sun_next_rising", 
              "sensor.sun_next_noon"
              ] +
              area_entities('system')
            -%}
            {#- When all device entities are either unavailable or unknown, the device will be marked as offline. -#}
            {%- set watched_states = ['unavailable', 'unknown'] -%}
            {#- ========= Find Entites ========= -#}
            {%- set ns = namespace(entity_id_list=[], entity_name_list=[], entity_count=0, device_id_list=[], device_name_list=[], device_count=0) -%}
            {%- set watched_entities = 
              included_domains
              | expand
              | rejectattr('entity_id', 'in', reject_entity_ids)
              | selectattr('state', 'in', watched_states)
            -%}
            {%- for entity in watched_entities -%}
              {%- set ns.entity_id_list = ns.entity_id_list + [entity.entity_id] -%}
              {%- set ns.entity_name_list = ns.entity_name_list + [state_attr(entity.entity_id, 'friendly_name')] -%}
            {%- endfor -%}
            {%- set ns.entity_count = ns.entity_id_list | count -%}
            {#- ========= Find Devices ========= -#}
            {%- set device_ids = 
              ns.entity_id_list
              | map('device_id') 
              | select('ne', None)
              | unique
              | sort
            -%}
            {%- for device_id in device_ids -%}
              {%- if device_entities (device_id) | list | count == expand(device_entities (device_id)) | selectattr('state', 'in', watched_states) | list | count -%}
                {%- set ns.device_id_list = ns.device_id_list + [device_id] -%}
                {%- set ns.device_name_list = ns.device_name_list + [device_attr(device_id, 'name')] -%}
              {%- endif -%}
            {%- endfor -%}
            {%- set ns.device_count = ns.device_id_list | count -%}
            {{ { 'entity_id_list': ns.entity_id_list | list, 'entity_name_list': ns.entity_name_list | list, 'entity_count': ns.entity_count, 'device_id_list': ns.device_id_list | list, 'device_name_list': ns.device_name_list | list, 'device_count': ns.device_count } }}
      - alias: Devices down
        if:
          - condition: template
            value_template: >
              {{ output.device_id_list | reject('in', state_attr(entity_id_of_sensor, 'device_id_list')) | list | count > 0 }}
        then:
          - repeat:
              for_each: "{{ output.device_id_list | reject('in', state_attr(entity_id_of_sensor, 'device_id_list')) | list }}"
              sequence:
                - event: device_availability_changed
                  event_data:
                    device_id: "{{ repeat.item }}"
                    device_name: "{{ device_attr(repeat.item, 'name') }}"
                    new_state: down
      - alias: Devices up
        if:
          - condition: template
            value_template: >
              {{ state_attr(entity_id_of_sensor, 'device_id_list') | reject('in', output.device_id_list) | list | count > 0 }}
        then:
          - repeat:
              for_each: "{{ state_attr(entity_id_of_sensor, 'device_id_list') | reject('in', output.device_id_list) | list }}"
              sequence:
                - event: device_availability_changed
                  event_data:
                    device_id: "{{ repeat.item }}"
                    device_name: "{{ device_attr(repeat.item, 'name') }}"
                    new_state: up
      - alias: Make the new state available for the sensor
        variables:
          attr: >
            {{ output }}

And an example of the automation.

automation:
  - id: "1716378990"
    alias: Device availability notification
    description: Get notifications when a device goes down or back up agian.
    trigger:
      - platform: event
        event_type: device_availability_changed
    condition:
      - alias: If HA startup is more than 3 minutes ago
        condition: template
        value_template: >-
          {{ states('sensor.ha_uptime') | as_datetime | as_local < now() - timedelta(minutes=3) }}
    action:
      - alias: If new state is down
        if:
          - condition: template
            value_template: "{{ trigger.event.data.new_state == 'down' }}"
        then:
          - service: persistent_notification.create
            metadata: {}
            data:
              title: Device offline
              message: >-
                {{ 'Device "' ~ trigger.event.data.device_name ~ '" is no longer available. <br><br><a href="/devices/device/' ~ trigger.event.data.device_id ~ '">More information.</a>' }}
              notification_id: "{{ trigger.event.data.device_id }}"
      - alias: If new state is up
        if:
          - condition: template
            value_template: "{{ trigger.event.data.new_state == 'up' }}"
        then:
          - service: persistent_notification.dismiss
            metadata: {}
            data:
              notification_id: "{{ trigger.event.data.device_id }}"
          - service: notify.persistent_notification
            metadata: {}
            data:
              title: Device was offline for a period of time.
              message: >-
                {{ 'Device "' ~ trigger.event.data.device_name ~ '" was unavailable, but is has been reactivated.' }}
    mode: parallel
    max: 50

Kind regards,
- Ingrid

1 Like
  1. I agree!
  2. I mainly use it for a dashboard, as mentioned above, so the polling approach (1) is aproriate (and could be even less frequent). If I wanted an action, then I would trigger an automation by a change in the state (number ıf devices unavailable) – so a ‘pull’ rather than a ‘push’ approach. For some crucial devices I already have automations with notifications, for example Heating X2: Schedule Thermostats with Calendars notifies when a TRV is unavailable or fails to repond to a setting.
  3. Good idea. I did find the picture was clouded by devices that had unused entities marked as unknown or unavailable, probably due to a faulty ZHA quirk. I got around that by manually disabling the entties is question.
  4. (MQTT) yes, of course, but not applicable in my case

Thanks for sharing! :grinning:

Hi @studioIngrid

Thirst thanks for sharing. I recently wrote a template macro for the same purpose.

The endeavor was not easy, as I encountered various things that should be taken into account.

So reading your code, some improvements to consider:

  • Some devices have a binary connection sensor (device_class connectivity). These remain available,when the device is off line. Your code misses them
  • Some integrations extend devices of other integrations with additional entities. Most of these extended entities will also become unavailable ( I.e a cast device gets extended by e media player, etc). BUT not all!, Example is powercalc, adding power/energy sensors to devices. So I had to exclude powercalc entities from the evaluation.

So I now share my template macro, that takes these exceptions into account. The flow is simple to avoid to much iterations over entities:

  • First I create an list of suspicious device_ids with unavailable entitie(s). (This limits the scope for the remaining code)
  • create an ignore list of entries not to use (powercalc only for now)
  • then I evaluate only the suspicious devices:
    • lookup the device entities and reject ignores
    • check there is at least one unavailable
    • check there are no availables, excluding connection types
  • for every unavailable device, a struct with information is added to a list to return.

I think this is a reasonable efficient approach:

{%- macro unavailable_devices() -%} 
 {#- list all devices that have at least one unavailable entity -#}
 {% set suspicious_devices = states | rejectattr('state','ne','unavailable') | map(attribute='entity_id') | map('device_id') | unique | reject('eq',None) | list -%}
 {%- set ns = namespace(unavailables = []) -%}
 {#- create a list of entities to ignore. 
     i.e. powercalc extends devices with entities. -#}
 {%- set ignore_lst=integration_entities('powercalc') -%}
 {%- for device_id in suspicious_devices -%}
   {%- set ids = device_attr(device_id, 'identifiers') -%}
   {%- set integration = 'unknown' if (not ids or ids|list|length == 0 or ids|list|first|length!=2) else ids|list|first|first -%}
   {#- exlude entities of i.e. powercalc which extends devices with power entities -#}
   {%- set entities= device_entities(device_id) | reject('in', ignore_lst) | list -%}
   {%- set unavailable = expand(entities) | selectattr('state','eq','unavailable') |list|count -%}
   {#- don't count available entities of connection type -#}
   {%- set available = expand(entities) | selectattr('state','ne','unavailable') | list | count -%}
   {#- so count connection entities in error state -#}
   {%- set contypes = expand(entities) | selectattr('state','eq','off') | selectattr('attributes.device_class', 'defined') |selectattr('attributes.device_class','eq','connectivity') | list | count  -%}
   {%- if unavailable!=0 and available-contypes==0  -%}
     {#- preferably use named_by_user -#}
     {%- set name = device_attr(device_id, 'name_by_user') -%}
     {%- set name = device_attr(device_id, 'name') if name==None else name -%}
     {%- set area = device_attr(device_id, 'area_id') or 'n/a' -%}
     {%- set model = device_attr(device_id, 'model') or 'n/a' -%}
     {%- set devices_info = {"name":name,"area":area,"model":model,"id":device_id,"available":available,"unavailable":unavailable,"integration":integration} -%}
     {%- set ns.unavailables = ns.unavailables + [ devices_info ] -%}
   {%- endif -%}
 {%- endfor -%}
 {{ ns.unavailables | to_json }}
{%- endmacro -%}

Usage, note the from_json:

 {{ unavailable_devices()|from_json|count }}

 {{ unavailable_devices()|from_json }}

Result snippet:

2

 [
  {'name': 'Keukenkast', 
   'area': 'keuken', 
   'model': 'Dimmable light (PL 110)', 
   'id': '720b81c19d128a2d4d9e7fa7d3383f61', 
   'available': 0, 
   'unavailable': 1,
   'integration': 'hue'},
  {'name': 'Athom Plug V3 5090e8', 
   'area': 'other',
   'model': 'Smart_Plug_V3', 
   'id': 'a05cbf2d7536349b1caefb4f8e65d4bf', 
   'available': 1,
   'unavailable': 14, 
   'integration': 'unknown'},
]

The second device in this snippet has a connectivity sensor. Therefor available is 1.

The macro can also be used in the markdown card and auto-entities. Example for auto entities & template-entity-row:

    filter:
      template: |
        {%- from 'config.jinja' import unavailable_devices -%}
        {%- set devices = unavailable_devices()|from_json -%}
        {%- set ns=namespace(rows=[]) -%}
        {%- for info in devices -%}
          {%- set sec_info = 'Area:'~info.area~', Mdl:'~info.model  -%}
            {%- set entry = {
              'type': 'custom:template-entity-row',
              'name': info.integration~': '~info.name,
              'state': info.available~'-'~info.unavailable,
              'secondary': sec_info,
              'tap_action': {
                  'action': 'navigate',
                  'navigation_path': '/config/devices/device/'~info.id },
              } -%}
            {%- set ns.rows= ns.rows + [entry] -%}
        {%- endfor -%}  
        {{ ns.rows }}
      include: []
      exclude: []

Result, when you tap it navigates to the device info page:

@AndySymons
For me there are no issues with MQTT or ZHA devices.

1 Like

I started a new architecture discussion for Home Assistant to add two template functions that would tremendously simplify the templates mentioned in this discussion:

  • get_integrations() → list
  • entity_integration(entity_id) → string

I’m new to that process, so lets find out if this proposal gets accepted :crossed_fingers:

2 Likes

This is great. I use the same workaround with identifiers. I liked your discussion post. I hope this will get added!

Kind regards,
- Ingrid

{{ states 
| map(attribute='entity_id') 
| map('device_attr', 'identifiers') 
| select('ne', None)
| map('first')
| select('ne', undefined)
| map('first')
| unique
| sort }}

1 Like

Thanks for your vote!

I have been unsuccessful in implementing this for myself and I was hoping you could point me in the right direction. I’ve added the macro to a file here: /homeassistant/custom_templates/tools.jinja . When I attempt to call the macro by entering {{ unavailable_devices()|from_json }} in the developer tools template area I get “‘unavailable_devices’ is undefined”. I’ve restarted Home Assistant. It’s probably something stupid but I am trying to learn more about advanced templating so any help is appreciated.

If you are talking about my version then it is a YAML file, not JSON. I put it in /config/templates.yaml (create it if you don’t have it already) .

In /config/configuration.yaml include the line

template: !include templates.yaml

You then need to reload the configuration to activate it. Clear the log file first and if it does not work check the log file afterwards.

To use macro 's one needs to import them:

{% for ‘tools.jinja’ import unavailable_devices %}
{{ unavailable_devices()}}

Sorry, I have not used macros in HA and don’t know anything about them. Why do you want to?

This allows to use the same template code on multiple places in you cards and template sensors

1 Like

Thanks for responding! Your solution with the auto-entities card and custom-template-entity-row was exactly what I was trying to achieve: getting only the device (no repeat entities) that was unavailable into a dynamic dashboard card and have the tap action go to that device page. I was able to get everything working.
Firstly, I was getting errors initially because I was accidentally commenting out a necessary line when editing the ignore_list portion of the macro. After figuring that out, I just needed to edit your macro call in the developer tools area to this:

{% from 'tools.jinja' import unavailable_devices %}
{{ unavailable_devices()|from_json }}
{{ unavailable_devices()|from_json|count }}

to make the code give me any results. I’ve now got the template working in an auto-entities card. Thanks again!

2 Likes

I have a question, can you have a group of entities that you use as an exclusion group to this sensor?

You can do it with a filter like this where exclusions is a list (e.g. an expanded group):

| rejectattr('entity_id', 'in', exclusions)

I like this solution.

Unfortunately I do not get the expected result from integration_entities.
For example HACS creates an update sensor for each custom integration, which has the value “unavailable” when no updates are available. Yet,

{{ integration_entities(‘hacs’) }}

returns just [‘sensor.hacs’] despite the UI showing more entities under HACS. All unavailable except the single one returned.

Same with UniFi.

If integration_entities does not report unavailable entities, it may not not be a good method of excluding devices.

@Jan4 You are right, integration_entities() is not returning all entities. I created an issue a while ago for that

when i paste this into my templates.yaml file i get the following

if i remove the template: portion i no longer see any errors but when i restart home assistant and check for an entity called sensor.unavailable_devices i cant find it.

here is what i get in my logs when i restart HA

Logger: homeassistant.config
Source: config.py:592
First occurred: 10:21:00 AM (1 occurrences)
Last logged: 10:21:00 AM

Invalid config for ‘template’ at templates.yaml, line 70: ‘sensor’ is an invalid option for ‘template’, check: sensor->1->sensor Invalid config for ‘template’ at templates.yaml, line 70: required key ‘state’ not provided