Unavailable / Unknown Entity Monitoring - Template Sensor

[jazzyisj (Jason)]

A group can only contains entities, not devices. My advice would be to exclude each device in your template as described in the readme.

[Mar1us]

ah ok - thank you!

[e-raser]

Someone ever ran into this issue (ignored entities group: maximum number of entities)?

Groups: maximum number of entities

Do groups have a maximum number of entities? I couldn’t find any at Group - Home Assistant Asking as I have this: group: problems_any_ignored: name: Problematische Entitäten (ignorierte) icon: mdi:debug-step-over entities: - ... - ... - ... with many entities. {{ expand('group.problems_any_ignored') | length }} gives: 128. …and that looks like a potential binary system limit to me. When adding more entities and reloading template sensors, the newly added on…

[jazzyisj (Jason)]

I have not run into that specific issue. Mobile, so I can’t test to confirm.

But as an immediate solution, are there common elements between some of your entities such as integration, naming, schema, etc. that you could use to filter them using other methods as described in the readme?

[e-raser]

Yes I could. But it seems like there’s another issue. The group had 131 elements before the last HA Core restart, then it went down to 128. So I need to investigate that.

[jazzyisj (Jason)]

Please update if you find anything out. Seems like something that should have been mentioned in the release notes if there was indeed a change.

[e-raser]

Solved by a simple restart, see Groups: maximum number of entities - #3 by e-raser

I did not update HA Core and I’m sure something like that would be mentioned in the breaking changes section. Sorry to bother you here.

Anyway I hope the topic will get an answer (in case IF there is a real limitation).

[itcsburnett (Steve)]

I’m struggling to get this to work and iterate only offline devices in the ZHA & ZWave_JS integrations. As those are mesh networks, they are really the only ones I am concerned about in my setup (I was troubleshooting a flakey ZHA network, and it turned out a tuya repeater had died).

Reading the docs, I came across that we can select just the entities from specific integrations. I can get it to work for one integration, but not both zha & zwave.

Here’s what works:

###################################################################################################
## Package - Unavailable Entities Sensor
## Count and list entities with a state of unknown or unavailable
## Home Assistant v2022.5 required. See README for customization options.
## https://github.com/jazzyisj/unavailable-entities-sensor/blob/main/README.md
###################################################################################################

# REQUIRED - This is the template sensor

template:
  - sensor:
      - name: "Unavailable Entities"
        unique_id: unavailable_entities
        icon: "{{ iif(states(this.entity_id)|int(-1) > 0,'mdi:alert-circle','mdi:check-circle') }}"
        state_class: measurement
        unit_of_measurement: entities
        state: >
          {% set entities = state_attr(this.entity_id,'entity_id') %}
          {{ entities|count if entities != none else none }}
        attributes:
          entity_id: >
            {% set ignore_seconds = 60 %}
            {% set ignored = state_attr('group.ignored_unavailable_entities','entity_id') %}
            {% set ignore_ts = (now().timestamp() - ignore_seconds)|as_datetime %}
            {% set entities = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) 
                |selectattr('entity_id','in',integration_entities('zwave_js'))
            {% set entities =  entities|rejectattr('entity_id','in',ignored) if ignored != none else entities %}
            {{ entities|map(attribute='entity_id')|reject('has_value')|list|sort }}


# OPTIONAL - Uncomment and add entities you want to ignore to this group.

group:
  ignored_unavailable_entities:
     entities:
      - sensor.1e0de372_f503aec8

I’ve tried adding zha in the same selectattr statement and get UNAVAILABLE:

        attributes:
          entity_id: >
            {% set ignore_seconds = 60 %}
            {% set ignored = state_attr('group.ignored_unavailable_entities','entity_id') %}
            {% set ignore_ts = (now().timestamp() - ignore_seconds)|as_datetime %}
            {% set entities = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) 
                |selectattr('entity_id','in',integration_entities('zwave_js','zha')) %}
            {% set entities =  entities|rejectattr('entity_id','in',ignored) if ignored != none else entities %}
            {{ entities|map(attribute='entity_id')|reject('has_value')|list|sort }}

then I tried just adding a separate selectattr line with zha:

        attributes:
          entity_id: >
            {% set ignore_seconds = 60 %}
            {% set ignored = state_attr('group.ignored_unavailable_entities','entity_id') %}
            {% set ignore_ts = (now().timestamp() - ignore_seconds)|as_datetime %}
            {% set entities = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) 
                |selectattr('entity_id','in',integration_entities('zwave_js'))
                |selectattr('entity_id','in',integration_entities('zha'))%}

Now I get 0 (zero) unavailable

Id also prefer to display the Device name rather than all the entities of the unavailable device if possible

[ktdt00]

I read through the thread and didn’t see it mentioned but if you’re using the code from the readme to display unavailable entities in the “More Info”, and you also want the default persistent notification to work too, you’ll also have to make the entity_id changes in the automation (or at least I did ) YMMMV

[mterry63 (Martin)]

@jazzyisj I also was interested if this template sensor could be modified to ignore entities of disabled devices after taking down my Patio Roku TV for the weather and disabling the device. After a bit of digging and testing, I can see that the core.entity_registry file marks entities as disabled by “config_entry” when the parent device is disabled.

Since this file is JSON, I worked out a command line sensor that uses jq to filter by entities that are marked as disabled by “config_entry”. Additionally, the jq filter has to flatten the resulting JSON to make the array of entities appear in the root level as command line doesn’t support specifying a path to pull data from.

Here is the resulting command line sensor configuration. I’m no JSON or jq expert, so the jq filter may easily be improved, but this sensor configuration has the count of disabled entities as the state, and an array of entity names as an attribute.

command_line:
  sensor:
      name: Disabled Device Entities
      unique_id: amn_disabled_device_entities
      json_attributes:
        - entities
      value_template: >
        {{ value_json.entities | length }}
      command: 'jq ''.data.entities |= map(select(.disabled_by? == "config_entry") | {name: .entity_id}) | del(.data.deleted_entities) | flatten | .[3]'' < .storage/core.entity_registry'

Adding this to configuration.yaml and reloading command line sensors results in this:

Each name is accessible via state_attr:

{{ state_attr('sensor.disabled_device_entities','entities')[0].name }}

Next steps are to incorporate checking these names in the template to ignore any matches.

This is kind of a hack, but I think it will work.

[bacco007]

@mterry63

command: 'jq ''.data.entities |= map(select(.disabled_by? == "config_entry") | {name: .entity_id}) | del(.data.deleted_entities) | flatten | .[3]'' < .storage/core.entity_registry'

I’ve extended this to add what integration the entity relates to, and to include all reasons for disabled_by - which might be assistance for someone else

'jq ''.data.entities |= map(select(.disabled_by? != null) | {name: .entity_id, platform: .platform, disabled_by: .disabled_by}) | del(.data.deleted_entities) | flatten | .[3]'' < .storage/core.entity_registry'

[mterry63 (Martin)]

Following up to completion, I modified the sensor in packages_unavailable_entities.yaml as follows:

under: 
{% set ignored = state_attr('group.ignored_unavailable_entities','entity_id') %}

I added:
{% set ignored2 = state_attr('sensor.disabled_device_entities', 'entities')|regex_replace(find='\[|\]|\{|\}|\'name\':', replace='') %}

and under :
{% set entities =  entities|rejectattr('entity_id','in',ignored)
    if ignored != none else entities %}

I added:
{% set entities =  entities|rejectattr('entity_id','in',ignored2)
    if ignored2 != none else entities %}

Checked the configuration, reloaded the template sensors and now my entities of disabled devices are ignored by the sensor.

I’m sure this could be improved but I’m pretty new at Jinja2 and Python.

[jazzyisj (Jason)]

            {% set entities = states
                |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
                |rejectattr('last_changed','ge',ignore_ts) 
                |selectattr('entity_id','in',integration_entities('zwave_js'))
                |selectattr('entity_id','in',integration_entities('zha'))%}

Template filters are progressive so in the first selectattr filter you have told the template to select only entities in the zwave integration so the second selectattr is pointless becaue the zha integration entities were filtered out by the first selectattr statement.

Set the entities for each integration into list object variables and add them together to use in the filter.

{% set ignore_seconds = 60 %}
{% set ignored = state_attr('group.ignored_unavailable_entities', 'entity_id') %}
{% set ignore_ts = (now().timestamp() - ignore_seconds) | as_datetime %}
{% set zwave = states | selectattr('entity_id','in',integration_entities('zwave_js')) | list %}
{% set zha = states | selectattr('entity_id','in',integration_entities('zha')) | list %}
{% set entities = (zwave + zha) 
    |rejectattr('domain','in',['button','event','group','input_button','input_text','scene'])
    |rejectattr('last_changed','ge',ignore_ts) %}
{% set entities =  entities | rejectattr('entity_id', 'in', ignored) if ignored != none else entities %}
{{ entities | map(attribute='entity_id') | reject('has_value') | list | sort }}  

Make sure zha is the proper slug. I don’t use it so I can’t check.
Open the integration in Settings → Devices. It will be the last part of the url.

[jazzyisj (Jason)]

This is interesting. When I get a minute I’ll have a good look at this, I might add an example to the documentation.

[jazzyisj (Jason)]

The README on the git gives a detailed explanation how to accomplish this.

But here ya go…

            {% set entities = states
                | rejectattr('domain', 'in', ['button', 'conversation', 'event', 'group', 'image', 'input_button', 'input_text', 'remote', 'tts', 'scene', 'select', 'stt', 'automation', 'script', 'notify', 'update', 'number'])
                | rejectattr('entity_id', 'search', 'octoprint')
                | rejectattr('last_changed', 'ge', ignore_ts) %}

However, if all those entities belong to one device, t might be more robust to just ignore the entire device.

[wtstreetglow (Weston)]

Yeah, I ended up seeing that and deleted my post. I apologize for that. Thank you for responding though!!

[sterlo (Brent Sterling)]

Hi Jason,

Thank you for the template and the detailed documentation. I have the sensor working but I have a lot of unavailable entities for various reasons. I think it would be easier instead to provide a nice clean list of entities that I really want it to monitor. It would be great if I could maintain that list in a group much like the ignored_unavailable_entities group. Your documentation clearly shows examples of how I can include various entities via the selectattr pattern matches (like the Shelly and _power examples). However, I want to figure out how to provide it with a list of 10 or so entities I care about and the only other criteria I’d want is the 60 seconds of being unavailable.

I’m far from a YAML expert but was hoping this modification to your template might work but clearly I’ve done something wrong. Any chance you can point me in the right direction?

template:
  - sensor:
      - name: "Unavailable Entities"
        unique_id: unavailable_entities
        icon: "{{ iif(states(this.entity_id)|int(-1) > 0, 'mdi:alert-circle', 'mdi:check-circle') }}"
        state_class: measurement
        unit_of_measurement: entities
        state: >
          {% set entities = state_attr(this.entity_id, 'entity_id') %}
          {{ entities | count if entities != none else none }}
        attributes:
          entity_id: >
            {% set ignore_seconds = 60 %}
            {% set ignored = state_attr('group.ignored_unavailable_entities', 'entity_id') %}
            {% set ignore_ts = (now().timestamp() - ignore_seconds)|as_datetime %}
            {% set entities = state_attr('group.monitored_unavailable_entities','entity_id')**
                | rejectattr('last_changed', 'ge', ignore_ts) %}
            {% set entities =  entities | rejectattr('entity_id', 'in', ignored) if ignored != none else entities %}
            {{ entities | map(attribute='entity_id') | reject('has_value') | list | sort }}

group:
  monitored_unavailable_entities:
    entities:
      - switch.main_water_valve
      - switch.illumino_wall_switch
      - switch.master_bedroom_accent_lights

I can’t figure out how to highlight my primary change above but I tried changing the “set entities =” row under attributes: entity_id: >

[jazzyisj (Jason)]

Change the first set entities line to this

{{ expand('group.monitored_unavailable_entities')
    | rejectattr('last_changed', 'ge', ignore_ts)
    | map(attribute='entity_id') | reject('has_value') | list | sort }}

Remove these lines.

{% set ignored = state_attr('group.ignored_unavailable_entities', 'entity_id') %}
{% set entities =  entities | rejectattr('entity_id', 'in', ignored) if ignored != none else entities %}
{{ entities | map(attribute='entity_id') | reject('has_value') | list | sort }}

[sterlo (Brent Sterling)]

Fantastic. Thank you for taking the time to help with this!