Offline detection for Z2M devices with last_seen

jarrah · April 22, 2022, 7:54pm

Ok, hopefully the following will help other users enable the _last_seen entity without having to manually enable them all.

After a discussion with Koenkk here, he said that setting the following in Z2M doesn’t by itself automatically make hidden _last_seen entities become enabled because “home assistant doesn’t update this after initial discovery”.

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

The solution is to basically remove the MQTT integration and have HA discover all the Z2M entities again. He did however say that “this will reset all your entities in HA (icons, friendly names)”. This doesn’t luckily impact me, so I went ahead and did the following:

stop Z2M
remove MQTT integration in HA
restart HA
confirm devices/entities have been removed in HA
start Z2M
add and reconfigure MQTT integration in HA
restart HA

I now have all 125 _last_seen entities in HA!

btw, if anyone wants to test part of the blueprint in the Template Editor, I was using the following:

{% set result = namespace(sensors=[]) %}
{% for state in states.sensor | selectattr('attributes.device_class', '==', 'timestamp') %}
{% if 'last_seen' in state.entity_id and (states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id))) > ((24 | int) * 60 * 60))) %}
{% set result.sensors = result.sensors + [state.name | regex_replace(find='_last_seen', replace='') ~ ' (' ~ relative_time(strptime(states(state.entity_id), '%Y-%m-%dT%H:%M:%S%z', 'unavailable')) ~ ')'] %}
{% endif %}
{% endfor %}
{{ result.sensors | join(', ') }}

@Mr_Groch it looks like you’re missing the underscores in the line below:

regex_replace(find=' last seen',

Mr_Groch · April 22, 2022, 8:50pm

Not really - this is for grabbing device name for notify - I’m using ‘last seen’ entity with name like ‘device last seen’, so I want to cut ’ last seen’ part from it (notice that regex_replace is done on state.name not state.entity_id)

jarrah · April 23, 2022, 8:42am

Interesting @Mr_Groch - it could be something to do with my testing via the template editor because the result without underscores looks like this:

If I then use find='_last_seen' it looks like this:

It’s also worth noting the lack of a timestamp when a device is unavailable due to having set availability: true in Z2M. Just mentioning it in case it helps anyone else who sees the same thing.

I’ve now disabled availability: true in HA so that timestamps are retained (plus this is the result using _last_seen):

jarrah · April 23, 2022, 9:07am

I’m intrigued as to what the difference is between our entities that would make your name have spaces because mine looks like this:

Maybe it’s because I don’t set a custom name so it uses the entity name by default?

Mr_Groch · April 23, 2022, 12:52pm

Something is wrong with your setup, because your devices have name/friendly name the same as entity_id. Default behaviour in Z2M is to add ‘last seen’ (and other sufixes) with spaces in name/friendly name

jarrah · April 23, 2022, 10:15pm

Z2M has so many options and legacy hangovers that it doesn’t surprise me that my setup is probably missing something. I’ll have a look through the docs to see if I can find something related.

iz3man · July 6, 2022, 1:16pm

Tried to use this blueprint, but it doesn’t show any sensors when run. I already set the options and manually enabled LQI and last_seen

And when I run the code in template editor:

{% set result = namespace(sensors=[]) %}
{% for state in states.sensor | selectattr('attributes.device_class', '==', 'timestamp') %}
{% if 'last_seen' in state.entity_id and (states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id))) > ((24 | int) * 60 * 60))) %}
{% set result.sensors = result.sensors + [state.name | regex_replace(find='_last_seen', replace='') ~ ' (' ~ relative_time(strptime(states(state.entity_id), '%Y-%m-%dT%H:%M:%S%z', 'unavailable')) ~ ')'] %}
{% endif %}
{% endfor %}
{{ result.sensors | join(', ') }}

it shows me this error:

UndefinedError: 'homeassistant.util.read_only_dict.ReadOnlyDict object' has no attribute 'device_class'

jarrah · July 12, 2022, 7:39pm

Same error for me too. I suspect something changed within Zigbee2MQTT that no longer uses timestamp as a device class.

I’ve ended up using this alternative sensor to detect missing devices which has been working well for me:

template:
  - trigger:
      - platform: time_pattern
        hours: "/1"
        minutes: 0
    sensor:
      - unique_id: z2m_last_seen_entities
        name: "Z2M Last Seen Entities"
        state: >
          {% set lapsed_hours = 36 %}
          {% set ns = namespace(count=0) %}
          {% for state in states.sensor | selectattr('entity_id', 'search', '.*_last_seen$')  %}
            {% if states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id),0)) > ((lapsed_hours | int) * 60 * 60)) %}
              {% set ns.count = ns.count + 1 %}
            {% endif %}
          {% endfor %}
          {{ ns.count }}     
        attributes:
          devices: >
            {% set lapsed_hours = 36 %}
            {% set result = namespace(sensors=[]) %}
            {% for state in states.sensor | selectattr('entity_id', 'search', '.*_last_seen$') %}
              {% if states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id),0)) > ((lapsed_hours | int) * 60 * 60)) %}
                {% set result.sensors = result.sensors + [state.name | regex_replace(find='_last_seen', replace='') ~ ' (' ~ relative_time(strptime(states(state.entity_id), '%Y-%m-%dT%H:%M:%S%z', 'unavailable')) ~ ')'] %}
              {% endif %}
            {% endfor %}
            {{ result.sensors }}

Along with this automation to report it:

automation:
  - alias: Offline Zigbee Devices
    id: offline_zigbee_devices
    description: Sends notification for offline Z2m devices
    trigger:
      - platform: time
        at: '20:00'
    condition:
      - condition: template
        value_template: '{{states(''sensor.z2m_last_seen_entities'')|int > 0}}'
    action:
      - service: notify.signal_justme
        data:
          title: Missing Devices
          message: '{% set phrase = ''s are '' if states(''sensor.z2m_last_seen_entities'')|int > 1 else '' is '' %} 
                    The following sensor{{ phrase }}missing: {{ state_attr(''sensor.z2m_last_seen_entities'', ''devices'') | join(', ') }}'

pannal · July 15, 2022, 3:48pm

Hey, thank you for this. I’m observing something strange, though: The z2m_last_seen_entities entity reports its value as “unknown”, even though I can use the template editor to get the state value of 0.

~~Does a state of 0 equal unknown?~~

Strike that. It’s working now. Took a couple of minutes.

Thank you!

ChileHA · August 22, 2022, 5:09pm

Sorry for the dumb question. How can I use the template code if I have a sensors.yaml file?

Thanks!!

iz3man · August 23, 2022, 11:11am

Same here. I also struggle “translating” those configuration.yaml entries in seperate files (like sensors.yaml). I now moved parts around a dozen time, but it still is in wrong format. Would be really great to see it formated for sensors.yaml.

ChileHA · August 23, 2022, 12:40pm

Dude, just copy the entire template code in configuration.yaml. That work for me, but is a mastery how to go on separate files.

ChileHA · August 23, 2022, 12:43pm

One more thing. Remember to allow your last seen entity option to be show. If not you will get zero results.

iz3man · August 23, 2022, 12:44pm

Sure. Of course that works, but doesn’t answer my (and your own prior) question. I got seperate files for a reason, and would like to incorporate those additions.

ChileHA · August 23, 2022, 12:50pm

For sure!!

jarrah · September 7, 2022, 7:21pm

A fair question! I guess you already have something like this in configuration.yaml?

sensor: !include sensors.yaml

If so you would need another file called templates.yaml and add an extra line in configuration.yaml as follows:

template: !include templates.yaml`

However, I use package files these days…

In configuration.yaml, I have:

homeassistant:
  packages: !include_dir_named packages/

Within my main homeassistant folder where configuration.yaml is located, I have a folder called “packages”. Within packages I have many yaml files that are named by function, and each yaml file can contain which ever domain is required, e.g.

my_package_file.yaml

automation:
  ...
  ...

sensor:
  ...
  ...

template:
  ...
  ...

input_boolean:
  ...
  ...

In this case I have a file called “pkg_xiaomi_devices.yaml” within packages folder that has the following:

automation:
  - alias: Offline Zigbee Devices
    id: offline_zigbee_devices
    description: Sends notification for offline Z2m devices
    trigger:
      - platform: time
        at: '20:00'
    condition:
      - condition: template
        value_template: '{{states(''sensor.z2m_last_seen_entities'')|int > 0}}'
    action:
      - service: notify.signal_justme
        data:
          title: Missing Devices
          message: '{% set phrase = ''s are '' if states(''sensor.z2m_last_seen_entities'')|int > 1 else '' is '' %} 
                    The following sensor{{ phrase }}missing: {{ state_attr(''sensor.z2m_last_seen_entities'', ''devices'') }}'

template:
  - trigger:
      - platform: time_pattern
        hours: "/1"
        minutes: 0
    sensor:
      - unique_id: z2m_last_seen_entities
        name: "Z2M Last Seen Entities"
        state: >
          {% set lapsed_hours = 36 %}
          {% set ns = namespace(count=0) %}
          {% for state in states.sensor | selectattr('entity_id', 'search', '.*_last_seen$')  %}
            {% if states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id),0)) > ((lapsed_hours | int) * 60 * 60)) %}
              {% set ns.count = ns.count + 1 %}
            {% endif %}
          {% endfor %}
          {{ ns.count }}     
        attributes:
          devices: >
            {% set lapsed_hours = 36 %}
            {% set result = namespace(sensors=[]) %}
            {% for state in states.sensor | selectattr('entity_id', 'search', '.*_last_seen$') %}
              {% if states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id),0)) > ((lapsed_hours | int) * 60 * 60)) %}
                {% set result.sensors = result.sensors + [state.name | regex_replace(find='_last_seen', replace='') ~ ' (' ~ relative_time(strptime(states(state.entity_id), '%Y-%m-%dT%H:%M:%S%z', 'unavailable')) ~ ')'] %}
              {% endif %}
            {% endfor %}
            {{ result.sensors }}

Here are a few more files that I use:

jarrah · January 17, 2023, 8:58pm

I’ve just realised that my offline devices automation hadn’t been working for a while, giving the following error message:

ERROR (MainThread) [homeassistant.components.automation] Automation with alias 'Offline Zigbee Devices' could not be validated and has been disabled: template value should be a string for dictionary value @ data['action'][0]['data']. Got OrderedDict([('title', 'Missing Devices'), ('message', "{% set phrase = ''s are '' if states(''sensor.z2m_last_seen_entities'')|int > 1 else '' is '' %} The following sensor{{ phrase }}missing: {{ state_attr(''sensor.z2m_last_seen_entities'', ''devices'') | join(', ') }}")])

I fixed it by removing all the double quotes and starting the templates on a new line, as seen below:

  - alias: Offline Zigbee Devices
    id: offline_zigbee_devices
    description: Sends notification for offline Z2m devices
    trigger:
      - platform: time
        at: '20:00'
    condition:
      - condition: template
        value_template: >
          {{states('sensor.z2m_last_seen_entities')|int > 0}}
    action:
      - service: notify.signal_justme
        data:
          title: Missing Devices
          message: >
            {% set phrase = 's are ' if states('sensor.z2m_last_seen_entities')|int > 1 else ' is ' %} 
            The following sensor{{ phrase }}missing: {{ state_attr('sensor.z2m_last_seen_entities', 'devices') | join(', ') }}

pOpY · April 3, 2023, 6:31pm

jarrah:

  - alias: Offline Zigbee Devices
    id: offline_zigbee_devices
    description: Sends notification for offline Z2m devices
    trigger:
      - platform: time
        at: '20:00'
    condition:
      - condition: template
        value_template: >
          {{states('sensor.z2m_last_seen_entities')|int > 0}}
    action:
      - service: notify.signal_justme
        data:
          title: Missing Devices
          message: >
            {% set phrase = 's are ' if states('sensor.z2m_last_seen_entities')|int > 1 else ' is ' %} 
            The following sensor{{ phrase }}missing: {{ state_attr('sensor.z2m_last_seen_entities', 'devices') | join(', ') }}

thanks, working great.

gdgeist · April 14, 2023, 2:58pm

Name/friendly name just replicates the underscored entity_id here as well. If your code is matching only against " last seen" vs “last_seen” would explain why this isnt working for me. Will look at your code

gdgeist · April 14, 2023, 7:14pm

looks like your looking for last_seen in the entity id which my entity id’s are formatted as. I have some zigbee devices i havent plugged in for months and its not catching those. Looking at your time stamp parser