Offline detection for Z2M devices with last_seen

[jarrah]

Yeah, this blueprint no longer works for me on 2022.3.x

[jarrah]

I’ve been trying to figure out where the issue is without much knowledge of templating, so hopefully the info below is correct.
At the moment I think the fault seems to be with this part of the template:

{{ strptime(state_attr('binary_sensor.door_front_contact', 'last_seen'), '%Y-%m-%dT%H:%M:%S%z', 0) }}

Running this in the template editor returns a 0 due to it failing. Remove the “,0” (as it is in the blueprint template) returns the timestamp string, e.g. 2022-04-14T17:56:31.029Z. This is why relative_time doesn’t work because strptime can’t convert the string to a datetime object.

# returns time @ 10 am, if fails returns 0
{{ strptime("10:00", "%H:%M", 0) }}

strptime(string, format) 
parses a string based on a format and returns a datetime object. 
If that fails, returns the default value, or if omitted the unprocessed input value.

What I can’t figure out is why strptime isn’t able to parse the date string. I suspect its something to do with the seconds and UTC offset that it can’t convert:

:%S%z
:31.029Z

According to this page, it says the following:

In fact, if I test the following template by removing “.029Z” from the timestamp, it successfully converts to a datetime object:

{{ strptime("2022-04-14T19:08:50", '%Y-%m-%dT%H:%M:%S', 0) }}
Result:
2022-04-14 19:08:50

My zigbee2mqtt configuration.yaml has last_seen set to ISO_8601 which might be related. An alternative is ISO_8601_local which I might try as well.

For reference, the full template (line wraps removed from the blueprint) is below:

{% set result = namespace(sensors=[]) %} 
{% for state in states.sensor | selectattr('attributes.unit_of_measurement', '==', 'lqi') %}
{% if state_attr(state.entity_id, 'last_seen') != None and (as_timestamp(now()) - as_timestamp(state_attr(state.entity_id, 'last_seen'))) > ((1 | int) * 60 * 60) %}
{% set result.sensors = result.sensors + [state.name | regex_replace(find=' linkquality', replace='') ~ ' (' ~ relative_time(strptime(state_attr(state.entity_id, 'last_seen'), '%Y-%m-%dT%H:%M:%S%z')) ~ ')'] %}
{% endif %}
{% endfor %}
{{ result.sensors | join(', ') }}

[jarrah]

doh, I really should RTFM. Changing to ISO_8601_local did the trick (as it tells me to do in the first post in this thread!!!). The last_seen timestamp is now 2022-04-14T20:37:02+01:00 which of course works…

[Mr_Groch]

Hi all!

Lot happens in Zigbee2MQTT and Home Assistant, so I’ve updated this blueprint! Device attributes are now legacy in Z2M and ‘last seen’ sesnors are now exposed as sensors with ‘timestamp’ device_class in Home Assstant by Z2M - so I’ve decided to drop checking for ‘lqi’ sensors - I’m checking all ‘last seen’ sensors with ‘timestamp’ device_class instead…

So all you need is to enable ‘last seen’ sensors in device page if it is disabled, and also modify excluded sensors if you are using it…

That is breaking change…

PS. I’m also checking now for ‘unavailable’ state

[jarrah]

Hmm, I’ve been trying to figure out how an easier way to do this:

@Mr_Groch

Remember also to enable ‘last seen’ sensors for all your Zigbee2MQTT devices (it is now disabled by default) in Home Assistant device page!

The problem is I have 125 Z2M devices, so enabling the last_seen entity for each device within HA will take far too long! I’ve not been able to find an easy way to enable this from within Z2M either, with comments near the end of this issue describing the same problem: last seen not showing up · Issue #9398 · Koenkk/zigbee2mqtt · GitHub

[Mr_Groch]

This is not documented, and I can’t test it now, but try adding this into your Z2M configuration, it should enable all last_seen sensors by default in HA:

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

EDIT: You need to restart Z2M and HA to refresh discovery for all devices

[jarrah]

Great find, thank you for the tip!

I was able to find this related Discussion based on your suggestion:

How to set which entities are enable in HA Discovery · Discussion #10484 ·...

However, having just tried it myself it doesn’t appear to work for me.

I copied the options into Z2M configuration.yaml, restarted Z2M, checked HA entities to find last_seen still disabled, so restarted HA as well, but unfortunately no change.

I raised this issue just before your reply and so I’ve now updated it with my results in case there’s another fix.

[Feature request]: Option to enable sensor.<device>_last_seen entity within HA from Z2M configuration settings

[Kosy (Igor St)]

We are on the same page here. Added both:

advanced:
  last_seen: ISO_8601_local

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

And there are no attributes or entities present.
Controary to this issue i cannot find any “last_seen” value in states. How about You ?

[Mr_Groch]

I’ve enabled all my ‘last seen’ sensors earlier, but I’ve added this to my Z2M configuration:

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

Restarted Z2M and HA, and checked with MQTT Explorer discovery topics for ‘last_seen’ sensors:

As you can see - enabled_by_default is true, so it works…

Try manually set Home Assistant status topic to offline and online few times (default is hass/status for Z2M)

[jarrah]

Thanks @Mr_Groch that’s useful to see - I’ll check mine when back home later today.

[Kosy (Igor St)]

Unfortunetly I`m still stuck.
My config:

data_path: /share/zigbee2mqtt
socat:
  enabled: false
  master: pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777
  slave: tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5
  options: '-d -d'
  log: false
mqtt:
  server: mqtt://192.168.0.111:1883
  user: mqtt
  password: '!secret mqttpwd'
serial:
  port: /dev/serial/by-path/pci-0000:00:07.0-usb-0:2.2:1.0-port0
external_converters: []
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: false
advanced:
  last_seen: ISO_8601_local
  log_level: info
  pan_id: 6756
  channel: XX
  network_key:
    - XX 
  availability_blocklist: []
  availability_passlist: []
  ikea_ota_use_test_url: true
device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental: {}
availability: true
zigbee_herdsman_debug: false

This config beside entries from my previous post was running quite long.
I have no clue what I am doing wrong. I have compered everything with z2m docs, restarted HA , server few times and was manually putting offline value for homeassistant/status (according to docs). Weird thing is that there is no such message by itself in that topic.

[jarrah]

Ok, so I also see the same enabled_by_default message within MQTT explorer from home assistant:

Along with the hidden last_seen entity within HA:

The last_seen attribute is there as usual:

I rebooted HA and Z2M a few times again, as well as publishing manual offline/online message, but last_seen remains firmly hidden.

I see that Koenkk responded to my issue and suggested we use the availability feature for this instead: Device-Availability | Zigbee2MQTT

I guess this would just mean monitoring the offline / online state of a device?

[jarrah]

So in your HA states, you don’t see the last_seen attribute like this?

Or are you just finding that the last_seen entity remains hidden like mine?

[Kosy (Igor St)]

You are lucky I do not have attribute or hidden entity “last_seen”.
If You have that attribute this all that blueprint is about.
Avaiabilty is not just that good since this is presented in documentation.
My colegue also tested this settings and has last_seen in debug for devices that are out of network.

[jarrah]

It is strange that you don’t at least have the last_seen attribute because this is what the last_seen: ISO_8601_local gives you. Sorry for the obvious question, but are you running the latest Z2M (1.25.0) and HA (I’m on 2022.3.x - I’ll try 2022.4 just in case)?

My problem is that sensor.motion_bed4_last_seen entity remains hidden (along with all the other 124 Z2M devices), unless I manually enable it within HA. For me I’d rather have them all enabled automatically, which is what I hoped the `device_options’ method would do.

[Kosy (Igor St)]

I`m running Z2m :1.25.0-1 and HA core-2022.4.4.
You can use this if You have attribute.

[Kosy (Igor St)]

This weirdness comes from fact that Z2M config is moving from config to WebUI. See supervisor log for proof. Check Z2M WebUI > Settings > Generated Configuration (do not know how it is signed in your UI language) and then look for enabled_by_ default. Id it is not there edit ..\share\zigbee2mqtt\configuration.yaml
manually. That helped me with attributes.

[Scoff (Scott)]

I have set this up and am seeing the last_seen attribute on all my Z2M devices, but I’m not seeing what I expected. Perhaps I have misunderstood the functionality that this automation provides.
I have a device in Z2M which has been in an ‘off’ state for over 20 hours now (intentionally).
I’ve set the automation to run every day (option 0) at a set time. It is set to report on device that have a last_seen time of over 1 hour.
I’ve set a basic push notification to come through on thre HA app on my phone.
If I run the automation manually, I get the push notification but it doesn’t show any devices or information.
Is this because my device isn’t ‘sensor’?

Here is the device I expect to see reported:

[jarrah]

If you are using the latest version of the blueprint, it now looks for a last_seen entity rather than the last_seen attribute within an entity.

These need to be enabled within HA → Configuration → Devices
Select the _last_seen entity within Diagnostic:

Change to enabled and then restart HA:

Hope that makes sense?

[Scoff (Scott)]

Ah I understand now, thank you for explaining
I’ve enabled that previously disabled entity for the one device that has been offline, and added {{sensors}} to the start of the notification message, and ran the automation - works perfectly