Offline detection for Z2M devices with last_seen

jarrah · April 14, 2022, 7:50pm

doh, I really should RTFM. Changing to ISO_8601_local did the trick (as it tells me to do in the first post in this thread!!!). The last_seen timestamp is now 2022-04-14T20:37:02+01:00 which of course works…

Mr_Groch · April 19, 2022, 11:59am

Hi all!

Lot happens in Zigbee2MQTT and Home Assistant, so I’ve updated this blueprint! Device attributes are now legacy in Z2M and ‘last seen’ sesnors are now exposed as sensors with ‘timestamp’ device_class in Home Assstant by Z2M - so I’ve decided to drop checking for ‘lqi’ sensors - I’m checking all ‘last seen’ sensors with ‘timestamp’ device_class instead…

So all you need is to enable ‘last seen’ sensors in device page if it is disabled, and also modify excluded sensors if you are using it…

That is breaking change…

PS. I’m also checking now for ‘unavailable’ state

jarrah · April 20, 2022, 7:54pm

Hmm, I’ve been trying to figure out how an easier way to do this:

The problem is I have 125 Z2M devices, so enabling the last_seen entity for each device within HA will take far too long! I’ve not been able to find an easy way to enable this from within Z2M either, with comments near the end of this issue describing the same problem: last seen not showing up · Issue #9398 · Koenkk/zigbee2mqtt · GitHub

Mr_Groch · April 20, 2022, 8:21pm

This is not documented, and I can’t test it now, but try adding this into your Z2M configuration, it should enable all last_seen sensors by default in HA:

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

EDIT: You need to restart Z2M and HA to refresh discovery for all devices

jarrah · April 20, 2022, 8:48pm

Great find, thank you for the tip!

I was able to find this related Discussion based on your suggestion:

However, having just tried it myself it doesn’t appear to work for me.

I copied the options into Z2M configuration.yaml, restarted Z2M, checked HA entities to find last_seen still disabled, so restarted HA as well, but unfortunately no change.

I raised this issue just before your reply and so I’ve now updated it with my results in case there’s another fix.

github.com/Koenkk/zigbee2mqtt

[Feature request]: Option to enable sensor.<device>_last_seen entity within HA from Z2M configuration settings

opened 08:21PM - 20 Apr 22 UTC

jarrah31

feature request

### Is your feature request related to a problem? Please describe Based on th…e latest update to the following HA Blueprint, I need to enable the last_seen entity on my devices so that I can be notified when they go offline. https://community.home-assistant.io/t/offline-detection-for-z2m-devices-with-last-seen/310138 <img width="581" alt="image" src="https://user-images.githubusercontent.com/3072303/164315039-08d71fbd-6f5e-440b-9c7c-5c6383b0e806.png"> The problem is I have 125 devices, so manually enabling last_seen for each one will take far too long! ### Describe the solution you'd like Related to this comment https://github.com/Koenkk/zigbee2mqtt/issues/8287#issuecomment-939254755, would it be possible to have an option that enables last_seen entities by default please? Perhaps called something like `last_seen_entity' ``` advanced: # Optional: Add a last_seen attribute to MQTT messages, contains date/time of last Zigbee message # possible values are: disable (default), ISO_8601, ISO_8601_local, epoch (default: disable) last_seen: 'ISO_8601_local' last_seen_entity: 'true' ``` ### Describe alternatives you've considered Other than manually enabling last_seen for all 125 devices, I'm not aware of an alternative method. ### Additional context Another user described the same problem in a similar closed thread: https://github.com/Koenkk/zigbee2mqtt/issues/9398#issuecomment-1003752055

Kosy · April 20, 2022, 10:11pm

We are on the same page here. Added both:

advanced:
  last_seen: ISO_8601_local

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

And there are no attributes or entities present.
Controary to this issue i cannot find any “last_seen” value in states. How about You ?

Mr_Groch · April 21, 2022, 7:23am

I’ve enabled all my ‘last seen’ sensors earlier, but I’ve added this to my Z2M configuration:

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

Restarted Z2M and HA, and checked with MQTT Explorer discovery topics for ‘last_seen’ sensors:

As you can see - enabled_by_default is true, so it works…

Try manually set Home Assistant status topic to offline and online few times (default is hass/status for Z2M)

jarrah · April 21, 2022, 8:01am

Thanks @Mr_Groch that’s useful to see - I’ll check mine when back home later today.

Kosy · April 21, 2022, 6:24pm

Unfortunetly I`m still stuck.
My config:

data_path: /share/zigbee2mqtt
socat:
  enabled: false
  master: pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777
  slave: tcp-listen:8485,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5
  options: '-d -d'
  log: false
mqtt:
  server: mqtt://192.168.0.111:1883
  user: mqtt
  password: '!secret mqttpwd'
serial:
  port: /dev/serial/by-path/pci-0000:00:07.0-usb-0:2.2:1.0-port0
external_converters: []
devices: devices.yaml
groups: groups.yaml
homeassistant: true
permit_join: false
advanced:
  last_seen: ISO_8601_local
  log_level: info
  pan_id: 6756
  channel: XX
  network_key:
    - XX 
  availability_blocklist: []
  availability_passlist: []
  ikea_ota_use_test_url: true
device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true
blocklist: []
passlist: []
queue: {}
frontend:
  port: 8099
experimental: {}
availability: true
zigbee_herdsman_debug: false

This config beside entries from my previous post was running quite long.
I have no clue what I am doing wrong. I have compered everything with z2m docs, restarted HA , server few times and was manually putting offline value for homeassistant/status (according to docs). Weird thing is that there is no such message by itself in that topic.

jarrah · April 21, 2022, 6:33pm

Ok, so I also see the same enabled_by_default message within MQTT explorer from home assistant:

Along with the hidden last_seen entity within HA:

The last_seen attribute is there as usual:

I rebooted HA and Z2M a few times again, as well as publishing manual offline/online message, but last_seen remains firmly hidden.

I see that Koenkk responded to my issue and suggested we use the availability feature for this instead: Device-Availability | Zigbee2MQTT

I guess this would just mean monitoring the offline / online state of a device?

jarrah · April 21, 2022, 6:39pm

So in your HA states, you don’t see the last_seen attribute like this?

Or are you just finding that the last_seen entity remains hidden like mine?

Kosy · April 21, 2022, 6:43pm

You are lucky I do not have attribute or hidden entity “last_seen”.
If You have that attribute this all that blueprint is about.
Avaiabilty is not just that good since this is presented in documentation.
My colegue also tested this settings and has last_seen in debug for devices that are out of network.

jarrah · April 21, 2022, 6:51pm

It is strange that you don’t at least have the last_seen attribute because this is what the last_seen: ISO_8601_local gives you. Sorry for the obvious question, but are you running the latest Z2M (1.25.0) and HA (I’m on 2022.3.x - I’ll try 2022.4 just in case)?

My problem is that sensor.motion_bed4_last_seen entity remains hidden (along with all the other 124 Z2M devices), unless I manually enable it within HA. For me I’d rather have them all enabled automatically, which is what I hoped the `device_options’ method would do.

Kosy · April 21, 2022, 6:55pm

I`m running Z2m :1.25.0-1 and HA core-2022.4.4.
You can use this if You have attribute.

Kosy · April 21, 2022, 7:10pm

This weirdness comes from fact that Z2M config is moving from config to WebUI. See supervisor log for proof. Check Z2M WebUI > Settings > Generated Configuration (do not know how it is signed in your UI language) and then look for enabled_by_ default. Id it is not there edit ..\share\zigbee2mqtt\configuration.yaml
manually. That helped me with attributes.

Scoff · April 21, 2022, 7:30pm

I have set this up and am seeing the last_seen attribute on all my Z2M devices, but I’m not seeing what I expected. Perhaps I have misunderstood the functionality that this automation provides.
I have a device in Z2M which has been in an ‘off’ state for over 20 hours now (intentionally).
I’ve set the automation to run every day (option 0) at a set time. It is set to report on device that have a last_seen time of over 1 hour.
I’ve set a basic push notification to come through on thre HA app on my phone.
If I run the automation manually, I get the push notification but it doesn’t show any devices or information.
Is this because my device isn’t ‘sensor’?

Here is the device I expect to see reported:

jarrah · April 21, 2022, 7:51pm

If you are using the latest version of the blueprint, it now looks for a last_seen entity rather than the last_seen attribute within an entity.

These need to be enabled within HA → Configuration → Devices
Select the _last_seen entity within Diagnostic:

Change to enabled and then restart HA:

Hope that makes sense?

Scoff · April 21, 2022, 8:53pm

Ah I understand now, thank you for explaining
I’ve enabled that previously disabled entity for the one device that has been offline, and added {{sensors}} to the start of the notification message, and ran the automation - works perfectly

jarrah · April 22, 2022, 7:54pm

Ok, hopefully the following will help other users enable the _last_seen entity without having to manually enable them all.

After a discussion with Koenkk here, he said that setting the following in Z2M doesn’t by itself automatically make hidden _last_seen entities become enabled because “home assistant doesn’t update this after initial discovery”.

device_options:
  homeassistant:
    last_seen:
      enabled_by_default: true

The solution is to basically remove the MQTT integration and have HA discover all the Z2M entities again. He did however say that “this will reset all your entities in HA (icons, friendly names)”. This doesn’t luckily impact me, so I went ahead and did the following:

stop Z2M
remove MQTT integration in HA
restart HA
confirm devices/entities have been removed in HA
start Z2M
add and reconfigure MQTT integration in HA
restart HA

I now have all 125 _last_seen entities in HA!

btw, if anyone wants to test part of the blueprint in the Template Editor, I was using the following:

{% set result = namespace(sensors=[]) %}
{% for state in states.sensor | selectattr('attributes.device_class', '==', 'timestamp') %}
{% if 'last_seen' in state.entity_id and (states(state.entity_id) == 'unavailable' or ((as_timestamp(now()) - as_timestamp(states(state.entity_id))) > ((24 | int) * 60 * 60))) %}
{% set result.sensors = result.sensors + [state.name | regex_replace(find='_last_seen', replace='') ~ ' (' ~ relative_time(strptime(states(state.entity_id), '%Y-%m-%dT%H:%M:%S%z', 'unavailable')) ~ ')'] %}
{% endif %}
{% endfor %}
{{ result.sensors | join(', ') }}

@Mr_Groch it looks like you’re missing the underscores in the line below:

regex_replace(find=' last seen',

Mr_Groch · April 22, 2022, 8:50pm

Not really - this is for grabbing device name for notify - I’m using ‘last seen’ entity with name like ‘device last seen’, so I want to cut ’ last seen’ part from it (notice that regex_replace is done on state.name not state.entity_id)