It took me a while to figure this out, so I just thought I would share in case it helps someone else out for their project, or general HA programming understanding.
You can paste the following into your Developer Tools->Template, to get a list of all offline Zigbee/ZWave Devices, should be easily extensible to other integrations also.
{% set result=namespace(offline_devices=[]) %}
{% set integrations=['mqtt', 'zwave_js', 'zha'] %}
{% for integration in integrations %}
{% for device in integration_entities(integration) | map('device_id') | sort | unique | list %}
{% for entity in device_entities(device) %}
{% if entity.startswith(('switch','sensor','binary_sensor')) and "unavailable" in expand(entity)|string %}
{# add error checking, in case name_by_user==None #}
{% set device_name=iif(device_attr(device,"name_by_user"), device_attr(device,"name_by_user"), "ERROR", device_attr(device,"name")) %}
{% set result.offline_devices= result.offline_devices + [device_name] %}
{% endif %}
{% endfor %}
{% endfor %}
{% endfor %}
{{"Offline Devices:\n --"}} {{result.offline_devices|sort|unique|join('\n -- ')}}
Basically we just get iterate through desired integrations, get a list of devices in each, and then check if any [sensor,binary_sensor,switch] is offline.
Additionally, here is a ready to use blueprint for a similar result, but I used a very different technique in that one (searching by device_class (battery, switch)