Templating knowledge needed to exclude elements of an array (CL sensor, curl)

e-raser · October 31, 2023, 1:36pm

I got one command line sensor:

  - platform: command_line
    name: Incidents Aqara Devices Missing Status Updates
    unique_id: xxx
    command_timeout: 60
    scan_interval: 1800
    command: 'curl -s http://xxx.xxx.xxx.xxx:40850/api/XXXXXXXXXX/sensors | jq -r "{ "devices": [ .[] | select(.manufacturername==\"LUMI\") | select(.lastseen < \"$(date -u -d@$(($(date +%s)-14460)) +%Y-%m-%dT%H:%M:%S)\")] | map(.name) | sort | unique, "total": [ .[] | select(.manufacturername==\"LUMI\") | select(.lastseen < \"$(date -u -d@$(($(date +%s)-14460)) +%Y-%m-%dT%H:%M:%S)\")] | map(.name) | sort | unique | length }"'
    value_template: "{{ value_json.total }}"
    json_attributes:
      - devices

The core part is:

"devices": [ .[] | select(.manufacturername==\"LUMI\") | select(.lastseen < \"$(date -u -d@$(($(date +%s)-14460)) +%Y-%m-%dT%H:%M:%S)\")] | map(.name) | sort | unique,

It basically catches all devices (output listed e. g. as Contact Kitchen Window, Motion Entrance, ...) from Phoscon/deCONZ which did not communicate within a certain timeframe. It’s basically perfect for discovering dead battery Aqara devices very early.

BUT: Has anyone an idea on how to exclude certain devices from the output?

Probably within the json_attributes “devices” itself - how to filter/remove (= ignore) elements of that array?

This way one could “silence” or whitelist/allowlist certain devices and this way reduce false positives for devices known to be less responsive in general (I got especially one of those kind).

I currently just lack the skills of doing so, really hoping to get some kick-start here.

Ideas:

Basic approach: exclude the “ignore devices” directly in the sensor
Optimization: create a group sensor with ignored devices maybe and link that group sensor in the sensor instead of having all excluded devices hard coded in the sensor

michaelblight · February 11, 2024, 2:26am

The way I’ve done this for low battery warnings is to use customizations. I have a domain customization that sets “ignore_low_battery” to false for all that can have batteries, and then override that to true for specific entities. Then I just use

selectattr('attributes.ignore_low_battery', 'eq', False)

fwiw I also store the date each battery was replaced in Grocy, and then a “tap_action” updates Grocy, writes all of the dates out to a customizations file, reloads all customizations, and calls “update_entity”. Works pretty well.

e-raser · February 11, 2024, 2:37am

Interesting and smart. But this is not especially about low batteries - and the actual purpose actually doesn’t really matter.

It’s more about the technical part: function and syntax on how to ignore / exclude certain devices as stated at

Can you help on that?

michaelblight · February 11, 2024, 4:54am

Ah yes I see. There’s no flexibility in json_attributes, as you’ve no doubt realised. Therefore you have 2 choices down this path: do it in the curl or have another template sensor to filter this one.

Another template sensor would be easier, as you could use an input_select to store the ones to ignore (for example).

I was hopeful you could use the deCONZ API to write a property to the sensor to ignore it, but there doesn’t appear to be any available. Therefore, you’d probably have to use --argfile (or --slurpfile) to load a list of names to ignore.

But I’d probably go down the path of getting all the lastseen data from deCONZ into HA (say every hour or daily). This would be useful to me too, as I recently realised Hue Dimmers only update the battery when they’re used, so I have some that are rarely used and die without me knowing. So I might look at leveraging the customization approach. This possibly also means reading the core.entity_registry file, because even though the link between deCONZ and HA is the unique_id, I don’t know any other way of reading it. And in my case I want to load lastseen against the right entity.

But then again, it’s probably easier to send myself a message to press the button if last_updated is over a week ago.

e-raser · February 12, 2024, 6:54pm

Much good theory. As you might have discovered, my sensor is already working, and it is doing great.

I really “just” need a solution to filter things, no matter which way.

Do you have a working sample?

michaelblight · February 14, 2024, 12:04am

As mentioned I want it linked to HA entities which is easiest to do with customizations, so there’s some stuffing about because of that. I use Node Red, so this is my prototype:

This loads all of the “lastseen” values against the relevant entities. Then I use the following template to show those that haven’t been seen recently. Unfortunately by this stage “lastseen” is a string, so (like you), I’m doing a string comparison which just feels wrong, but never mind.

{% set then = utcnow() - timedelta(hours=8) %}
{% set test = then.strftime("%Y-%m-%dT%H:%M:%SZ") %}

{{ states.sensor
  | selectattr('attributes.deconz_lastseen_utc', 'defined')
  | selectattr('attributes.deconz_lastseen_utc', 'lt', test)
  | map(attribute='entity_id')
  | list
}}

This could then be extended to have a “ignore_lastseen” customization, and exclude them.

If you end up looking at the code below, there are a couple of things to note.

It uses a global to get the path to the deCONZ API.
The purple MQTT-out node is sending that data to an MQTT sensor. I’m not using it, but just wanted the data available.
I’m using a stand-alone Node Red, so my path to HA is different to if it was the addon.

[{"id":"a372aaa5e0f3f8ae","type":"group","z":"2f4619ac.4996b6","name":"Load deCONZ sensor data into HA","style":{"label":true,"fill":"#e3f3d3"},"nodes":["9ef0f958b6acbe59","b0a01888b41f747c","ec0d42c8bca597f7","536ab81ab10b33d9","b3f5f2bcedeadca7","e61cbaa74de159a6","74a14541d5d550b2","94a9cd8e0729cd28","d7765b1309a403ba","7135d0356fe127ba","2e7fd2467720e272","21a7cccfe2d61e6a","3c5982a9d69bbe1d","c91b4d4c6cdbd1dd","8f505bd3204082b5","3dfdc85b0af10de0","3b3994b26e07ea3f","7f03dda64ef75ee2","0b2d1d5ef3cbd462","8abb5cc9347809df","c3965558a3df1ae4"],"x":14,"y":5239,"w":1372,"h":262},{"id":"9ef0f958b6acbe59","type":"http request","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"deCONZ sensors","method":"GET","ret":"obj","paytoqs":"ignore","url":"","tls":"","persist":false,"proxy":"","insecureHTTPParser":false,"authType":"","senderr":false,"headers":[],"x":470,"y":5340,"wires":[["e61cbaa74de159a6"]]},{"id":"b0a01888b41f747c","type":"cronplus","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"4h","outputField":"payload","timeZone":"","storeName":"","commandResponseMsgOutput":"output1","defaultLocation":"","defaultLocationType":"default","outputs":1,"options":[{"name":"schedule1","topic":"topic1","payloadType":"default","payload":"","expressionType":"cron","expression":"0 0 */4 * * * *","location":"","offset":"0","solarType":"all","solarEvents":"sunrise,sunset"}],"x":110,"y":5280,"wires":[["7f03dda64ef75ee2"]]},{"id":"ec0d42c8bca597f7","type":"mqtt out","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"","topic":"mb/nr/sensor/deconz_sensors","qos":"","retain":"","respTopic":"","contentType":"","userProps":"","correl":"","expiry":"","broker":"2a3b12ee.85570e","x":1150,"y":5400,"wires":[]},{"id":"536ab81ab10b33d9","type":"file in","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"HA entities","filename":"/ha/config/.storage/core.entity_registry","filenameType":"str","format":"utf8","chunk":false,"sendError":false,"encoding":"none","allProps":false,"x":450,"y":5400,"wires":[["b3f5f2bcedeadca7"]]},{"id":"b3f5f2bcedeadca7","type":"json","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"","property":"payload","action":"","pretty":false,"x":610,"y":5400,"wires":[["e61cbaa74de159a6"]]},{"id":"e61cbaa74de159a6","type":"join","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Join","mode":"custom","build":"object","property":"payload","propertyType":"msg","key":"topic","joiner":"\\n","joinerType":"str","accumulate":false,"timeout":"","count":"2","reduceRight":false,"reduceExp":"","reduceInit":"","reduceInitType":"","reduceFixup":"","x":770,"y":5340,"wires":[["d7765b1309a403ba"]]},{"id":"74a14541d5d550b2","type":"change","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Topic","rules":[{"t":"set","p":"topic","pt":"msg","to":"deconz","tot":"str"},{"t":"set","p":"url","pt":"msg","to":"$globalContext('deconz_api') &\t\"sensors\"","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":5340,"wires":[["9ef0f958b6acbe59"]]},{"id":"94a9cd8e0729cd28","type":"change","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Topic","rules":[{"t":"set","p":"topic","pt":"msg","to":"registry","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":5400,"wires":[["536ab81ab10b33d9"]]},{"id":"d7765b1309a403ba","type":"function","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Merge","func":"var ha_entities = msg.payload.registry.data.entities;\nvar deconz = msg.payload.deconz;\n\nmsg.payload = {\n    devices: Object.entries(deconz).map(([k, v]) => {\n        const id = deconz[k].uniqueid;\n        return { \n            deconz: {\n                key: k,\n                value: v\n            },\n            ha_entities: ha_entities\n                .filter((e) => e.disabled_by == null)\n                .filter((e) => e.unique_id.startsWith(id))\n                .map((e) => e.entity_id)\n        }\n    })\n}\nmsg.payload.state = msg.payload.devices.length;\nglobal.set('deconz', msg.payload);\n\nreturn msg;\n","outputs":1,"timeout":0,"noerr":0,"initialize":"","finalize":"","libs":[],"x":910,"y":5340,"wires":[["ec0d42c8bca597f7","3b3994b26e07ea3f"]]},{"id":"7135d0356fe127ba","type":"api-call-service","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Update","server":"5c211963.f91a88","version":5,"debugenabled":false,"domain":"homeassistant","service":"update_entity","areaId":[],"deviceId":[],"entityId":[],"data":"","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":1080,"y":5460,"wires":[["0b2d1d5ef3cbd462"]]},{"id":"2e7fd2467720e272","type":"change","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Payload","rules":[{"t":"set","p":"deconz","pt":"msg","to":"deconz","tot":"global"},{"t":"set","p":"payload","pt":"msg","to":"(\t    deconz.devices.ha_entities\t        ~> $distinct()\t        ~> $sort()\t        ~> $map(function($entity_id) {(\t                $deconz := $$.deconz.devices[$entity_id in ha_entities].deconz.value;\t                {\t                    $entity_id: {\t                        \"deconz_lastseen_utc\": $deconz.lastseen,\t                        \"deconz_lastseen\": $moment($deconz.lastseen).format('D MMM YY')\t                    }\t                }\t            )})\t        ~> $merge()\t)","tot":"jsonata"},{"t":"set","p":"refresh","pt":"msg","to":"{\t    \"target\": {\t        \"entity_id\": $keys(payload)\t    }\t}","tot":"jsonata"}],"action":"","property":"","from":"","to":"","reg":false,"x":300,"y":5460,"wires":[["21a7cccfe2d61e6a"]]},{"id":"21a7cccfe2d61e6a","type":"yaml","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","property":"payload","name":"","x":450,"y":5460,"wires":[["3c5982a9d69bbe1d"]]},{"id":"3c5982a9d69bbe1d","type":"file","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Save","filename":"/ha/config/customize_deconz.yaml","filenameType":"str","appendNewline":true,"createDir":false,"overwriteFile":"true","encoding":"none","x":590,"y":5460,"wires":[["c91b4d4c6cdbd1dd"]]},{"id":"c91b4d4c6cdbd1dd","type":"api-call-service","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Reload","server":"5c211963.f91a88","version":5,"debugenabled":false,"domain":"homeassistant","service":"reload_core_config","areaId":[],"deviceId":[],"entityId":[],"data":"","dataType":"jsonata","mergeContext":"","mustacheAltTags":false,"outputProperties":[],"queue":"none","x":740,"y":5460,"wires":[["8f505bd3204082b5"]]},{"id":"8f505bd3204082b5","type":"change","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Payload","rules":[{"t":"move","p":"refresh","pt":"msg","to":"payload","tot":"msg"}],"action":"","property":"","from":"","to":"","reg":false,"x":900,"y":5460,"wires":[["7135d0356fe127ba"]]},{"id":"3dfdc85b0af10de0","type":"link in","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"deCONZ sensor","links":[],"x":120,"y":5340,"wires":[["74a14541d5d550b2","94a9cd8e0729cd28"]],"l":true},{"id":"3b3994b26e07ea3f","type":"link out","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Create sensor return","mode":"return","links":[],"x":1120,"y":5340,"wires":[],"l":true},{"id":"7f03dda64ef75ee2","type":"link call","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"","links":["3dfdc85b0af10de0"],"linkType":"static","timeout":"30","x":320,"y":5280,"wires":[["c3965558a3df1ae4"]]},{"id":"0b2d1d5ef3cbd462","type":"link out","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"Create cust return","mode":"return","links":[],"x":1270,"y":5460,"wires":[],"l":true},{"id":"8abb5cc9347809df","type":"link in","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"deCONZ cust","links":[],"x":110,"y":5460,"wires":[["2e7fd2467720e272"]],"l":true},{"id":"c3965558a3df1ae4","type":"link call","z":"2f4619ac.4996b6","g":"a372aaa5e0f3f8ae","name":"","links":["8abb5cc9347809df"],"linkType":"static","timeout":"30","x":520,"y":5280,"wires":[[]]},{"id":"2a3b12ee.85570e","type":"mqtt-broker","name":"Home","broker":"mqtt.blight.duckdns.org","port":"1883","clientid":"nr_docker","autoConnect":true,"usetls":false,"protocolVersion":"4","keepalive":"60","cleansession":true,"birthTopic":"","birthQos":"0","birthPayload":"","birthMsg":{},"closeTopic":"","closeQos":"0","closePayload":"","closeMsg":{},"willTopic":"","willQos":"0","willPayload":"","willMsg":{},"userProps":"","sessionExpiry":""},{"id":"5c211963.f91a88","type":"server","name":"Home Assistant","version":5,"addon":false,"rejectUnauthorizedCerts":true,"ha_boolean":"y|yes|true|on|home|open","connectionDelay":false,"cacheJson":true,"heartbeat":false,"heartbeatInterval":"30","areaSelector":"friendlyName","deviceSelector":"friendlyName","entitySelector":"friendlyName","statusSeparator":"at: ","statusYear":"hidden","statusMonth":"short","statusDay":"numeric","statusHourCycle":"h23","statusTimeFormat":"h:m","enableGlobalContextStore":true}]```

e-raser · February 14, 2024, 12:21pm

OK tbh I

on the one hand appreciate that you implemented a solution for yourself and shared your results with us, while at the same time
on the other hand it looks like I still don’t have a solution to my original question

Maybe I’m missing the transfer knowledge currently. But to stamp it into a question:
how does your custom integration help me to filter/ignore certain strings?

michaelblight · February 14, 2024, 11:12pm

Like I said earlier…

I have changed my template sensor to include checking the customization:

        {% set then = utcnow() - timedelta(hours=8) %}
        {% set test = then.strftime("%Y-%m-%dT%H:%M:%SZ") %}

        {{ states.sensor
          | selectattr('attributes.deconz_lastseen_utc', 'defined')
          | rejectattr('attributes.ignore_unseen', 'eq', true)
          | selectattr('attributes.deconz_lastseen_utc', 'lt', test)
          | map(attribute='entity_id')
          | list | count
        }}

Once you’ve got all the data into a template sensor you can filter it however you like.

e-raser · August 23, 2024, 9:28pm

I came up with a working solution meanwhile (note: created with AI assistance, even it took almost a dozen attempts to fine tune and remove all the errors). Here’s the command part, just in case anyone ever tries to achieve the same:

      command: >
        curl -s http://xxx.xxx.xxx.xxx:40850/api/XXXXXXXXXX/sensors | 
        jq -r '
        def ignore_list: ["device name A", "device name B", "device name C"];
        def is_recent: .lastseen > (now - 14460 | strftime("%Y-%m-%dT%H:%M:%S"));
        {
          "devices": [.[] | select(.manufacturername == "LUMI" and (is_recent | not) and (.name as $n | ignore_list | index($n) | not))] | map(.name) | sort | unique,
          "total": [.[] | select(.manufacturername == "LUMI" and (is_recent | not) and (.name as $n | ignore_list | index($n) | not))] | map(.name) | sort | unique | length
        }'

Unfortunately (my initial ultimate goal) it is still not possible to use a ignore group (so another “group.ignored_devices”. That’s for the version 3 then:

It would only be able to contain entities, while the string comparison in the jq part (rather: the API output) has device names, so there needs to be an extra mile doing “sensor.device_a_temperature → get device name → compare that one”.
That was one too much even for the AI.
Therefore for now I maintain my list of ignored devices in the ignore_list array.
Only actual downside of this: I need to reload the command line sensor after making changes to that ignore_list every time. A group would be much more convenient because rather “dynamic” here.