MQTT sensor, Filter states to ignore "Null" "None "Unknown"

Configuration
,
1

Trying to create a secondary sensor as the payloads I am interested in are overwritten in a fraction of a second (original Z2M HA sensors); though they leave a trace in the logbook, they become hard to track in graphs/reports.
Therefore I want to get rid or filter out all undefined states like “null” “none” “unknown” etc.

image

Here are my 2 unsuccessful attempts at it:



  - name: "DOORLOCK front action tag2"  
    unique_id: doorlock_front_action_tag2
    state_topic: "zigbee2mqtt/DOORLOCK FRONT"  
    value_template: >-
      {% if value_json.action !== "None"  %}
        {{ value_json.action }}
      {% else %}
        {{ this.state }}  
      {% endif %}


  - name: "DOORLOCK front action tag3"  
    unique_id: doorlock_front_action_tag3
    state_topic: "zigbee2mqtt/DOORLOCK FRONT"  
    value_template: >-
      {% if value_json.action is defined  %}
        {{ value_json.action }}
      {% else %}
        {{ this.state }}  
      {% endif %}
2

I generally use:

 not in 'unavailable,unknown'

See if that helps

3 
  - name: "DOORLOCK front action tag8"  
    unique_id: doorlock_front_action_tag8
    state_topic: "zigbee2mqtt/DOORLOCK FRONT"  
    value_template: >-
      {% if value_json.action  not in ["unavailable","unknown"]  %}
        {{ value_json.action }}
      {% else %}
        {{ this.state }}  
      {% endif %}

Still get “None” and empty states:

image

4

So, apologize. I missed the json request.

I am looking at HA states. Does that change things? Not sure.

Certainly worth looking into!

5

Post an example of the payload published to this topic:

zigbee2mqtt/DOORLOCK FRONT
6 
{
  "action": null,
  "action_source_name": null,
  "action_user": null,
  "auto_relock_time": 984,
  "battery": 35,
  "battery_low": true,
  "linkquality": 189,
  "lock_state": "unlocked",
  "pin_code": null,
  "sound_volume": "high_volume",
  "state": "UNLOCK",
  "users": {
    "0": {
      "pin_code": "111111",
      "status": "enabled"
    },
    "1": {
      "pin_code": "111111",
      "status": "enabled"
    },
    "2": {
      "pin_code": "111111",
      "status": "enabled"
    },
    "3": {
      "pin_code": "111111",
      "status": "available"
    },
    "4": {
      "pin_code": "111111",
      "status": "enabled"
    },
    "5": {
      "pin_code": "111111",
      "status": "enabled"
    },
    "6": {
      "pin_code": "111111",
      "status": "enabled"

...
7

image
it also briefly sends “” just after a command. therefore the empty payloads in the logbook

8

From a quick test in the template editor.

{% if value_json.action is defined and value_json.action != "" %}

From my tests, when action = null, Jinja sees that as not defined. when it is “” then the normal string test != "" will pick that up.

{% set j = { "action": null } %}
{{ j.action is defined }} #False
{{ j.action is not none }} #True
{{ j.action != "" }} #True

{% set j = { "action": "" } %}
{{ j.action is defined }} #True
{{ j.action is not none }} #True
{{ j.action != "" }} #False
9

For some reason it still reports “None”

image

image
image905×111 9.38 KB

even with this…

  {% if value_json.action is defined and value_json.action != ""  and value_json.action != "None" and value_json.action != "null" %}

it does however get rid of “” so the sensor/template does work.

Other ideas?

10

My solution would be specifically look for the states you do want, rather than the states you don’t want:

{{ value_json.action if value_json.action is defined and value_json.action in ['lock','unlock'] else this.state }}

This a one liner, and gets rid of all the multiline if stuff. Just put the states you are interested in, in the [ array ], and if the state is not one of those, it will just return the previous state.

11

Thanks, this will work where states are known but in other cases, pincode for instance, this won’t be possible. I will try other exclusion types, we must be missing something obvious. Otherwise, maybe it’s a timing issue, since the state change seems to be quite rapid. even then, all states are captured by the original Z2M sensor so I doubt timing is the issue…

image

Since the “None” status is the most common/current state, is there a good way with the template editor to extract all it’s characteristics to double check if we’re missing something?

12

Out of interest, did you try going to the settings for that device in Z2M?

By default the debounce setting is not set.

image
image1228×363 8.02 KB

If you set this to 2, it will prevent Z2M sending multiple messages a second. Hopefully this would allow the first state change through, without following it up with the null action.

13

Perfect :+1:

Wasn’t aware of these Z2M options; you can also filter with regex so everything can be shaped upstream so no need for the secondary sensor. case closed.

… but still puzzled why the “None” will make it through the HA sensor template :thinking:

1 Like
14

We’d need to see a stream of all the MQTT messages between a real action and the updates that followed. To have a chance of maybe figuring out what update triggers None.

15

mmm. spoke a bit too soon.
The Regex filter seems to only filter out whole attributes, not payload contents.
And the “None” state is also sent by itself after a few minutes so the debounce doesn’t work there. will have to investigate more to figure out why this null state is sent in the first place or how to filter it out…

image

16

I have an idea, I will create a trigger template sensor with a state trigger (from the Z2M sensor) which excludes the “None” and “” states. stay tuned.

1 Like
17

The issue has been identified before, but unresolved:

still trying to get the trigger/sensor template sensor to work…

18

Looking at those MQTT payloads that are sent - it should definitely only need:

{{ value_json['action'] if (value_json['action'] is defined and value_json['action'] != '') else this.state }}

That should be literally the only thing needed. Note I have moved from value_json.action to the index way of accessing variables, which I prefer to use whenever I can, to ensure that I am not confusing things by referencing a built in function or something.
Also note I don’t know if this is available to MQTT sensors, it definitely is for trigger based sensors though, which I mistakenly thought were being used. IF you are using MQTT based sensors, and not trigger based template sensors, then just replace this.state with states('sensor.my_sensor')

19

Still getting “unknown” states…

image

under mqtt.yaml:

  - name: "DOORLOCK front action z2mtag666"  
    unique_id: doorlock_front_action_z2mtag666
    state_topic: "zigbee2mqtt/DOORLOCK FRONT"  
    value_template: >-
      {{ value_json['action'] if (value_json['action'] is defined and value_json['action'] != '' and value_json['action'] != 'unknown') else states('sensor.doorlock_front_action_z2mtag666') }}
    <<: *doorlockfront

Z2M debug log:


 
Info 2024-01-13 15:34:48MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT', payload '{"action":"pin_code_added","action_source":1,"action_source_name":"rf","action_user":3,"auto_relock_time":1007,"battery":37,"battery_low":true,"linkquality":193,"lock_state":"locked","pin_code":null,"sound_volume":"high_volume","state":"LOCK","users":...
Info 2024-01-13 15:34:48MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT', payload '{"action":"","action_source_name":null,"action_user":null,"auto_relock_time":1007,"battery":37,"battery_low":true,"linkquality":193,"lock_state":"locked","pin_code":null,"sound_volume":"high_volume","state":"LOCK","users":...
Info 2024-01-13 15:34:48MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT/action', payload 'pin_code_added'

Info 2024-01-13 15:35:00MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT', payload '{"action":"pin_code_deleted","action_source":1,"action_source_name":"rf","action_user":3,"auto_relock_time":1007,"battery":37,"battery_low":true,"linkquality":193,"lock_state":"locked","pin_code":null,"sound_volume":"high_volume","state":"LOCK","users":
Info 2024-01-13 15:35:00MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT', payload '{"action":"","action_source_name":null,"action_user":null,"auto_relock_time":1007,"battery":37,"battery_low":true,"linkquality":193,"lock_state":"locked","pin_code":null,"sound_volume":"high_volume","state":"LOCK","users":
Info 2024-01-13 15:35:00MQTT publish: topic 'zigbee2mqtt/DOORLOCK FRONT/action', payload 'pin_code_deleted'
20

Wait…
Why aren’t we using that?
All you want is the actual action right?

{{ value if value is defined else states('sensor.doorlock_front_action_z2mtag666') }}