Is_defined filter with expire_after on unconsistant MQTT messages

[hygap]

HI

Below is the simplified config for a sensor using messages sent from an ESP32 board running OpenMQTTGateway

- platform: mqtt
  name: "temp_sensor"
  state_topic: "sensors/+/BTtoMQTT/UIDHERE"
  value_template: "{{ value_json.tem | is_defined }}"
  expire_after: 120

The JSON payload sent to this topic is not consistent. On some messages it contains the JSON field tem on the rest it does not.

From my understanding using the is_defined filter causes the sensor to report the last known value if the payload does not contain the tem field. This is good because it stops the sensor flipping from the actual value to zero every time a message is received without the tem field. This gives a constant view in any dashboard.

The expire_after: 120 only triggers an Unavailable status if no message is received to this topic after 120 seconds rather than if no message containing the field tem is received within 120 seconds. I believe this is because if a message to this topic is received within the 120 that does not contain the tem field the is_defined filter is substituting the past value which is in turn is satisfying the expire_after check.

I would like to know how to rewrite this config to show an Unavailable value if no message is received containing the field tem within the time frame set by expire_after. This must not be at the expense of setting the sensor value to zero if a message is received which does not contain the required field.

Thanks for any help in advance.

[123 (Taras)]

@hygap

expire_after: 120 only triggers an Unavailable status if no message is received to this topic after 120 seconds rather than if no message containing the field tem is received within 120 seconds

Correct.

expire_after isn’t concerned with what is in the received payload only if a payload is received (within the 120 sec interval).

The fact that the received payload lacks a tem key is of no concern to expire_after. It received a payload within 120 seconds (with or without a tem key) and that’s sufficient for it to reset its countdown.

You want expire_after to report Unavailable if the payload lacks the tem key (and 120s have passed). I don’t know of any configuration option to change the behavior of expire_after.

The only way I know how to achieve your goal is to duplicate the functionality of expire_after using a timer and automations. The automation serves as a middleman, receiving the payload directly from OMG. If the payload contains the tem key, it resets the 120-second timer and publishes the value to a topic subscribed by sensor.temp_sensor. I won’t explain the rest because I suspect you probably don’t want this complicated proposal.

Question: where did you see the is_defined filter documented? I wasn’t able to find it either Jinja2’s or Home Assistant’s documentation. It behaves as you described but I’d like to read its official description.

[hygap]

Thanks for the quick reply Taras

I found three references to is_defined.

The first as a solution posted to the zero flipping behavior described here:

Problem with OpenMQTTGateway / MQTT Sensor and Xiaomi MIJIA

Hey Guys, i’m using OpenMQTTGateway to read the temperature and humidity from Xiaomi Mijia hygrothermometer and reading it in homeassistant as mqtt sensor. Configuration: - platform: mqtt name: "Temperatur Baby" state_topic: "home/OMG_Baby/BTtoMQTT/582D3431C7BF" unit_of_measurement: '°C' value_template: "{{ value_json.tem }}" - platform: mqtt name: "Luftfeuchtigkeit Baby" state_topic: "home/OMG_Baby/BTtoMQTT/582D3431C7BF" unit_of_measurement: '%' …

Second in the OpenMQTTGateway documentation on the final example here:

Integrate Home Assistant | OpenMQTTGateway v1.7.0

Lastly and probably what you are looking for in the initial commit to HA:

Fail when rendering undefined objects in Jinja2 templates

I could find very little documentation elsewhere. Which is a shame as it seems a quick & concise way of solving the zeroing sensor issue I described.

Lastly I would say that if what you say is true about expire_after then the documentation on HA could do with updating as it defines it’s behavior as

Defines the number of seconds after the sensor’s state expires, if it’s not updated. After expiry, the sensor’s state becomes unavailable

Which is not strictly true. It should read

Defines the number of seconds after which the sensor’s state expires, if no new message is received on the state_topic. After expiry, the sensor’s state becomes unavailable

Because it is not the state that is being examined, rather the receipt of a MQTT message to that topic.

MQTT Sensor

[123 (Taras)]

Thanks for the links. Very interesting. Aside from that PR from 2016, there’s nothing in the documentation for is_defined.

In addition, it behaves oddly when used in the Template Editor. If given a defined variable, it works as you described:

If given an undefined variable, it reports an error:

Nevertheless, it seems to function correctly when used in a sensor’s value_template.

[hygap]

It seems the core-2022.2.0 may have broken the is_defined functionality.

I have posted a bug report

As you @123 seem to have a good knowledge of the templating engine, I was wondering if you had an alternative solution to achieve the same result

[123 (Taras)]

Use defined.

[hygap]

So i think I can get it working again by changing

"{{ value_json.tempc | is_defined }}"

to
{% if value_json.tempc is defined %}{{ value_json.tempc }}{% else %}{{ states('sensor.plant_1_temp') }}{% endif %}

Seems to work but not quite as concise

[123 (Taras)]

@hygap

not quite as concise

Here are two other ways to do it that are more concise:

{{ value_json.tempc if value_json.tempc is defined else states('sensor.plant_1_temp') }}

{{ iif(value_json.tempc is defined, value_json.tempc, states('sensor.plant_1_temp')) }}

For more information, refer to Immediate If

[hygap]

@123

{{ iif(value_json.tempc is defined, value_json.tempc, states('sensor.plant_1_temp')) }}

Much appreciated