Expiration of mqtt entity

maxym · May 18, 2022, 8:19am

After the behavior of MQTT entities expiration has been fixed I decided to give a chance to expiration settings set for battery-powered devices.

Here is a snippet of one of my entities:

- platform: mqtt
  name: "Barus 1 Window State"
  state_topic: "shellies/shellydw2-barus1/sensor/state"
  availability:
    - topic: "shellies/shellydw2-barus1/online"
      payload_available: "true"
      payload_not_available: "false"
  payload_on: "open"
  payload_off: "close"
  device_class: "window"
  expire_after: 90000

I reloaded MQTT entities at about 00:45. At 07:35 it went unavailable.
What’s wrong with that?

Coincidently, I did use ‘Listen to the topic’ available in Mosquito Broker settings (official add-on) to check the value of the availability topic. Using it has fixed the entity state immediately
I had 3 such entities and this phenomenon repeated for each of them.

Can anyone explain why it happened?
The only explanation I think of is, that:

HA didn’t register new (availability) topic for listening after I configured entities and reloaded them
‘Listen to the topic’ feature uses the same mqtt client connection as HA does. So attempt to listen to it, loaded data needed for HA entity

It still doesn’t answer the question why the entity went unavailable after about 7 hours

maxym · May 18, 2022, 11:01am

And after about 2h 20 minutes one of those entities went unavailable again. Why?
And again resurrected by enabling listening to its availability topic in Mosquito config.

Just curious if battery-powered devices need to use the availability topic at all.

Anyway I have enabled debug logging for core and mqtt components

maxym · May 21, 2022, 3:06pm

Probably I found the answer. Actually two answers
At first: for battery-powered devices, I should not use the availability topic. Shelly sets it to false when going to deep sleep. I found it sets it to true later (after some hours), cannot get the reasons, but anyway, I should use expire_after without the availability topic.

The second issue lies down probably in Shelly recent firmwares. I found similar behavior in DW and DW2 with firmware v1.11.7 and 1.11.8. The shelly provides a setting to ensure a periodical refresh of mqtt topics. Unfortunately for some reason, it doesn’t refresh the state topic. It refreshes battery and tilt topics (if tilt sensor is enabled) but never the state one. So if a window is not used for some time, it properdly reports unavailability.

Here is debug log for core and mqtt components:

2022-05-20 08:27:19 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=binary_sensor.pcroom_window_state, old_state=<state binary_sensor.pcroom_window_state=on; device_class=window, friendly_name=PCroom Window State @ 2022-05-20T08:16:26.617687+02:00>, new_state=<state binary_sensor.pcroom_window_state=off; device_class=window, friendly_name=PCroom Window State @ 2022-05-20T08:27:19.147289+02:00>>
2022-05-20 08:27:19 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on shellies/shellydw-pc/sensor/battery: b'100'
2022-05-20 08:29:16 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=binary_sensor.pcroom_window_state, old_state=<state binary_sensor.pcroom_window_state=off; device_class=window, friendly_name=PCroom Window State @ 2022-05-20T08:27:19.147289+02:00>, new_state=None>
2022-05-20 08:29:18 DEBUG (MainThread) [homeassistant.components.mqtt] Unsubscribing from shellies/shellydw-pc/sensor/state, mid: 4202
2022-05-20 08:29:20 DEBUG (MainThread) [homeassistant.components.mqtt] Unsubscribing from shellies/shellydw-pc/sensor/battery, mid: 4344
2022-05-20 08:29:20 DEBUG (MainThread) [homeassistant.components.mqtt] Subscribing to shellies/shellydw-pc/sensor/state, mid: 4372
2022-05-20 08:29:20 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on shellies/shellydw-pc/sensor/state (retained): b'close'
2022-05-20 08:29:20 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=binary_sensor.pcroom_window_state, old_state=None, new_state=<state binary_sensor.pcroom_window_state=off; device_class=window, friendly_name=PCroom Window State @ 2022-05-20T08:29:20.435033+02:00>>
2022-05-20 08:29:23 DEBUG (MainThread) [homeassistant.components.mqtt] Subscribing to shellies/shellydw-pc/sensor/battery, mid: 4458
2022-05-20 08:29:23 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on shellies/shellydw-pc/sensor/battery (retained): b'100'
2022-05-20 19:43:20 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on shellies/shellydw-pc/sensor/battery: b'100'
2022-05-21 07:03:31 DEBUG (MainThread) [homeassistant.components.mqtt] Received message on shellies/shellydw-pc/sensor/battery: b'100'
2022-05-21 09:29:20 DEBUG (MainThread) [homeassistant.core] Bus:Handling <Event state_changed[L]: entity_id=binary_sensor.pcroom_window_state, old_state=<state binary_sensor.pcroom_window_state=off; device_class=window, friendly_name=PCroom Window State @ 2022-05-20T08:29:20.435033+02:00>, new_state=<state binary_sensor.pcroom_window_state=unavailable; device_class=window, friendly_name=PCroom Window State @ 2022-05-21T09:29:20.435968+02:00>>

For sensor above I have set expirer_after to 25hours (90000 secs) which matches results found in the log.

Of course I’m going to report this issue to Shelly. Just letting you know if you are using mqtt for Shelly devices. BTW… how many users use that config? I can see no similar reports at all. Maybe nobody uses expiration settings…

xJavix · March 26, 2024, 6:19am

Hi @maxym,

I have not gone to debug mode, but I was wondering if you had have more issues with the door sensor configuration?

I have problems getting the values from the broker to home assistant.
Here is my config file of my door sensor:

## /sensors/doors/frontdoor.yaml
- name: "Front door - lux"
  unit_of_measurement: 'lm'
  expire_after: 86400
  device_class: illuminance
  #icon: mdi:theme-light-dark
  qos: 0
  state_topic: "shellies/shellydw2-xxxx/sensor/lux"
- name: "Front door - tilt"
  unit_of_measurement: '°'
  expire_after: 86400
  icon: mdi:format-rotate-90
  qos: 0
  state_topic: "shellies/shellydw2-xxxx/sensor/tilt"
- name: "Front door - battery"
  unit_of_measurement: '%'
  expire_after: 86400
  device_class: battery
  #icon: mdi:battery
  qos: 0
  state_topic: "shellies/shellydw2-xxxx/sensor/battery"

and here for my binary sensor

## /binsensors/doors/frontdoor.yaml
- name: "Front door - Status"
  value_template: "shellies/shellydw2-xxxx/sensor/state"
  expire_after: 90000
  device_class: door
  payload_on: "open"
  payload_off: "close"
  qos: 0
  state_topic: "shellies/shellydw2-xxxx/info"
- name: "Front door - Vibration"
  expire_after: 90000
  device_class: vibration
  payload_on: 0
  payload_off: 1
  qos: 0
  state_topic: "shellies/shellydw2-xxxx/sensor/vibration"
- name: "Front door - Firmware"
  state_topic: "shellies/shellydw2-xxxx/announce"
  value_template: "{{ value_json.new_fw }}"
  payload_on: true
  payload_off: false
  expire_after: 90000
  qos: 0

I can get the topics updated on mqtt explorer but not in home assistant.

While another shelly sensor uses mqtt, and the status is displayed, (motion sensor), i cannot get why this door sensor does not show up in home assistant.

# Motion 2 Shelly
- name: "Stairs Motion - status"
  state_topic: "shellies/shellymotion2-YYYY/status"
  value_template: "{{ value_json.motion  }}"
  payload_on: true
  payload_off: false

Any idea?
Maybe I should try the json announce from the door sensor instead?

maxym · April 13, 2024, 6:50am

No, not at all.
BTW one of my door-window v1 sensor, mounted on doors to the garden has registered 4700 state changes for more than 3 years without battery change. And still counting

Here is its yaml:

pck_cube_door_garden:
  mqtt:
    binary_sensor:
      - name: "Cube Door Garden state"
        state_topic: "shellies/shellydw-door-cube-garden/sensor/state"
        payload_on: "open"
        payload_off: "close"
        device_class: "window"
    sensor:
      - name: "Cube Door Garden Battery"
        state_topic: "shellies/shellydw-door-cube-garden/sensor/battery"
        unit_of_measurement: "%"
        force_update: true
        device_class: battery
      - name: "Cube Door Garden Tilt"
        state_topic: "shellies/shellydw-door-cube-garden/sensor/tilt"
        unit_of_measurement: "degrees"

Just by comparing our source code, it seems to me the difference is in the value of payload_on and payload_off.

xJavix · June 9, 2024, 5:52am

Thanks. I will give it a try once i resurrect my server . I had an issue with my server and at the moment my backup is running in a HAOS and no mqtt broker is set up there.