Is it possible to force Zigbee2Mqtt to trigger state_changed (or similar) even if sensor value did not changed?

ldoppea · October 10, 2023, 11:08pm

Hi,

I’m have some sensors connected to Zigbee2Mqtt that are transmitted to HASS using Mosquitto Broker.

Those sensors send their value to Zigbee2Mqtt multiple time per hour. For example:

2023-10-10 21:31:01MQTT publish: topic 'zigbee2mqtt/sonoff_snzb_02d', payload '{"battery":100,"humidity":53.8,"linkquality":228,"temperature":23.9}'
2023-10-10 21:41:01MQTT publish: topic 'zigbee2mqtt/sonoff_snzb_02d', payload '{"battery":100,"humidity":53.8,"linkquality":228,"temperature":24.0}'

Each message contains the entire device state but only changed values are transmitted to HASS

In previous example, at 21:41 a state_changed event is sent to HASS containing only the new temperature, but the humidity is not updated because it is the same as in previous message (I observed this in the developer tools > events page).

This as three impacts:

First the last updated value is not updated for a sensor (it is always the same as last changed) until this sensors’ value changes and triggers a state_changed event. Here the same device has 2 different last updated values even if every payload contains both sensors values. Sometimes values are not updated for many hours
Second, when the value changes, if the change is big enough, then graphs draw a straight line from previous values many hours ago to the new value. In the following example the graph suggests that the line growth from 12:00 to 21:00, but the value was still 0 at 20:59
Third, if none of the sensor’s value changes for many hours, my dashboard may suggest that the connection was lost because no update was received for too long even if zigbee2mqtt received many messages from the device during that interval

Is there a way to update the entire device state on each zigbee2mqtt event? Or am I misunderstanding how it works? (I’m pretty new at using HASS)

dproffer · October 11, 2023, 1:51am

Welcome!

I’m not sure if I am tracking with your issue or not, that said I tried mightily with zigbee temperature and humidity sensor and gave up to use low cost BLE sensors. My assessment is that zigbee sensors choose to minimize transmissions (read this as temperature readings) in order to maximize battery life. If you search this forum and perhaps the zigbee2mqtt forum, you might see some folks that have found zigbee sensors that allow some customization of their reading frequency. But from my memory, it was still limited to values that IMHO are not useful for home automation control.

So I send all my zigbee temperature sensors to the ‘dead technology drawer’ and moved to these low cost BLE sensors. The battery life thing, I get 1 to 2 years battery life from my BLE sensors. I guess some of these zigbee sensors may go 5 to 10 years, however what good is that if they do not yield useful data? Some of the zigbee sensors do seem to wake and sense more often when larger changes in short period occur, but it seems a ‘roll of the dice.’ one reading every 5, 10 or more minutes apart is not useful.

Below is a link to some of the testing of sampling rates I did on some BLE and Wifi sensors, I had already given up on zigbee when I did this test.

However since apparently I like to waste money, 6 or so months ago I ran across this zigbee sensor on Aliexpress, link below. However, as is shown in the chart picture below the lack of updates from this zigbee (18 minutes ago was the last update as I write this) vs. as is shown in my BLE study, I get hundreds of ‘readings’ from the BLE sensors (do note as I wrote, you still do not know how may actually readings the device is doing, but as is shown a number of the BLE sensors give 10’s if not 100’s per hour). To be clear as is shown in the graphs in my study, even though most all of the BLE sensors are sending a message 100’s of times per hours, some of the BLE sensors still only sampled there sensors at a very much lower rate. The ‘chunky’ graphs show this.

To make matters worse for this zigbee sensor, the real reason I purchased was that I have always wanted a good temp and humidity sensor with a clock. Well guess what? The frickin device does not even keep time!!! I reset it about a month ago because the time was wrong and it is now currently 2.5 hours behind .

Hope you can find a solution to your requirement in the zigbee space ( I have built custom zigbee sensors where I control the reading and reporting rates) using this very nice firmware Zigbee Configurable Firmware (ptvo) link below, but that requires some DIY work. That said, I highly recommend the BLE route, if you search this forum you will find a number of ways to incorporate BLE sensors that cover a large physical space.

Good hunting!

https://www.zigbee2mqtt.io/devices/SZ-T04.html#nous-sz-t04

https://ptvo.info/

mekaneck · October 11, 2023, 2:33am

I don’t use zigbee2mqtt but since it uses MQTT, there is an option that can be set per sensor in the MQTT payload to force sensor data update even if the data hasn’t changed. See force_update

If Z2M works by using HA’s mqtt auto discovery feature, then you need to post the force_update payload to the discovery topic for that sensor. If the retain flag is not set, you’ll have to post this on an automation that runs at every HA startup.

MQTT is one of the few integrations that support the force_update option so you are fortunate. The root cause is just an artifact of the strategy that HA uses to record data in the database. If a state (or attribute) doesn’t change, an update is not recorded, and therefore HA will say there has been no update. And there is no way to distinguish a sensor that hasn’t reported a value for 10 hours compared to one which has been reporting the same value every 2 minutes for 10 hours.

ldoppea · October 11, 2023, 9:39am

Hi @dproffer ,

My problem is not on the Zigbee side as my sensors correctly send their values when they changes or, if no changes, every 15 min.
My problem is that if they send identical values, then those values are not reported in the HASS device (Zigbee2Mqtt receive them but don’t send corresponding state_change event).

I heard that some devices allow to configure their minimum and maximum update delay, did you try that on your devices?

ldoppea · October 11, 2023, 9:50am

@mekaneck this seems to be a good solution. Thanks.

However I didn’t find yet how to do this.

If I’m using the discovery topic, should I just create a payload with only the device ID and the force_update value? Or should I put the entire device configuration?
If the answer is to send the entire device configuration, is there a way to get the current registered configuration JSON somewhere?

As an alternative I found that there is a customization mechanism using customize.yaml file. I tried to create it and declare it in the configuration.yaml. But this seems to have no effect.

Here is the configuration.yaml content:

homeassistant:
  customize: !include customize.yaml

Here is the customize.yaml content:

sensor.tuya_power_01_power:
  force_update: true

Then I executed the Reload customization command:

For now this does not trigger any update event. I’m still reading some documentation on the subject. Maybe customizations cannot impact the force_update setting.

mekaneck · October 11, 2023, 11:29pm

I think the customize should work but fairly certain you’ll need to reboot HA for the changes you made to work.

ldoppea · October 13, 2023, 12:13pm

I just found this github issue and it seems like force_update can work only for binary sensors: Questions about inconsistent implementation of 'force_update' attribute in MQTT · Issue #93380 · home-assistant/core · GitHub

ldoppea · October 13, 2023, 9:03pm

Ok, this was a false assumption, I just found how to do it.

For people in the same situation, here is how to do it.

What did not work:

Using the customization mechanism did nothing
Trying to post a discovery topic from the Developper tools > Events menu did nothing
Trying to post a discovery topic from the Settings > Devices & Services > MQTT > Configure page did nothing

What did work:

Download MQTT Explorer (or equivalent): http://mqtt-explorer.com
Connect to HomeAssistant
- The IP adress is the one from HomeAssistant
- The port is the one you can find in MQTT the network section of Mosquitto broker addon
- For me user and password were the one from HomeAssistant
The list of MQTT topics should appear
Find your device in homeassistant/sensor/<your_sensor_id>
Find and select the entity you want to force update
Then on the right pane, copy the current Value (it should be JSON)
Paste it on the Publish payload, and add "force_update": true, in the JSON
Check the retain option
Click publish
Go to Settings > Devices & Services > MQTT > Devices > your_device > MQTT INFO and check that the corresponding entity now has the force_update field

Now my entity is correctly updated even if its value is the same as previous one.

Thanks a lot @mekaneck your solution was the correct one, I was just too newbie to understand how to apply it

ldoppea · October 15, 2023, 4:45pm

I just found that my previous configuration would be lost on HASS restart

So I made a NodeRED flow to automatically add force_update on expected entities

The MQTT IN node listen for any homeassistant/sensor/+/+/config topic

Then the Function node checks for every configured entities if they have the force_upload: true member. Otherwise it adds it to the topic’s payload + add the should_reconfig tag so the next node knows if it should resend the modified topic back to MQTT or ignore it.

Here is the Function content:

const topics = [
    // Topics corresponding to my Power plug entities
    // Replace `<some_device_id>` by your device's id
    'homeassistant/sensor/<some_device_id>/power/config',
    'homeassistant/sensor/<some_device_id>/current/config',
    'homeassistant/sensor/<some_device_id>/voltage/config',
]

if (topics.includes(msg.topic)) {
    if (msg.payload.force_update !== true) {
        msg.payload.force_update = true;
        msg.should_reconfig = true;
    }
}

return msg;

The Switch node just check if the should_reconfig tag is present

If present, the msg payload is passed to the next MQTT OUT node. Only the retain option is set. Topic and Payload left empty so the node takes the ones from previous nodes

With that workflow, the force_update is applied on HASS restart but also on every new discovery message (i.e. if you re-pair a device)

Jay_from_NuZiland · March 10, 2024, 4:28am

There is two other working options without using NodeRed, although one relies on using Zigbee2MQTT (I’m not sure if there is a ZHA equivalent). Posting here in the hope that it helps someone in the future given that the question has been raised in a few different places without good answers using native components/options.
From the HA side, it is possible to manually define the mqtt sensor (rather than rely on discovery) by replicating the discovery payload data into configuration.yaml. This makes future changes a little less easy, but works fine. Get your current entity discovery data by navigating to “Settings > Devices & Services > MQTT > Devices > your_device > MQTT INFO”, and use the data there to build out your manual sensor config and add the force_update option.
From the Z2M side, the process described in Home Assistant | Zigbee2MQTT can be used to add a per-sensor force_update: true setting to the z2m configuration.yaml. Because only this one option is set in the file, all GUI-controlled settings remain editable. Looks like this;

  '0xa4c1384cc0c2d3c2':
    friendly_name: Washing Machine Plug
    description: ''
    homeassistant:
      power:
        force_update: true