Filter Duplicate Values From Incoming MQTT Topic

[kc3]

I have rtl_433 listening to temperature & humidity updates from an Acurite 609TXC and Inkbird IBS-P01R sensors and sending them to the Mosquitto broker running as an HA add-on:

 rtl_433 -C customary -F "mqtt://1.2.3.4:1883,user=aaa,pass=bbb,events=rtl_433[/model][/id]"

The Inkbird is working perfectly – it sends a single update every couple of minutes and it’s showing up in the HA sqlite3 DB and in InfluxDB. I’m using this in my configuration.yaml:

sensor:
  - platform: mqtt
    state_topic: "rtl_433/Inkbird_ITH-20R/23456"
    name: "Inkbird IBS-P01R Temp F"
    unit_of_measurement: "°F"
    force_update: true
    value_template: "{{ value_json.temperature_F | round (1)}}"

I have a very similar setup for the Acurite:

sensor:
  - platform: mqtt
    state_topic: "rtl_433/Acurite-609TXC/123"
    name: "Acurite-609TXC Temp F"
    unit_of_measurement: "°F"
    force_update: true
    value_template: "{{ value_json.temperature_F | round (1)}}"

But once a minute I get a “burst” of 6 updates with timestamps that are essentially the same (off by milliseconds). That’s quickly adding a ton of data to both sqlite3 and InfluxDB (especially because I have multiple of these reporting multiple values each). I have been searching for a way to filter those down to one a minute (or even less), but I’m having trouble finding a solution. I tried a condition on the platform: mqtt a number of different ways, but I get an error from the config validator that conditions aren’t support there. I tried adding “if” logic to my value_template but all that does is put undefined into the DB and InfluxDB.

Is there a a way to filter incoming MQTT data like I’m trying to do?

Many thanks!
kc3

[Troon]

Filter

In particular, the Time Throttle filter. You’ll still need to receive the multiple messages in the “raw” sensor, but your filter sensor will only record the ones you want.

You can then set the system up to not record the “raw” sensor [docs, see exclude].

[123 (Taras)]

@kc3

But once a minute I get a “burst” of 6 updates with timestamps that are essentially the same (off by milliseconds). That’s quickly adding a ton of data to both sqlite3 and InfluxDB

That’s odd. An MQTT Sensor is designed to not update its state if the received payload value is the same as its current state.

I confirmed it using the following simple experiment.

I created this MQTT Sensor:

- platform: mqtt
  name: test_mqtt
  state_topic: temp
  value_template: "{{ value_json.temperature }}"

Then I published this payload to the temp topic:

{"temperature": 23}

I re-published it several times then checked the SQLite database. In its states table, it contains only two records for sensor.test_mqtt:

• the initial unknown value
• the value of 23 (just once).

I repeated the test after modifying the MQTT Sensor’s template:

  value_template: "{{ value_json.temperature | round(1) }}"

Publishing {"temperature": 23.1} and {"temperature": 23.2} created two additional records. However, publishing {"temperature": 23.23} and {"temperature": 23.24} did not (because their rounded values are 23.2 which is the sensor’s existing value).

I’m curious to know why you’re seeing the addition of a “ton of data”.

[kc3]

@Troon & @123 Thank you both so much. Let me play with what both of you are saying and I’ll report back. Thanks again… I really appreciate the tips! (I am still learning Home Assistant… I have been reading the manual and through various posts, but there are still many parts that haven’t quite become clear to me yet).

[kc3]

@123 Thank you again for taking the time to do that test. Here is what I’m seeing when I query the SQLite3 DB:

Does anything there explain why it’s keeping all of them?

[kc3]

@123 Is it the force_update: true in my sensor definition?

[kc3]

I guess if I comment out force_update: true then I end up with too many gaps in my data if the values don’t change very often. The ideal would be to get on update every minute. I’m curious why the de-duplication logic isn’t working in my case. I’ll keep digging, but if anyone has ideas let me know. Thanks!

[123 (Taras)]

@kc3

Is it the force_update: true in my sensor definition?

No.

I tested this MQTT Sensor configuration:

- platform: mqtt
  name: test_mqtt
  state_topic: temp
  value_template: "{{ value_json.temperature | round(1) }}"
  force_update: true

I loaded it at UTC 17:25 and published a payload to it moments later. Then I waited about a half-hour.

The only records in the database’s states table for this sensor are its initial unknown value and the first payload I published. That’s it, that’s all.

Further confirmation that it was updated exclusively when I published it (and no more) is here:

@kc3

I guess if I comment out force_update: true then I end up with too many gaps in my data if the values don’t change very often.

There’s no “gap”. The sensor reports the last different value it received. If the same value is repeatedly received, it simply reports its existing value (and the date and time of its “last_updated” property remains unchanged).

---

Post the output of whatever tool you are using that shows these multiple identical values being stored in your database.

[kc3]

@123 OK thanks re force_update: true. Any idea when I have it set to true I get the results shown here: Filter Duplicate Values From Incoming MQTT Topic

[123 (Taras)]

Post an example of a payload published to rtl_433/Acurite-609TXC/123.

[kc3]

@123 Here is an example of the payload as captured by tcpdump:

0....rtl_433/Acurite-609TXC/123{"time":"2021-04-02 00:07:20","model":"Acurite-609TXC","id":173,"battery_ok":0,"temperature_F":66.2,"humidity":40,"status":9,"mic":"CHECKSUM"}
0....rtl_433/Acurite-609TXC/123{"time":"2021-04-02 00:07:20","model":"Acurite-609TXC","id":173,"battery_ok":0,"temperature_F":66.2,"humidity":40,"status":9,"mic":"CHECKSUM"}
0....rtl_433/Acurite-609TXC/123{"time":"2021-04-02 00:07:20","model":"Acurite-609TXC","id":173,"battery_ok":0,"temperature_F":66.2,"humidity":40,"status":9,"mic":"CHECKSUM"}
0....rtl_433/Acurite-609TXC/123{"time":"2021-04-02 00:07:20","model":"Acurite-609TXC","id":173,"battery_ok":0,"temperature_F":66.2,"humidity":40,"status":9,"mic":"CHECKSUM"}

The hex values for the 0.... seem to all be: 30 aa 01 00 1a.

[kc3]

PS: A few more points just in case they are relevant:

1. I wasn’t sure if you only wanted the JSON payload from each MQTT message or the MQTT headers, so that’s why I just gave you the whole packet after the TCP header.
2. And this was one of two MQTT messages it sends each time (the second TCP packet had 2 MQTT messages in it).

Thanks again for the help!

[123 (Taras)]

Maybe a dumb question but why is rtl_433 publishing the same payload repeatedly? Those sample payloads appear to be absolutely identical so what makes rtl_433 obliged to repeat the payload? Publishing a payload once is all that is needed.

As for why Home Assistant doesn’t simply ignore the duplicates is a mystery to me. As you’ve seen from the tests I’ve performed, they confirm that duplicates are rejected; I can’t replicate your results (and I’m using 2021.3.4).

[kc3]

@123

Maybe a dumb question but why is rtl_433 publishing the same payload repeatedly? Those sample payloads appear to be absolutely identical so what makes rtl_433 obliged to repeat the payload? Publishing a payload once is all that is needed.

I assume it’s because the Acurite is sending 6 messages each time since with the Inkbird (and other devices) I only get one each time the device sends something. I’ll see if I can fire up some other SDR software to prove the theory (or have rtl_433 dump some raw data).

@123

As for why Home Assistant doesn’t simply ignore the duplicates is a mystery to me. As you’ve seen from the tests I’ve performed, they confirm that duplicates are rejected; I can’t replicate your results (and I’m using 2021.3.4).

Yeah, makes sense. I’m confused too. I definitely appreciate the help! Thank you so much.