MQTT Event keeps coming back as 'Retained' - how to turn that off?

[123 (Taras)]

@xAPPO

Shouldn’t be needed

I think the OP will agree to that statement because they claimed they disabled the retain flag at the source. Yet here we are.

However, what the OP is overlooking is that instructing the Octopi to cease publishing payloads as retained messages doesn’t change the fact that it had already published several as retained messages. That topic will continue to retain payloads published to it unless it is purged first (i.e. publish an empty string as a retained message). That’s just the way MQTT’S payload retention policy works.

Combine that with when MQTT Explorer’s does/doesn’t indicate a retained message and it explains everything the OP has reported (including the seemingly mysterious and magical appearances of RETAIN in Explorer).

[DaHai]

@123

That topic will continue to retain payloads published to it unless it is purged first (i.e. publish an empty string as a retained message).

And this doesn’t do that??
I done this again and again and again as everyone has told me to do again and again and again and posted here ad nauseum. Is there some Other way to Purge?? (ie pubish an empty string as a retain message)??

[DaHai]

@123

MQTT Explorer reports if a message is retained in a manner that may be misleading you. When it’s connected to the broker and you publish a retained message to a topic, MQTT Explorer will not report it’s retained. If you disconnect MQTT Explorer from the broker and then reconnect, now it will show the topic has a retained message.

Yes, MQTT does report messages retained messages as RETAINED.
Before I even started this post, I saw that these topics where being posted by OctoPi as Retained (BIG YELLOW ‘RETAINED’ in MQTT Explorer. Went over to the OctoPi MQTT config and turned OFF the Retain Flag. Once that was done and the topic was purged, those message sent by OctoPi no longer had the retain Flag (NO BIG YELLOW ‘RETAINED’) in MQTT Explorer.

[DaHai]

@123

I think the OP will agree to that statement because they claimed they disabled the retain flag at the source. Yet here we are.

And yet, after purging that topic with a empty payload (please read my posts), the only time that topic reapears as RETAINED is when MQTT is restarted and OctoPi is POWERED OFF.
Please tell me why, after purging that topic with an empty payload and Retain Flag On (as repeated instructed to do, and I have repeated done so), does that OLD topic reappear ONLY when MQTT is restarted?
Aparently MQTT doesn’t actually Purge?
Can you explain that?

[DaHai]

Ok, I really do appreciate everyone’s attempt to help with this.
But the fact of the matter, which I think has been lost, is this:

Even after purging a Topic in Mosquitto MQTT Broker, when MQTT restarts, that old topic is still there. And because of that my Automation (shown above in a previous post) gets executed and turns off the corresponding switch.

So Plane and Simple for Everyone: MQTT never actually Purges

[lmamakos (Louis Mamakos)]

Just for clarity, the retention of MQTT messages that are published with the “retain” flag set true is in the MQTT broker, not in Home Assistant. When such a message is published to a topic, the MQTT broker will persist it. When a new client connects to the broker and subscribes to that topic, it will get sent any retained messages that match the subscription.

As @123 pointed out, you can tell the broker to throw away the retained message by publishing a message to that topic with an empty payload (not “blank” characters or whitespace) with the “retain” flag set. That will remove it from the persisted storage in the broker.

If you can seem to make that work, you could try to stop the broker and look for the database that (presumably mosquitto) uses to store persisted messages across restarts and delete that database. This is fraught with peril since it’s a pretty blunt instrument. In my mosquitto container, I have an external volume that’s mounted inside the container at /mosquitto/db and there is a single file in there mosquitto.db where I believe these messages are stored.

However, I just tried this using the Home Assistant MQTT publish service. I think you want to do this:

Put the topic you want to clear the retained message for in the topic field of the form. Then make sure:

• The payload box is unchecked
• The retain box is checked
• The retain switch to the right is turned on
  Then punch that “Call Service” button and that should publish a retained, null payload message to that topic and clear any retained message at the broker.

[DaHai]

@lmamakos

However, I just tried this using the Home Assistant MQTT publish service. I think you want to do this:

Nope (and if you read back in my previous posts I also tried this with the ‘Payload’ checked but nothing in the entry.

[lmamakos (Louis Mamakos)]

I’m not sure what to suggest; I don’t use MQTT Explorer so I can’t confirm the topic name or know I’m interpreting what I see correctly. I suppose if that’s what you have configured for your sensor, then it’s probably a match.

Are you using the Home Assistant companion application on a Mac? I’ve noticed that if you copy and paste stuff into it, it includes leading whitespace. Is is possible you pasted the topic name into that field and it has a leading space character? This is a long shot.

Just restarting Mosquitto won’t clear anything; it has a database on disk that it stores persistent messages to so that they survive restarts of the broker. I don’t run HassIO, er, “Home Assistant OS” so I don’t know if you can find the persistent volume with the database and delete it, without having to just remove that mosquitto container entirely and reinstall it.

[lmamakos (Louis Mamakos)]

If you have the ability to run a python script, then I have a tool that you can use to purge the retained messages on topics. You’ll need to install the paho-mqtt python library to do this. (Doing something like pip install paho-mqtt.) Here’s a script: https://www.transsys.com/stuff/watch.py and you might edit the beginning to change the default MQTT broker address and stuff.

Of course, you can run this from anywhere you have a python environment that can connect to your MQTT broker.

So you should be able to do something like

python watch.py --broker 192.168.0.42 --retained AuroraA4/#

and see messages published to that (wildcard) topic. (The # is a wildcard match for anything that follows.) Normally, the script won’t print any messages with the retained flag, unless you pass that --retained option. So you should see the messages with with (retained) right after the topic name immediately after the script connects to the broker.

The retained flag indicates that you received it because it was previously published and waiting for you. If another message is published with the retain flag set, current subscribers won’t see that indication delivered to them.

Anyway.

To purge those messages, kill/interrupt the script and run it again with the --purge-retained option added and it will delete the retained messages it receives when the script subscribes to the topics on the broker.

Here’s an example, with the topic foo on my MQTT broker.

louie@kzin[84] $ ./watch.py --retained foo
[2021-04-21 23:21:17] foo(retained): bar
^C
louie@kzin[85] $ ./watch.py --purge-retained --retained foo
[Purging retained message on 'foo']
[2021-04-21 23:24:12] foo(retained): bar
[2021-04-21 23:24:12] foo:
^C
louie@kzin[86] $ ./watch.py --purge-retained --retained foo
^C
louie@kzin[87] $

The first invocation subscribes to the foo topic and shows there’s a retained message present.

Then the next with the purge option and an indication that each message it discovers is purged. You’ll also see the null message that was published to the topic.

Finally, I kill that and run it again to show the retained message is gone.

I normally use this script just to watch MQTT traffic on my broker; you can subscribe to multiple topics (or topic patterns), dump it out in JSON and stuff. I just hacked in this purge thing over the last 15 minutes, so it’s a quick-and-dirty solution. Add the --help option and maybe that will answer more questions than it creates on how to use it.

[DaHai]

Thanks everyone for all your suggestions. But I’m tired of digging this rabbit hole.

Apparently MQTT Publish doesn’t actually work, so a Python script should be tried in its place.
Or just delete the whole MQTT database and start over.
No

There are 397 open issues for MQTT on GitHub, this is one of them. Seeing all those, and that it is unable perform the simple task of Honoring the Retain Flag, its clear Eclipse MQTT Broker is a product riddled with bugs and not fit for use.

I’m going to find a different process that doesn’t require MQTT or Home Assistant to accomplish this simple task.

Thanks again for everyone’s suggestions. Sorry MQTT is so far beyond hope.

[123 (Taras)]

For the folks still reading this thread, I’ve been using MQTT since 2018 and have never experienced the difficulties the OP has reported.

My introduction to MQTT began when I wanted to add MQTT support to an old home automation system. In order to do that, I had to gain a sufficient understanding of MQTT’s operation to ensure success. The resulting MQTT converter has worked very reliably and predictably for nearly 3 years now.

What the OP has experienced is anomalous. In my tests, using Developer Tools > Services > MQTT: Publish, to purge a retained message, does indeed work; the topic’s payload does not reappear as a retained message.

• Using MQTT Explorer, published a retained message to a new topic.
• Using Developer Tools > Services > MQTT: Publish, published an empty string, as a retained message, to the topic .
• Disconnected/reconnected MQTT Explorer and now the topic and its payload were no longer present (i.e. retained payload removed from the broker’s storage).
• Restarted the MQTT Mosquitto Broker Add-On.
• Disconnected/reconnected MQTT Explorer. Topic and its payload did not return and were truly purged from the broker’s storage.
• Using Developer Tools > Services > MQTT: Publish, published a non-retained message to the previous topic.
• In MQTT Explorer (already running) it displayed the topic and its payload (not as a retained message).

In short, everything worked as expected.

The OP carried out a similar procedure but reported a different outcome, one where the topic always contains a retained message despite attempts to disable it.

There have been reports of performance issues with the MQTT Mosquitto Broker Add-on version 5.1.1 and maybe (just maybe) this is one of its manifestations. I can’t say for sure because I still use version 5.1 and it runs flawlessly.

---

NOTE

While performing this test, I noticed a UI anomaly (in 2021.4.6) in Developer Tools > Services > MQTT: Publish. If you check the Retain option’s checkbox, its toggle button doesn’t automatically turn on. If you switch from UI Mode to YAML mode, it is clear that the Retain option is still disabled. To truly enable the Retain option, you must not only check the checkbox but also turn on the toggle button. This “double-enabling” requirement seems excessive to me and I would classify it as a “UI glitch”.

[lmamakos (Louis Mamakos)]

I’ve always had good luck with MQTT, been trouble-free for me and I have a pretty high volume of message traffic through my MQTT broker. MQTT is my method of choice when I need to integrate some random hardware I have… I can just cobble up a standalone python script that does what’s needed to poke the device. Then I know there’s an obvious way to push that data into Home Assistant with MQTT.

I posted a pointer to that script on the offhand chance that the topic in question might have contained some invisible character, either when the topic was created, or got pasted into the Home Assistant services UI. GIven the right wildcard match, it could just delete a message to any topic it encountered, even it it contained invisible characters or some wacky unicode lookalike.

I’ve found the MQTT integration method in Home Assistant really reliable and flexible; others ought not to shy away from using it. I wish we could have figured out what the underlying problem was in this particular case…

[xAPPO (Kevin Hawkins)]

@DaHai

Is there some Other way to Purge??

In MQTT Explorer you can click select the topic, or a higher hierarchy and then click the trash can icon to purge everything selected. You must then disconnect and reconnect MQTT Explorer. The topics will be gone.