So after the update that removed the option for the retain flag, your sensors started to retain their messages? Have you rolled back to a prior snapshot to see if that is the cause of your problem.?
This has nothing to do with the configuration of the two MQTT Sensors in Home Assistant and everything to do with whatever is responsible for publishing payloads to the two MQTT topics.
If “Octopi” is responsible for publishing to AuroraA4/event/Error and AuroraA4/event/Disconnected then it’s the source of the retained messages. No amount of tinkering in Home Assistant will change that.
I don’t know what “Octopi” is but if you don’t want those two topics to receive payloads as retained messages, you will have to change that in Octopi.
123 Taras, I’m sorry you missed this in my First Post on this topic, so here it is again:
I have OctoPi with MQTT. I have turn OFF the retain flag in that config so that ALL event messages have retain=false.
In MQTT Explorer. I connect to my Home Assistant MQTT and delete ALL events for that OctoPi Instance.
I turn on OctoPi, Connect to my 3D Printer and look at all the events from that printer coming into Home Assistant.
All of them have the Retain flag OFF.
No. That would mean rolling back a couple of years. If you noticed in the Discussion thread I linked to previously on the ‘retain’ flag for sensors, this was back in Feb of 2019. I’m sorry, but I don’t even have Snapshots that far back.
I’m sick and tired of working around this stupid issue of rouge ‘Disconnected’ messages shutting down my printers. This is supposed to be a safety measure for 3D Printing, but instead its a logistics nightmare of an issue that should not be happening.
Since this only happens when I reboot Home Assistant, and they are MQTT messages, I think it’s logical to point the finger at MQTT.
I just now sent a Null Payload for the topic AuroraA4/event/Disconnect with the retain flag ON (as instructed above) which cleared out that topic. Then restarted JUST Mosquitto MQTT, and that topic reappeared as ‘RETAINED’ and triggered my Automation to turn off the corresponding switch.
I have looked through all the files in Home Assistant (configuration, automations, etc.) and nothing generates that topic. The sensor looks for that topic, but it also looks for the ‘Error’ topic, which is not regenerated by MQTT. And no other devices on my network generate that topic and the only one that does is Powered OFF.
sensor A4:
- platform: mqtt
state_topic: "AuroraA4/event/Error"
name: "Aurora A4 Error"
value_template: '{{ value_json._timestamp }}'
- platform: mqtt
state_topic: "AuroraA4/event/Disconnected"
name: "Aurora A4 Disconnected"
value_template: '{{ value_json._timestamp }}'
I will also point out that the timestamps on these ‘regenerated’ topics are not current, they are the same time stamps of those previously removed through the MQTT:Publish tool as I stated above.
So, no device is sending ‘new’ Disconnect messages, MQTT is retaining them and re-populating/re-publishing in them causing that automation to trigger in Home Assistant
Here’s another walkthrough:
Current state of Home Assistant MQTT Broker as viewed in MQTT Explorer:
Click on “Call Serivce” to send this topic:
Result shown in MQTT Explorer: that topic is now GONE:
Restart MQTT Brokers:
After Restart Complete, result in shown in MQTT Explorer:
Notice that the timestamp of the topic is the same as before, but the time of the event (on the right) is that of when MQTT was restarted.
Whatever you did failed to work because it continues to publish its payloads as retained messages.
MQTT Explorer reports if a message is retained in a manner that may be misleading you. When it’s connected to the broker and you publish a retained message to a topic, MQTT Explorer will not report it’s retained. If you disconnect MQTT Explorer from the broker and then reconnect, now it will show the topic has a retained message.
If you use NodeRED (or even maybe via an HA automation) you could read this payload and republish it non retained to a different one. Shouldn’t be needed but maybe will work around your issue.
I think the OP will agree to that statement because they claimed they disabled the retain flag at the source. Yet here we are.
However, what the OP is overlooking is that instructing the Octopi to cease publishing payloads as retained messages doesn’t change the fact that it had already published several as retained messages. That topic will continue to retain payloads published to it unless it is purged first (i.e. publish an empty string as a retained message). That’s just the way MQTT’S payload retention policy works.
Combine that with when MQTT Explorer’s does/doesn’t indicate a retained message and it explains everything the OP has reported (including the seemingly mysterious and magical appearances of RETAIN in Explorer).
1 Like
And this doesn’t do that??
I done this again and again and again as everyone has told me to do again and again and again and posted here ad nauseum. Is there some Other way to Purge?? (ie pubish an empty string as a retain message)??
Yes, MQTT does report messages retained messages as RETAINED.
Before I even started this post, I saw that these topics where being posted by OctoPi as Retained (BIG YELLOW ‘RETAINED’ in MQTT Explorer. Went over to the OctoPi MQTT config and turned OFF the Retain Flag. Once that was done and the topic was purged, those message sent by OctoPi no longer had the retain Flag (NO BIG YELLOW ‘RETAINED’) in MQTT Explorer.
And yet, after purging that topic with a empty payload (please read my posts), the only time that topic reapears as RETAINED is when MQTT is restarted and OctoPi is POWERED OFF.
Please tell me why, after purging that topic with an empty payload and Retain Flag On (as repeated instructed to do, and I have repeated done so), does that OLD topic reappear ONLY when MQTT is restarted?
Aparently MQTT doesn’t actually Purge?
Can you explain that?
Ok, I really do appreciate everyone’s attempt to help with this.
But the fact of the matter, which I think has been lost, is this:
Even after purging a Topic in Mosquitto MQTT Broker, when MQTT restarts, that old topic is still there. And because of that my Automation (shown above in a previous post) gets executed and turns off the corresponding switch.
So Plane and Simple for Everyone: MQTT never actually Purges
Just for clarity, the retention of MQTT messages that are published with the “retain” flag set true is in the MQTT broker, not in Home Assistant. When such a message is published to a topic, the MQTT broker will persist it. When a new client connects to the broker and subscribes to that topic, it will get sent any retained messages that match the subscription.
As @123 pointed out, you can tell the broker to throw away the retained message by publishing a message to that topic with an empty payload (not “blank” characters or whitespace) with the “retain” flag set. That will remove it from the persisted storage in the broker.
If you can seem to make that work, you could try to stop the broker and look for the database that (presumably mosquitto) uses to store persisted messages across restarts and delete that database. This is fraught with peril since it’s a pretty blunt instrument. In my mosquitto container, I have an external volume that’s mounted inside the container at /mosquitto/db and there is a single file in there mosquitto.db where I believe these messages are stored.
However, I just tried this using the Home Assistant MQTT publish service. I think you want to do this:
Put the topic you want to clear the retained message for in the topic field of the form. Then make sure:
- The
payloadbox is unchecked - The
retainbox is checked - The
retainswitch to the right is turned on
Then punch that “Call Service” button and that should publish a retained, null payload message to that topic and clear any retained message at the broker.
1 Like
Nope (and if you read back in my previous posts I also tried this with the ‘Payload’ checked but nothing in the entry.
I’m not sure what to suggest; I don’t use MQTT Explorer so I can’t confirm the topic name or know I’m interpreting what I see correctly. I suppose if that’s what you have configured for your sensor, then it’s probably a match.
Are you using the Home Assistant companion application on a Mac? I’ve noticed that if you copy and paste stuff into it, it includes leading whitespace. Is is possible you pasted the topic name into that field and it has a leading space character? This is a long shot.
Just restarting Mosquitto won’t clear anything; it has a database on disk that it stores persistent messages to so that they survive restarts of the broker. I don’t run HassIO, er, “Home Assistant OS” so I don’t know if you can find the persistent volume with the database and delete it, without having to just remove that mosquitto container entirely and reinstall it.
If you have the ability to run a python script, then I have a tool that you can use to purge the retained messages on topics. You’ll need to install the paho-mqtt python library to do this. (Doing something like pip install paho-mqtt.) Here’s a script: https://www.transsys.com/stuff/watch.py and you might edit the beginning to change the default MQTT broker address and stuff.
Of course, you can run this from anywhere you have a python environment that can connect to your MQTT broker.
So you should be able to do something like
python watch.py --broker 192.168.0.42 --retained AuroraA4/#
and see messages published to that (wildcard) topic. (The # is a wildcard match for anything that follows.) Normally, the script won’t print any messages with the retained flag, unless you pass that --retained option. So you should see the messages with with (retained) right after the topic name immediately after the script connects to the broker.
The retained flag indicates that you received it because it was previously published and waiting for you. If another message is published with the retain flag set, current subscribers won’t see that indication delivered to them.
Anyway.
To purge those messages, kill/interrupt the script and run it again with the --purge-retained option added and it will delete the retained messages it receives when the script subscribes to the topics on the broker.
Here’s an example, with the topic foo on my MQTT broker.
louie@kzin[84] $ ./watch.py --retained foo
[2021-04-21 23:21:17] foo(retained): bar
^C
louie@kzin[85] $ ./watch.py --purge-retained --retained foo
[Purging retained message on 'foo']
[2021-04-21 23:24:12] foo(retained): bar
[2021-04-21 23:24:12] foo:
^C
louie@kzin[86] $ ./watch.py --purge-retained --retained foo
^C
louie@kzin[87] $
The first invocation subscribes to the foo topic and shows there’s a retained message present.
Then the next with the purge option and an indication that each message it discovers is purged. You’ll also see the null message that was published to the topic.
Finally, I kill that and run it again to show the retained message is gone.
I normally use this script just to watch MQTT traffic on my broker; you can subscribe to multiple topics (or topic patterns), dump it out in JSON and stuff. I just hacked in this purge thing over the last 15 minutes, so it’s a quick-and-dirty solution. Add the --help option and maybe that will answer more questions than it creates on how to use it.
Thanks everyone for all your suggestions. But I’m tired of digging this rabbit hole.
Apparently MQTT Publish doesn’t actually work, so a Python script should be tried in its place.
Or just delete the whole MQTT database and start over.
No
There are 397 open issues for MQTT on GitHub, this is one of them. Seeing all those, and that it is unable perform the simple task of Honoring the Retain Flag, its clear Eclipse MQTT Broker is a product riddled with bugs and not fit for use.
I’m going to find a different process that doesn’t require MQTT or Home Assistant to accomplish this simple task.
Thanks again for everyone’s suggestions. Sorry MQTT is so far beyond hope.
For the folks still reading this thread, I’ve been using MQTT since 2018 and have never experienced the difficulties the OP has reported.
My introduction to MQTT began when I wanted to add MQTT support to an old home automation system. In order to do that, I had to gain a sufficient understanding of MQTT’s operation to ensure success. The resulting MQTT converter has worked very reliably and predictably for nearly 3 years now.
What the OP has experienced is anomalous. In my tests, using Developer Tools > Services > MQTT: Publish, to purge a retained message, does indeed work; the topic’s payload does not reappear as a retained message.
- Using MQTT Explorer, published a retained message to a new topic.
- Using Developer Tools > Services > MQTT: Publish, published an empty string, as a retained message, to the topic .
- Disconnected/reconnected MQTT Explorer and now the topic and its payload were no longer present (i.e. retained payload removed from the broker’s storage).
- Restarted the MQTT Mosquitto Broker Add-On.
- Disconnected/reconnected MQTT Explorer. Topic and its payload did not return and were truly purged from the broker’s storage.
- Using Developer Tools > Services > MQTT: Publish, published a non-retained message to the previous topic.
- In MQTT Explorer (already running) it displayed the topic and its payload (not as a retained message).
In short, everything worked as expected.
The OP carried out a similar procedure but reported a different outcome, one where the topic always contains a retained message despite attempts to disable it.
There have been reports of performance issues with the MQTT Mosquitto Broker Add-on version 5.1.1 and maybe (just maybe) this is one of its manifestations. I can’t say for sure because I still use version 5.1 and it runs flawlessly.
NOTE
While performing this test, I noticed a UI anomaly (in 2021.4.6) in Developer Tools > Services > MQTT: Publish. If you check the Retain option’s checkbox, its toggle button doesn’t automatically turn on. If you switch from UI Mode to YAML mode, it is clear that the Retain option is still disabled. To truly enable the Retain option, you must not only check the checkbox but also turn on the toggle button. This “double-enabling” requirement seems excessive to me and I would classify it as a “UI glitch”.
5 Likes
I’ve always had good luck with MQTT, been trouble-free for me and I have a pretty high volume of message traffic through my MQTT broker. MQTT is my method of choice when I need to integrate some random hardware I have… I can just cobble up a standalone python script that does what’s needed to poke the device. Then I know there’s an obvious way to push that data into Home Assistant with MQTT.
I posted a pointer to that script on the offhand chance that the topic in question might have contained some invisible character, either when the topic was created, or got pasted into the Home Assistant services UI. GIven the right wildcard match, it could just delete a message to any topic it encountered, even it it contained invisible characters or some wacky unicode lookalike.
I’ve found the MQTT integration method in Home Assistant really reliable and flexible; others ought not to shy away from using it. I wish we could have figured out what the underlying problem was in this particular case…
In MQTT Explorer you can click select the topic, or a higher hierarchy and then click the trash can icon to purge everything selected. You must then disconnect and reconnect MQTT Explorer. The topics will be gone.