MQTT Discovery: to retain or not?

Configuration
1

I have been playing with MQTT Discovery with a home-built device that integrates to Home Assistant using MQTT. It is mostly working, but I can’t work out whether the device’s Discovery configuration messages should be sent using the MQTT retain flag or not.

  • If I do not set the retain flag, HA does not notice the device after a restart. It is still shown as a device under the MQTT Integration, but its entities are listed as ‘Not Available’ and you can no longer see its configuration under the ‘Device Info’ card. This is despite the fact that the device’s availability topic is retained, and still pops up as “online” if I start listening to it on the MQTT Configuration page. To make HA reconnect, I have to restart the device, so that its discovery messages are re-sent. Once this is done, HA starts seeing all the device data, including previously retained messages that haven’t been re-sent.
  • If I do set the retain flag on Discovery messages, then HA doesn’t delete the device properly. As soon as HA re-starts, it sees a retained Discovery message and re-discovers the device, even if it isn’t physically present any more. While the devices entities (a switch, a sensor and a binary_sensor) disappear, its device_automation trigger lingers on. To fully delete the device, I have to either delete it a second time in the UI, or manually connect to the MQTT broker and delete the retained configuration messages.

I think this is a generic problem, not specific to my device. It looks like there is a bug here somewhere in the MQTT Discovery implementation, but I’m not sure what the desired behaviour should be.

It would initially seem quite elegant if Discovery configuration messages were not sent with the retain flag set, but HA remembered the device configuration until the user deleted the device through the UI. The big downside to this approach is that Home Assistant (or anything else you wanted to integrate with) would have to be running at the time the discovery configuration message was sent. It wouldn’t be able to automatically discover the device later.

If Discovery messages are sent as retained messages, then HA could offer an option to delete them from the MQTT broker when you deleted the device from the Integrations UI (by sending a blank, retained message to the device’s configuration topic). It seems to do this automatically already, but only with partial success. This approach also has the side-effect of stopping other systems discovering the device properly (at least until it restarts) - there might be a need for HA to be able to ignore unwanted devices, so you can delete them from one automation system but still use them with another.

I’m happy to help improve the MQTT Discovery documentation (and perhaps the code if I can figure out how), but I’d appreciate some views from people with more experience. How do you think it ought to work?

(Using Home Assistant Core 2021.5.4 on Raspbian)

4 Likes
2

I’m having a bit of an issue with discovery on MQTT too. I’ve also got a home-made bit of hardware connected to a Mosquitto broker (garage door as a “cover”), but the problem I’m having is this:

When I restart Mosquitto (without restarting Home Assistant), Home Assistant sort of re-discovers the device so I briefly end up with two instances shown in the UI, before Home Assistant “forgets” the old instance. The problem with this is that the new instance has a different entity identifier (the old one with “_2” at the end), which breaks any automations that depend on that.

I’d be interested to know if yours has the same behaviour.

(also if anybody has any ideas about why its doing this and how to debug/fix discovery on MQTT, that’d be appreciated)

3

I did initially have a similar problem (hence some of my interest in deleting auto-discovered devices working properly). I guess it could be because Home Assistant receives an extra copy of the device’s discovery message when it reconnects to Mosquitto, if the message has been retained by the broker.

Have you tried including the unique_id variable in your discovery configuration payload? Setting that ought to get HA to recognise your hardware as being the same device each time it finds it.

4

The retain flag should be set for discovery messages.

As for repeating devices, you need to set a unique identifier. The device’s MAC address can help with this

To prevent multiple identical entries if a device reconnects a unique identifier is necessary.

5

I had been coming to the conclusion that the retain flag should be set for discovery messages, but I can’t see it mentioned anywhere in the user instructions or the developer docs.

I think there’s some scope for improvement here. As well as the missing detail about retained messages, the examples aren’t great from the viewpoint of getting things working reliably. Only one of the examples actually uses a unique_id, for instance, and that example doesn’t follow the Unique ID requirements from the developer docs.

I’ll do a bit of research through the forums to see if there are other details with Discovery setup that people find problematic, then see if I can propose some changes to the MQTT Discovery page. I’ll also research the problem with deleting devices a bit further. It looks like there might be a bug, if Home Assistant deletes some of the retained discovery messages on device deletion, but not all of them.

6

I cannot see now where I read that discovery messsages should have retain set, but if I am right, it should be in the docs.

I have found the minor doc changes I have made have been well received and quickly approved.

In other words, fill your boots, it is encouraged (and good) to contribute.

7

Interesting reading in your first post. I am not sure there is an easy solution as both options have issues.

I personally use a retained config topic, I don’t really like the idea of HA autocratically removing that payload for the reason you mention (other consumers of that payload)

8

Can you elaborate on what you mean by “partial success”? Are you saying sometimes it purges the retained message and sometimes it doesn’t?

MQTT Discovery is Home Assistant’s means of discovering devices. Anything else using the same discovery topics is out of scope. Just speculating, but it may partially explain the Tasmota project’s decision to create its own MQTT-based discovery protocol (which also employs retained messages). Their own discovery protocol is now the default and Home Assistant’s has been deprecated (but can still be enabled via SetOption19 1).

1 Like
9

When I tried to delete my device, it was consistently deleting some of the retained config messages, but not all of them. The one that was left behind was for a device_automation trigger, which would cause the device to reappear after restarting Home Assistant.

After experimenting with this, I found that a second device with identical hardware and firmware (but different unique IDs) did not have the same problem. Further, the problem disappeared on the original device after I upgraded HA Core to the latest version. My guess is that I had somehow confused HA while prototyping my design, and it had managed to store some kind of misconfigured state that survived device deletions, but was reset by the update.

This made me think further about unique IDs for devices. It isn’t entirely clear from the documentation when a ‘unique’ ID is actually supposed to be unique (so that the system can refer unambiguously to a specific element), and when it’s not (so that the system can recognise that different elements are part of the same unique device). While trying to debug the problem above, I sent various MQTT config messages for testing, and noticed that new device_automation triggers would get recognised by HA as part of the same MQTT device if they shared the same MAC address in the connections map, within the config payload, even though all the unique IDs were different. The details in the connection map aren’t documented as needing to be unique (several devices could be connected via one connection). I also noticed that the configuration for a device_automation does not permit a unique_id to be set like a normal sensor. I wonder if this could have contributed to my problems.

I think I’ve now got the answer to my original question: MQTT configuration messages ought to be sent with the retain flag set.

However, I think I’ll also be opening a new thread about all the different types of unique ID…

2 Likes
10

Did you start an edit of the Docs. Ran into the same problem and the same solution for retain, but if it were in the Docs It would have been helpful. Otherwise I’ll look into that. Don’t want to step on toes.
Update. I pulled a PR for the Dos…
–UPDATE-- The update has been merged…

I found that creating a Device with a Unique ID then using that device for all the sensors in that device works well keeping them from duplicating. Picking the unique ID I use the Serial number side of the MAC address and plug that in. For Python that looks like this:

# Pulling the unique MAC SN section address using uuid and getnode() function 
DEVICE_ID = (hex(uuid.getnode())[-6:]).upper()

My Python script pushing Home Assistant MQTT Discovery and sensor values:

Update MUCH later–
Use caution with the mac address unique thing. If your device, for example a raspi, sometimes connects on WIFI and sometimes connects on an ethernet wire, the mac will change and the device becomes a new device.
So either lock in the mac for all interfaces or only use 1 interface for network.

(Take a wild guess on how I know this…)

Now, back to the show…

2 Likes