Auto Discovery

mark1evans · April 26, 2020, 8:30am

Hello there I hope someone can help me.

My setup all Tasmota devices connecting via MQTT, Mosquito running as part of the install from Ubuntu Server 18, not using HA plugin.

Using Auto Discovery, the discovered entities work fine, but they become unavailable after a reboot, sending the autodiscovery option from Tasmota restores them, as does a reboot of the Mqtt Clients.
I understand this is a long standing problem, I have seen post going back to 2017, is there a correct way to rectify this problem. I dont want to go back to manual entry in the YAML file as it becomes bloated enough as it is, and the suggestion is that HA is moving more towards GUI configuration in the first place.
I have tried more of my fair share of using MQTT persistence and none have worked as yet.

Thanks in advance Mark

123 · April 26, 2020, 4:29pm

That behavior implies the MQTT Discovery topics, created by your devices running Tasmota, do not have retained messages.

If I publish this payload:

{
  "name": "Front Door",
  "device_class": "door",
  "state_topic": "home/front_door",
  "unique_id": "abc123",
  "payload_on": "1",
  "payload_off": "0"
}

to this topic:

homeassistant/binary_sensor/front_door/config

then Home Assistant’s MQTT Discovery service will automatically create a new entity called:

binary_sensor.front_door

If I now restart Home Assistant (or restart the MQTT Broker), it reconnects to the MQTT Broker but the topic and its payload are now gone so its flags binary_sensor.front_door as unavailable.

Why is the topic and its payload gone? Because I overlooked to tell the MQTT Broker to store (“retain”) the topic and payload I had published. If I publish the payload as a retained message, the MQTT Broker will store it and make it available to Home Assistant on startup.

I don’t use Tasmota so I can’t offer you much advice beyond the fact you have to enable SetOption 19 which provides Home Assistant automatic discovery. In theory, Tasmota should be publishing discovery payloads as retained messages.

You can confirm this for yourself by using an MQTT client like MQTT Explorer. Once MQTT Explorer is connected to your MQTT Broker, you should see topics beginning with homeassistant. If you don’t, then restart one of your Tasmota devices. If you now see a topic starting with homeassistant that means it was created without a retained message.

mark1evans · April 27, 2020, 7:08am

Thats Exactly whats happening, what I am currently doing is sending a command as a global MQTT message to all my Tasmota devices the Command SetOption 19 1.
This is places in the Automations.yaml.

It works, but I feel is a bodge, Its currently working fine with about 6 devices, but I think would come apart if I had 50 or so.
I have only just downloaded MQTT explorer, so need to play around with it to find which messages are which.
Thanks for the quick reply I will post back my findings, Im sure Im not the only one with this setup so will help others… Mark

francisp · April 27, 2020, 7:20am

What mqtt broker are you using ? The HA built-in broker (deprecated) does not store retained MQTT messages.

I have over 50 tasmota devices ( and a lot other MQTT-based devices), none show this behavior.

123 · April 27, 2020, 2:02pm

Good point! That would be the culprit because if SetOption 19 1 is being used, the MQTT Broker should be storing the config payloads (and Home Assistant wouldn’t “lose” them after a restart).

mark1evans · April 27, 2020, 10:06pm

O.K done some tests

When running as normal seeing topic of Home Assistant on MQTT Explorer, there are messages shown as config, running at QOS 0.
The message is showing as retained
Did a full reboot, without the Setoption19 1 in the Automation file.
Entities not available as usuall
On system running again, MQTT Explorer shows no topic for Home Assistant anymore.

Could it be that under QOS 0 the messages are lost and it cannot resend, since Mosquitto is likely to be up and running way before Home Assistant.

mark1evans · April 27, 2020, 10:47pm

I have done some reading of QOS levels and still cant get my head around which party determines the QOS level.
Since Mosquitto shows the message as retained this should be stored on the HDD and would survive the reboot, this would be the server.
In this case I assume Home Assistant is the client to have the message sent to.
To my thinking Tasmota has done its part getting the message to Mosquitto and have it set as retained, its now between Mosquitto and Home Assistant to complete the proccess.
???Totally confused???

123 · April 28, 2020, 1:07am

If the topics disappear, it means the payloads published to those topics were NOT published as retained messages.

Imagine you are attending a meeting where no one is taking notes. Everyone in the meeting knows exactly what has been discussed. If someone arrives late, they have no idea what has happened in the meeting. However, if someone had been talking notes, the late-comer could review them and be apprised of everything that had been discussed. That’s similar to how retained messages work. The payloads are stored by the MQTT Broker so if a client, like Home Assistant, disconnects from the MQTT Broker and reconnects later, it will have access to all the stored payloads (“retained messages”).

Could it be that under QOS 0 the messages are lost …

Quality Of Service level has absolutely nothing to do with this issue.

123 · April 28, 2020, 1:35am

Here’s an easy way to learn how MQTT Discovery works. Add these two scripts to your system:

#script:
  example_create:
    alias: 'Create Example MQTT Light'
    sequence:
      service: mqtt.publish
      data:
        topic: 'homeassistant/light/example/config'
        retain: true
        payload: >
          {
            "name": "Example",
            "unique_id": "ABC123",
            "stat_t": "home/example",
            "cmd_t": "home/example/set",
            "bri_stat_t": "home/example/brightness",
            "bri_cmd_t": "home/example/brightness/set",
          }

  example_delete:
    alias: 'Delete Example MQTT Light'
    sequence:
      service: mqtt.publish
      data:
        topic: 'homeassistant/light/example/config'
        retain: true
        payload: ''

Go to Developer Tools > Services and execute script.example_create. It will make MQTT Discovery create a new entity called light.example. Now use MQTT Explorer to connect to the MQTT Broker and confirm that this discovery topic exists:

homeassistant/light/example/config

It contains a payload that was published as a retained message. If you restart Home Assistant, the light.example entity will still exist. If you restart the MQTT Broker, the topic and its payload will persist.

If you observe any behavior that differs from what I have described then there is something wrong with the configuration or operation of your MQTT Broker.

If you execute script.example_delete it will make MQTT Discovery remove the light.example entity. MQTT Explorer will no longer display the discovery topic for that entity.

francisp · April 28, 2020, 4:03am

Did you read the manual ?

persistence [ true | false ]

If true , connection, subscription and message data will be written to the disk in mosquitto.db at the location dictated by persistence_location. When mosquitto is restarted, it will reload the information stored in mosquitto.db. The data will be written to disk when mosquitto closes and also at periodic intervals as defined by autosave_interval. Writing of the persistence database may also be forced by sending mosquitto the SIGUSR1 signal. If false , the data will be stored in memory only. Defaults to false .

mark1evans · April 28, 2020, 8:07am

This is what Explorer is showing me.

Suggests messages are retained

francisp · April 28, 2020, 8:15am

Yes, but if mosquitto is not configured right, they are lost when mosquitto restarts (or your pi reboots)

mark1evans · April 28, 2020, 9:41am

God why do things so complicated, so quickly, so its looking at config for mosquitto now.

Did another check, booted so as not to be discovered, stopped home assistant, rebooted Tasmota devices.
All topics appeared in MQTT explorer after the reboot.

My conclusion is either Mosquitto is not storing the config as its supposed to
Or Tasmota is not sending to messages to be retained.

Question how to I find out which one is not playing ball, I must admit Im a bit nervous at playing around in Mosquitto as I could break everything, and its all running in Docker what ever that means. Thanks for all your help by the way, my backround is electronics not programming… Thanks Mark

francisp · April 28, 2020, 10:02am

I don’t know much about running mosquitto in docker (my mosquitto’s run on bare metal), but normally your mosquitto.conf must be somewhere outside the container.

The solution is simply to add

persistence true

to your mosquitto.conf file and restarting mosquitto.

francisp · April 28, 2020, 10:51am

Everything you need to know is nicely explained here :

https://hub.docker.com/_/eclipse-mosquitto

including how to setup persistence

123 · April 28, 2020, 1:24pm

I currently have two instances of Mosquitto.

The first instance was created by an automatic installation of openHAB on a Raspberry Pi 3B (openHABian, like HASSbian). This instance has been running for about 1.5 years. I checked /etc/mosquitto/mosquitto.conf and it has persistence true. The installer software (created by an openHAB volunteer) pre-configured it like that.
The second instance was recently created by the automatic installation of Home Assistant Supervised on an old laptop. I checked /usr/share/hassio/addons/core/mosquitto/data/mosquitto.conf and it also has been pre-configured with persistence true.

So now tell us how you installed mosquitto on your machine. You said it was based on docker. What instructions did you use to install it? This information will help us point you to the location of the mosquitto.conf file.

mark1evans · April 28, 2020, 2:11pm

Both of you are JEMS
im sure I can find my way from here now, I think I will get on just fine with this HASS as the community are so helpfull.
The Node Red forums are a little less helpfull, probably because it is picked up on by fewer armatures, the usual reply, is READ THE DOCUMENTATION, which most people do, but at as programming novice, the documentation means very little. Its people like yourselves that allow us novices to gain knowledge.
Thanks Mark.

One useful tip for running 5V power to sensors and tablets, that has worked 100% for the last five years is,using the thin copper tape on the emulsioned walls and ceilings.
Seal with a non water based paint, prior to repainting to stop electrolysis action. You will have to sweat solder any joints.
The result is completely invisible and safe. Its not like you score your walls with a knife all the time.
I thought I,d share something back

mark1evans · April 28, 2020, 4:45pm

Just fully read your message, my Mosquitto was installed at the same time as Ubuntu as stated before,so I guess that will not be in a Docker container making the file easy to find. Only the HASS installation is in a Docker container, it was script installed

123 · April 28, 2020, 5:38pm

What installation instructions did you use because mosquitto is not normally installed by default with Ubuntu.

francisp · April 28, 2020, 6:34pm

If not in docker, your configuration file will probably be in /etc/mosquitto