MQTT auto discovery isn't working, but messages are visible

I installed the MQTT integration “fresh” without any configuration in configuration.yaml, set it up to connect to my MQTT broker and made sure discovery was enabled, but I’m still not seeing my MQTT device auto-discovered. Note that I am not using a username/password to connect to my broker right now.

I used the “listen to a topic” function in HA to verify the messages are being seen, and they definitely are, but this is from a custom Python app I wrote, so I wanted to see if I had format issues. Here’s a screenshot of what I see in HA, never mind the order of the messages: normally the config message would come first when the Python app starts, I just happened to capture the operations in reverse order here.

Am I on the right track here?

At first glance, it looks OK. I wouldn’t normally set the entity’s state_topic to something so similar to its discovery topic but I think it should be OK.

Just for fun, try this script.

  create_entities:
    sequence:
      - service: mqtt.publish
        data:
          topic: homeassistant/binary_sensor/bathroom_door/config
          payload: >
            {
              "name":"Bathroom Door",
              "state_topic":"home/sensor1",
              "device_class":"door"
            }

Alternately, just go to Developer Tools > Services, select mqtt.publish, then set the options:

topic: homeassistant/binary_sensor/bathroom_door/config
name: Bathroom Door
state_topic: home/sensor1
device_class: door

and click CALL SERVICE.

Normally you would include retain: true so that the discovery topic and its payload are stored on the broker. This ensures that when you restart Home Assistant, the entity will be re-created (otherwise it won’t). For testing purposes, it’s not necessary (you probably don’t want this entity to persist beyond the next restart).

Should the state and discovery topics be unique? I’m new to MQTT and got the impression that the discovery topic and state topic had to be identical, and I had to use the full topic in the discovery declaration; whereas you have home/sensor1, I thought it had to be something like homeassistant/binary_sensor/bathroom. So how do these topics actually work?

As you may have already seen in the documentation for MQTT Discovery, it requires the discovery topic to adhere to a defined structure (which you have done).

In contrast, the entity’s state_topic follows whatever structure you prefer. As you’ve seen in my example, it uses `home/sensor1’ but it can be whatever you like as long as it isn’t identical to any existing discovery topic.

“Not Identical to the discovery topic” meaning it has the “state” instead of “config” with the associated dictionary, as I’ve done, or does it need to be more unique?

Beats me because I have never set a state_topic to be nearly identical to a discovery topic.

I would imagine that when Home Assistant looks for discovery topics, it subscribes to homeassistant/# and that’ll definitely include what you’ve used as a state_topic.

Just change the state_topic and see if it helps.

Let me ask a different question: have YOU been able to get MQTT discovery to work? I get the impression you have given your earlier example, but if we’re just spit-balling here I might just save my effort and manual add the entity. My understanding is most 3rd-party MQTT devices will require manual integration anyway as they won’t publish in the defined format.

I have 89 mqtt devices auto discovered, compromising 371 entities. Quite a lot of the autodiscovery messages I have written myself, so I think mqtt autodiscovery in HA works quite well.

Yes.

The ‘6 devices’ and ‘56 entities’ reported by the MQTT Integration were all created via MQTT Discovery.

Screenshot from 2020-08-21 12-12-40

Previously, I had defined them in configuration.yaml. However, you cannot define devices this way, only through MQTT Discovery. So I scrapped all the MQTT-based entities I had in the config file and created scripts to generate them via MQTT Discovery.

Here’s a fragment of one script that creates binary_sensors (just the first binary_sensor is shown):

  create_binary_sensors:
    alias: "Create binary_sensors via MQTT Discovery"
    sequence:
      - service: mqtt.publish
        data:
          topic: homeassistant/binary_sensor/front_door/config
          retain: true
          payload: >
            {
              "name": "Front Door",
              "device_class": "door",
              "state_topic": "premise/frontdoor/sensor/dooropened",
              "unique_id": "elk_m1_front_door_contact_sensor",
              "payload_on": "1",
              "payload_off": "0",
              "device": {
                  "identifiers": ["elk_m1_security_panel"],
                  "name": "ELK M1",
                  "model": "M1 Gold",
                  "manufacturer": "ELK",
                  "sw_version": "4.XX"
              }
            }

      - service: mqtt.publish
        data:
          topic: homeassistant/binary_sensor/rear_door/config
          retain: true
          #... etc ...
1 Like

So your devices weren’t fully auto-discovered without using the script?

I’m just trying to figure out what I’m going to have to do when I start bringing in Z-Wave devices via MQTT 2 Z-Wave bridge. Currently I have a a Raspberry Pi zero W set up as a Mozquitto broker, and I’m waiting for my USB Z-Wave adapter to arrive, along with the Z-Wave switches I ordered.

In order for an MQTT-based device to be “auto-discovered”, it has to publish its configuration to an MQTT Discovery topic.

If the device is incapable of doing that, you have one of two choices:

  1. Define it in configuration.yaml
  2. Write a script that creates it in Home Assistant via MQTT Discovery.

The devices I’m using don’t support Home Assistant’s MQTT Discovery. To create them in Home Assistant, I chose option 2.

2 Likes

Thanks for the help. Given the manual configuration I’ll have to do for my Z-Wave devices, I think I’ll just stick with manual configuration rather than trying to fight with auto-discovery.

FWIW, given the topic’s title, by switching to manual configuration it’s exactly a “solution” and more of a “capitulation”. Sorry to hear you gave up. As you’ve seen, both Burningstone and I use it successfully.

If you’re interested in giving it a second shot, we’re here to help.

I definitely appreciate your help; it is indeed more of a capitulation. In my particular case, I’m not sure there’s much value added if the auto discovery works, as I do plan on having mostly Z-wave devices, which I will have to add manually anyway if I understand correctly. This particular device was a python project I wrote some time ago which I just added some super simple MQTT into, but even then the python library I’m using (paho-MQTT) has some odd idiosyncrasies I don’t quite understand in terms of how it creates the MQTT client, and it just wasn’t worthwhile for me to chase all that down right now.

I may play with some of your other suggestions later to see if the discovery picks up on any of it, but at least for the scope of the items I’m looking to integrate in the near future, auto discovery just isn’t going to be feasible anyway.

Hello,

Trying to set up MQTT Discovery for a couple of ESP8266’s that I’m writing through Arduino IDE.

Topic I’m publishing to for discovery:

homeassistant/sensor/8C-AA-B5-59-2C-F1/config

Payload:

 {
  "uniq_id": "8C:AA:B5:59:2C:F1",
  "~": "homeassistant/sensor/8C:AA:B5:59:2C:F1",
  "stat_t": "~/state",
  "avty_t": "~/status",
  "name": "Moisture 2C:F1",
  "sw": 4.1,
  "mdl": "ESP8266"
}

The discovery topic (configured IAW the prescribed format) is receiving the message, Discovery is on (confirmed in logs), but devices / entities aren’t being created. Am I missing something? Do I need a script like mentioned above to actually create the entity?

If so, what benefit does using Discovery have if you still need to manually create a script to use it?

Personally, I never set the device’s state topic to have the same path as Home Assistant’s discovery topic. Therefore I can’t comment on whether this choice influences the failure of discovering the device.

Also, the one thing you didn’t mention is if you have configured the MQTT integration. Maybe you have but I can’t be certain. Are you confirming the payload was published by using a separate MQTT client or by observing it with Home Assistant?

Thanks, I did figure it out using a similar post.

For future reference for anyone else, I found two different issues I was having that I resolved.

  1. The JSON payload ended up larger than the default maximum size. To fix this, either adjust the MQTT_MAX_PACKET_SIZE in <PubSubClient.h> to a suitable number, or, add the following line to your program:
    client.setBufferSize(1024);

  2. The Device Map needs to be a JSON Object within the JSON document. My final version looks something like this:

StaticJsonDocument<1024> pl_config;
void mqttpub_dscvy(char* node) {
  if (!client.connected()) {
    mqtt_reconnect();
  } else {
    ;
  }

  String mqtt_dscvy_t = mqtt_prefix_d  + "/" + node + "/config";
  mqtt_dscvy_t.replace(":", "-");

  Serial.println("");
  Serial.print("Initiating MQTT discovery (-t ");
  Serial.print(mqtt_dscvy_t + ")...");

  pl_config.clear();

  pl_config["uniq_id"] = MAC + "_" + String(node);
  pl_config["~"] = mqtt_prefix;
  pl_config["stat_t"] = "~/state";
  pl_config["avty_t"] = "~/status";
  pl_config["name"] = MAC.substring(12) + " " + DEVICE_CLASS + " (" + String(node) + ")";
  pl_config["ic"] = "mdi:water-percent";
  pl_config["val_tpl"] = "{{ value_json." + String(node) + "}}";

  mqtt_setDeviceConfig(mqtt_dscvy_t, node);
}
void mqtt_setDeviceConfig(String config_t, String node) {
  JsonObject device = pl_config.createNestedObject("dev");

  device["name"] = "Garden Sensor (" + MAC.substring(12) + ")";
  device["sw"] = SOFTWARE_ID;
  device["mdl"] = MODEL_ID;
  device["mf"] = MANUFACTURER_ID;
  JsonArray identifiers = pl_config["dev"].createNestedArray("ids");
  identifiers.add(MAC);
  JsonArray connections = device.createNestedArray("cns");
  connections.add(serialized("[\"MAC\",\"" + MAC + "\"]"));
  //connections.add(serialized("[\"ip\",\"" + ipstr + "\"]"));

  char buffer[1024];
  serializeJsonPretty(pl_config, buffer);

  if (client.publish((char *)config_t.c_str(), buffer, true) == true) {
    Serial.println("...success");
  } else {
    Serial.println("...failed...");
  }
}

@123, I did test it with the same MQTT base path, and with different ones as you suggested, and can confirm both work without issue. I did end up using different paths in the end so that I could subscribe to all config topics without subscribing to the state topics.

Please note, I am in no way an expert on this, so if anyone has any comments / suggestions, let me know!

3 Likes

Would highly recommend using GitHub - dawidchyrzynski/arduino-home-assistant: ArduinoHA allows to integrate an Arduino/ESP based device with Home Assistant using MQTT. as it presents a nice api to do what you need

Hi!
Hope it’s ok to reactivate this old thread, seems like you have a lot of knowledge on this topic and I’m out of options after trying all things found here in the forum.

I’m setting up an Arduino with ArduinoHA and MQTT Mosquitto in Home Assistant. Using a boilerplate for ArduinoHA first try was a success, my device and entity showed up in HA. But after I renamed the device in Arduino nothing happened in Home Assistant. So I deleted the device and entities in HA hoping it would re-add again… After this, it won’t show up again.
My setup:
Arduino side using ArduinoHA lib. Discovery message is built into the HAMqtt:begin or HAMqtt:loop command. The stripped version of my code to show MQTT setup:

#include <WiFiNINA.h>
#include <ArduinoHA.h>

#define LED_PIN         2
#define BROKER_ADDR     "192.168.0.161"//IPAddress(192,168,0,161)
#define MQTT_USER       "ccc"
#define MQTT_PASS       "xxx"
#define WIFI_SSID       "xxx"
#define WIFI_PASSWORD   "zzz"

WiFiClient client;
HADevice device(mac, sizeof(mac));
HAMqtt mqtt(client, device);
HASwitch openPatio("home/patio/open");

/*********************************
** FUNCTION MQTT callback on message
**********************************/
void onMessage(const char* topic, const uint8_t* payload, uint16_t length) {
    Serial.print("MQTT message recieved on topic: ");
}

/*********************************
** FUNCTION MQTT callback on connect
**********************************/
void onConnected() {
  Serial.println("MQTT connection established.");
  openPatio.setAvailability(true);
  mqtt.subscribe("home/patio/open");
}

/*********************************
** FUNCTION MQTT switch-callback on command
**********************************/
void onSwitchCommand(bool state, HASwitch* sender)
{
    Serial.println("MQTT command recieved.");
}


/*********************************
** SYSTEM SETUP
**********************************/
void setup() {
  Serial.begin(9600);

  // Declare pins:
  pinMode(LED_PIN, OUTPUT);

  // Connect to wifi hidden for this purpose

   // set device's details (optional)
  device.setName("PatioCTRL");
  device.setSoftwareVersion("1.0");  

  // handle switch state
  openPatio.onCommand(onSwitchCommand);
  openPatio.setName("open_patio"); 
  openPatio.setRetain(true); 

  //Set up callbacks
  mqtt.onMessage(onMessage);
  mqtt.onConnected(onConnected);

  //Set login credentials
  mqtt.begin(BROKER_ADDR, 1883, MQTT_USER, MQTT_PASS);

}

/*********************************
** SYSTEM MAIN LOOP
**********************************/
void loop() {
  mqtt.loop();
  delay(100);
  openPatio.setState(true); //just testing...
  delay(1000);
  openPatio.setState(false); //just testing...}


The topic is shown in MQTT explorer as expected and toggle its value in aha>ecxxx>home etc. See picture:

But in Home Assistant it is empty in the Mosquitto Broker add-on.

So far, I’ve tested:

  • Restarting HA and arduino, of course…
  • Deleting and adding topics
  • Made sure Arduino sends messages and config message
  • Made sure discovery prefix is right
  • Made sure auto discovery is enabled in Mosquitto
  • Nothing in any logs are showing any leads
  • The device/entity is correctly removed from core.entity_registry
  • I’ve also tried from HA to activate the device by adding a HA-script to publish a config message:
mqtt_test:
  alias: "Test MQTT Discovery"
  sequence:
    - service: mqtt.publish
      data:
        topic: homeassistant/switch/ec626081039c/home/patio/open/config
        retain: true
        payload: >
          {
            "name": "open_patio",
            "device_class": "switch",
            "uniq_id": "home/patio/open",
            "ic": "mdi:lightbulb",
            "ret": true,
            "payload_on": "1",
            "payload_off": "0",
            "dev": {
              "ids": "ec626081039c",
              "name": "PatioCTRL",
              "sw": "1.0"
            }
            "avty_t": "aha/ec626081039c/home/patio/open/avty_t",
            "stat_t": "aha/ec626081039c/home/patio/open/stat_t",
            "cmd_t": "aha/ec626081039c/home/patio/open/cmd_t"
          }

But it didn’t help either. The message is received in MQTT Explorer and updates the config topic with some small changes compared to the one sent from Arduino.

Does anyone know what I can do to rediscover the device and entity?

Mikail,

Have you solved this? If so, how? I’ve been developing a “gateway” between LoRa sensors and HA’s MQTT using a highly-modified copy of that same ArduinoHA library. Things were working great, until suddenly only a few entities are auto-discovered, even though the others had worked briefly. This happened as I was working with the names and unique_ids passed to the constructors in ArduinoHA.

Solved! Auto-discovery doesn’t like empty strings for icon or device-class and possibly other data items. Modifying my code to not set it if it’s empty.

Paul