Can someone please explain MQTT in HA

I am so confused.

There is this guide:

saying you can run your own broker and add the IP to the config.

There is an integration allowing you to add MQTT by specifying the Broker name (assume this is IP)

There is then mqtt discovery in the config, or auto discovery in the integration.

There is then the mqtt sensor.
There is also the “listen” option under MQTT in dev tools

Currently I have Mosquitto running on a synology nas.
I have the MQTT integration with the following settings:

broker: 192.168.1.124
port: i left as default 1883
username and password I left blank.

So, how do I now get information that is going to MQTT broker into HA now?

Sorry, I have tried but theres just too many mentions of MQTT and different methods. The end result is I would like to listen for MQTT data and make sensors from the topics.

Thanks!

You create entities that use MQTT. Sensors, switches, etc.

MQTT autodiscovery is explained here:

The messages have to come through formatted in a certain way in order for discovery to work. It has to have a certain configuration, not on HA, on the device sending the message.

So create the sensors and have them subscribe to the topics you want.

This link well help you also you’re specifically trying to create MQTT based sensors.

Thanks, so is MQTT Discovery different to this option?

image

If I add mqtt discovery I would add this:

mqtt:
  discovery: true
  discovery_prefix: homeassistant

for discovery_prefix the docs say:

"discovery_prefix
(string)(Optional)

The prefix for the discovery topic."

So my Esp32 sketch says:

// MQTT topic gets defined by "<MQTT_BASE_TOPIC>/<MAC_ADDRESS>/<property>"
// where MAC_ADDRESS is one of the values from FLORA_DEVICES array
// property is either temperature, moisture, conductivity, light or battery

const char*   MQTT_HOST       = "10.10.10.1";
const int     MQTT_PORT       = 1883;
const char*   MQTT_CLIENTID   = "miflora-client";
const char*   MQTT_USERNAME   = "username";
const char*   MQTT_PASSWORD   = "password";
const String  MQTT_BASE_TOPIC = "flora"; 
const int     MQTT_RETRY_WAIT = 5000;

So should I just use flora as the prefix in place of homeassistant?
If this works how can I see the results, will it generate the sensors or what exactly will it “discover” sorry.

also, if i wanted to use MQTT for flora and something else can I do something like this:

mqtt:
  discovery: true
  discovery_prefix: flora
  discovery_prefix: blinds

or would I need to amend my code where it says flora to say something generic and then do the same on the other mqtt topic?

Yes.

No. It won’t find the device that way. Read the link I sent. Your device has to broadcast IT’S CONFIGURATION over MQTT, which is where that prefix comes in. The device sends it’s config over the homeassistant prefix, using the details line out in the documentation. Think of it like a message before all the messages that follow. I set my esp devices up with the configuration broadcast at startup of the device.

Ok, its starting to click a bit then, but not entirely.

So the sketch I have put onto my esp32 from here:

needs to send information with a discovery prefix. It also needs to send the config topic in a set format?

the docs say:

<discovery_prefix>/<component>/[<node_id>/]<object_id>/config

my sketch currently sends topics in the format:

"<MQTT_BASE_TOPIC>/<MAC_ADDRESS>/<property>"

Is this on the right lines then?

The “discovery” message is different than the state messages. Your device needs to send 2 separate topics.

One is the configuration topic, with json that defines the device. Send this on connection to the broker.

The second topic is the normal topic. The sensor sending it’s data.

Please read the link I sent you. Read it several times.

To be honest, if you’re programming your own devices (your esp32s), it’s much easier to use the default flora sketch you have and enter the mqtt sensor config manually in your HA config.

1 Like

Think you might be right. I am reading and reading the docs and just not getting it, and even if it does make sense I do not know how to change the code really.

Thanks.

Especially if you’re still learning the format, much easier to adjust HA config and retest rather than recompile and load on ESP.

(That said, if you’re using ESP devices, you should check out EspHome - much easier than writing your own code in Arduino IDE. )

Thanks, I thought about esphome but wanted to try the sensor via mqtt route. I was also not entirely sure with esphome what exact integration I would need to work with the plant sensor but may have a play (oh and my HA Pi is in the attic so plugging the esp32 into it was slightly more of a pain than plugging it into my laptop) but based on all the MQTT issues I am having (see other posts) I may need to go ESPHome

You can download the .hex file from esphome web interface and use the separate Loader app on your laptop to upload to ESP. This is the way I do all of mine.

If your external sensors are supported by esphome, I highly suggest it.

1 Like

To give you an idea of the type of stuff discovery is looking for, this is an entire excerpt from one of my ESP sensors (I use platformio). Have a look at the publishTempConfig() and publishPressConfig() sections

#include <Arduino.h>

#include <ESP8266WiFi.h>
#include <Wire.h>
#include <Adafruit_BMP085_U.h>
#include <PubSubClient.h>
#include <Adafruit_Sensor.h>
#include <ESP8266mDNS.h>
#include <ArduinoOTA.h>
#include <ESP8266HTTPUpdateServer.h>
#include <ESP8266WebServer.h>
#include <ArduinoJson.h>

/************************* WiFi Access Point *********************************/

#define wifi_ssid "MYSSID"
#define wifi_password "MYPASSWORD"
#define hostname "MYSENSORNAME"

/************************* MQTT Broker Setup *********************************/

#define mqtt_server "MYSERVERIP"
#define mqtt_user ""
#define mqtt_password ""
#define mqtt_clientId "office-sensor"

/****************************** Feeds ***************************************/

#define TOPIC_TEMP_CONF "homeassistant/sensor/office_temp/config"
#define TOPIC_PRESS_CONF "homeassistant/sensor/office_press/config"
#define TOPIC_STATE "homeassistant/sensor/office/state"

#define TEMP_NAME "Office Temp"
#define TEMP_CLASS "temperature"
#define PRESS_NAME "Office Pressure"

Adafruit_BMP085_Unified bmp = Adafruit_BMP085_Unified(10085);

#define SDA D5
#define SCL D6

//fahrenheit = 9.0/5.0*celsius+32

void setup_wifi();
void reconnect();
float convertToF(float tempVal);
bool checkBound(float newValue, float prevValue, float maxDiff);

WiFiClient espClient;
PubSubClient client(espClient);
ESP8266WebServer httpServer(80);
ESP8266HTTPUpdateServer httpUpdater;

void publishTempConfig() {
  const size_t bufferSize = JSON_OBJECT_SIZE(8);
  DynamicJsonBuffer jsonBuffer(bufferSize);
  JsonObject& root = jsonBuffer.createObject();
  root["name"] = TEMP_NAME;
  root["dev_cla"] = TEMP_CLASS;
  root["stat_t"] = TOPIC_STATE;
  root["unit_of_meas"] = "°F";
  root["val_tpl"] = "{{value_json.temperature}}";
  root["pl_avail"] = "online";
  root["pl_not_avail"] = "offline";
  // root["uniq_id"] = mqtt_clientId"_temp";
  root.prettyPrintTo(Serial);
  Serial.println("");
  char message[256];
  root.printTo(message, sizeof(message));
  client.publish(TOPIC_TEMP_CONF, message, true);
}

void publishPressConfig() {
  const size_t bufferSize = JSON_OBJECT_SIZE(8);
  DynamicJsonBuffer jsonBuffer(bufferSize);
  JsonObject& root = jsonBuffer.createObject();
  root["name"] = PRESS_NAME;
  root["stat_t"] = TOPIC_STATE;
  root["unit_of_meas"] = "mbar";
  root["val_tpl"] = "{{value_json.pressure}}";
  root["pl_avail"] = "online";
  root["pl_not_avail"] = "offline";
  // root["uniq_id"] = mqtt_clientId"_press";
  root.prettyPrintTo(Serial);
  Serial.println("");
  char message[256];
  root.printTo(message, sizeof(message));
  client.publish(TOPIC_PRESS_CONF, message, true);
}

void setup() {
  Serial.begin(9600);
  delay(10);
  setup_wifi();
  client.setServer(mqtt_server, 1883);
  Wire.begin(SDA, SCL); //sda,scl
  Serial.println("Sensor Test");
  if (!bmp.begin())
  {
    Serial.print("Ooops, no BMP180 detected ... Check your wiring or I2C ADDR!");
    while (1);
  }
  else {
    Serial.println("BMP180 ready.");
    delay(1000);
  }

  /* OTA code */
  ArduinoOTA.onStart([]() {
    Serial.println("Start");
  });
  ArduinoOTA.onEnd([]() {
    Serial.println("\nEnd");
  });
  ArduinoOTA.onProgress([](unsigned int progress, unsigned int total) {
    Serial.printf("Progress: %u%%\r", (progress / (total / 100)));
  });
  ArduinoOTA.onError([](ota_error_t error) {
    Serial.printf("Error[%u]: ", error);
    if (error == OTA_AUTH_ERROR) Serial.println("Auth Failed");
    else if (error == OTA_BEGIN_ERROR) Serial.println("Begin Failed");
    else if (error == OTA_CONNECT_ERROR) Serial.println("Connect Failed");
    else if (error == OTA_RECEIVE_ERROR) Serial.println("Receive Failed");
    else if (error == OTA_END_ERROR) Serial.println("End Failed");
  });
  ArduinoOTA.begin();

  // Set up mDNS responder:
  // - first argument is the domain name, 
  // - second argument is the IP address to advertise
  //   we send our IP address on the WiFi network
  if (!MDNS.begin(hostname, WiFi.localIP())) {
    Serial.println("Error setting up MDNS responder!");
    while(1) {
      delay(1000);
    }
  }
  Serial.print("mDNS responder started: ");
  Serial.print(hostname);
  Serial.println(".local");
  httpUpdater.setup(&httpServer);
  httpServer.begin();
  MDNS.addService("http", "tcp", 80);
}

void setup_wifi() {
  delay(10);
  // We start by connecting to a WiFi network
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(wifi_ssid);

  WiFi.begin(wifi_ssid, wifi_password);

  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }

  Serial.println("");
  Serial.println("WiFi connected");
  Serial.println("IP address: ");
  Serial.println(WiFi.localIP());
}

void reconnect() {
  // Loop until we're reconnected
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    // Attempt to connect
    // If you do not want to use a username and password, change next line to
    // if (client.connect("ESP8266Client", mqtt_username, mqtt_password)) {
    if (client.connect(mqtt_clientId)) {
      Serial.println("connected");
      delay(1000);
      publishTempConfig();
      delay(1000);
      publishPressConfig();
      delay(500);
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
    }
  }
}

float convertToF(float tempVal) {
  float fah;
  fah = 9.0 / 5.0 * tempVal + 32;
  return fah;
}

bool checkBound(float newValue, float prevValue, float maxDiff) {
  return newValue < prevValue - maxDiff || newValue > prevValue + maxDiff;
}

void publishData(float p_temperature, float p_pressure) {
  const size_t bufferSize = JSON_OBJECT_SIZE(2);
  DynamicJsonBuffer jsonBuffer(bufferSize);
  JsonObject& root = jsonBuffer.createObject();
  // INFO: the data must be converted into a string; a problem occurs when using floats...
  root["temperature"] = (String)p_temperature;
  root["pressure"] = (String)p_pressure;
  root.prettyPrintTo(Serial);
  Serial.println("");
  char data[74];
  root.printTo(data, sizeof(data));
  client.publish(TOPIC_STATE, data, true);
}

long lastMsg = 0;
float diff = 1.0;
float temp = 0.00;
float tempF = 0.00;
float pressure = 0.00;

void loop() {
  client.loop();
  //start the OTA handler
  ArduinoOTA.handle();
  httpServer.handleClient();

  long now = millis();

  if(now - lastMsg > 1000) {
    lastMsg = now;

    if (!client.connected()) {
      reconnect();
    }

    /* Get a new sensor event */
    sensors_event_t bmpEvent;
    bmp.getEvent(&bmpEvent);

    if (bmpEvent.pressure)
    {
      bmp.getTemperature(&temp);
      float newPress = bmpEvent.pressure;
      float newTempF = convertToF(temp);

      if (checkBound(newTempF, tempF, diff)) {
        tempF = newTempF;
        Serial.print("New temperature:");
        Serial.println(String(tempF).c_str());
        publishData(tempF, pressure);
      }

      if (checkBound(newPress, pressure, 3.00)) {
        pressure = newPress;
        Serial.print("New pressure:");
        Serial.println(String(pressure).c_str());
        publishData(tempF, pressure);
      }
    } else {
        Serial.println("Sensor error");
    }
  }
}
3 Likes

Thanks for the sketch. I had no idea about arduinoJson.h and was wondering about how to format the payloads.
I had to change the bit of code I used as arduinoJson is now version 6 not 5. So my code snippet to notify HA that I’ve got a new Humidity sensor connected to my testbed ESP32 is:

void publishHumConfig() {
  DynamicJsonDocument root(1024);
  root["name"] = "ESP32 Humidity";
  root["stat_t"] = "tele/ESP32/SENSOR";
  root["pl_avail"] = "online";
  root["pl_not_avail"] = "offline";
  root["uniq_id"] = "000001_ESP32_Humidity";
  root["unit_of_meas"] = "%";
  root["dev_cla"] = "humidity";
  root["frc_upd"] = true;
  root["val_tpl"] = "{{value_json['ESP32']['Humidity']}}";

  serializeJsonPretty(root, Serial); //Prints it out Prettily!
  Serial.println();
  char message[1024];
  serializeJson(root, message);
  Serial.println(message); //Prints it out on one line.
  client.publish("homeassistant/sensor/000002_ESP32_Temperature/config", message, true);
}

I’ve not got the content as I want it yet but it gives an idea. I’m struggling with a nested/nested array to give …,“dev”:{“ids”:[“000002”]},… Can’t quite work it out. Be easier to just type it as a string!

I quite struggled with defining a device, too - so I thought sharing my current solution for a device containing multiple entities and autodetection could be helpful to someone.

Currently my sensor contains a LDR and the BME280 (temperature, humidity, pressure)

Here just the relevant code:

/************* MQTT TOPICS  **************************/
const string SENSOR_ID {"sn01"};

const string state_topic{"homeassistant/sensor/" + SENSOR_ID + "/state"};
const string set_topic{"homeassistant/sensor/" + SENSOR_ID + "/set"};

// {<entityId>, <entityUnit>, <entityClass>}
const vector<vector<string>> ENTITIES{
    {"ldr", "%", "illuminance"},
    {"temp", "°C", "temperature"},
    {"hum", "%", "humidity"},
    {"pres", "hPa", "pressure"},
};

// ...

void sendState()
{
  // Save the last time a new reading was published
  lastPublishMillis = millis();

  DynamicJsonDocument configPayload(1024);

  //////////////////////////////////////////////////////
  ///////// Shared config for every entity /////////////
  //////////////////////////////////////////////////////

  configPayload["stat_t"] = state_topic.c_str();
  configPayload["expire_after"] = EXPIRE_AFTER;
  configPayload["dev"]["name"] = SENSOR_ID.c_str();
  configPayload["dev"]["model"] = "NodeMCU";
  configPayload["dev"]["manufacturer"] = "Manufacturer";
  JsonArray identifiers{configPayload["dev"].createNestedArray("ids")};
  identifiers.add(SENSOR_ID.c_str());

  //////////////////////////////////////////////////////
  ///////// Setting the config for each entity /////////
  //////////////////////////////////////////////////////

  for (auto entity : ENTITIES)
  {
    const string unique_id {SENSOR_ID + "_" + entity.at(0)};
    const string value_template {"{{value_json['" + entity.at(0)+ "']}}"};

    configPayload["name"] = unique_id.c_str();
    configPayload["uniq_id"] = unique_id.c_str();
    configPayload["val_tpl"] = value_template.c_str();
    configPayload["unit_of_meas"] = entity.at(1).c_str();
    configPayload["device_class"] = entity.at(2).c_str();

    const string config_topic_entity{"homeassistant/sensor/" + SENSOR_ID + "_" + entity.at(0) + "/config"};

    char configPayloadSerialized[1024]{};
    serializeJson(configPayload, configPayloadSerialized);
    Serial.println(config_topic_entity.c_str());
    Serial.println(configPayloadSerialized);

    // Publishing
    bool publishSuccessful {0};
    if (client.publish(config_topic_entity.c_str(), configPayloadSerialized, true))
      publishSuccessful = 1;

    const string publishMessage {
      "config " + entity[0] + (publishSuccessful ? " " : " NOT ") + "published\n"
    };
    Serial.println(publishMessage.c_str());
  }

  ////////////////////////////////////
  //////// updating the state ////////
  ////////////////////////////////////

  // Inverting value to receive bright = high numbers + converting to %.
  const unsigned short LDRPIN_MAX{1024};
  const float PERCENTAGE_DIVIDER{10.24};
  LDR = (LDRPIN_MAX - analogRead(LDRPIN)) / PERCENTAGE_DIVIDER;

  // New BME280 sensor readings
  // Celsius:
  temp = bme.readTemperature();
  // Fahrenheit:
  //temp = 1.8*bme.readTemperature() + 32;
  hum = bme.readHumidity();
  pres = bme.readPressure() / 100.0F;

  DynamicJsonDocument sensorData(1024);
  sensorData["ldr"] = round(LDR*10)/10;
  sensorData["temp"] = round(temp*10)/10;
  sensorData["hum"] = round(hum*10)/10;
  sensorData["pres"] = round(pres*10)/10;

  char buffer[1024]{""};
  serializeJson(sensorData, buffer);
  Serial.println(buffer);

  if (client.publish(state_topic.c_str(), buffer, true))
    Serial.println("state published");
  else
    Serial.println("state NOT published");
  Serial.println();
}

Important for a device including multiple entities is that each entity has to be configured independently to the config_topic. Something like
homeassistant/sensor/<sensorId>_<entityName>/config
E.g. for the ldr:
homeassistant/sensor/sensornode1_ldr/config

Sending the state is then handled to the same stateTopic that’s defined in the entity’s json (in my example within “shared config”):
homeassistant/sensor/sensornode1/state

6 Likes

@manu17, Just wanted to say thanks. The snippets you posted helped me figure out why my MQTT Discovery wasn’t working correctly.

The only additional thing I’d note for others having issues, especially if the payload isn’t publishing to the topic, is you may have to adjust the MQTT_MAX_PACKET_SIZE in <PubSubClient.h> or using client.SetBufferSize(1024);

2 Likes

Can you share an example of the full JSON outout?

Would highly recommend using GitHub - dawidchyrzynski/arduino-home-assistant: ArduinoHA allows to integrate an Arduino/ESP based device with Home Assistant using MQTT.