Automated way to generate MQTT Sensors?

Hi,

I was reading the MQTT discovery docs but I am unsure if this matches my case…

I am populating a mqtt topic with weather forecast values through json-string:

  "wetter_vorhersage": {
    "vorhersage_1h": {
      "clouds": 27,
      "humidity": 63,
     },
    "vorhersage_2h": {
      "clouds": 26,
      "humidity": 62,
     },
...

Of course I can configure each sensor manually in configuration.yaml as follows:

 - platform: mqtt
   state_topic: hzg/status
   name: "wv1_clouds"
   unit_of_measurement: "%"
   force_update: true
   value_template: '{{value_json.wetter_vorhersage.vorhersage_1h.clouds}}'

But with adding forecast hours (up to 24) you can imagine this is not really useable- besides the issue when something changes. then I have to edit at least 24 items.

The MQTT messages are fully configurable for me- so I can easily change them to match the needs of HA discovery. Just: how to?
As far as I understood I need to have a configuration topic where the sensor will send its configuration structure.
And once a message was send in configuration topic HA is supposed to read the data from state topic.

For my case it appears to use the multiple values section.
So I will need a configuration and a state topic for every forecast hour, right? Will be 48 topics for 24h forecast…

But I did not get it how to tell HA to read the sensors. How does HA know it has to read the configuration topic? And how does HA know it has to count (hour++) them?

Is there a step-by-step example how to use?
Thanks a lot!

/KNEBB

You make an automation that creates the configuration topic, or your device has to announce it when HA restarts.

That’s based on your discovery topic. If the information you pass to HA on restart tells home assistant what to do and when to do it, home assistant will just follow your directions.

The discovery topic is basically instructions and then home assistant follows the instructions.

Yes, you’re looking at it in the docs you linked. All the examples at the bottom are discovery information for each device type. You put that information into a discovery topic and home assistant will run with that information.

1 Like

You might find this useful if you scroll down the README: GitHub - devWaves/SwitchBot-MQTT-BLE-ESP32: Allows for multiple SwitchBot bots and curtains to be controlled via MQTT sent to ESP32. ESP32 will send BLE commands to switchbots and return MQTT responses to the broker. Also supports Temperature, Motion, Contact sensors

devWaves has given a full write up on how MQTT discovery is implemented for his switchbot integration

Can you give me an example of this? It is currently the one thing I do not understand…

So from wording I have:

  • discovery tropic
  • configuration topic
  • state topic

Ok, in the discovery topic I have to put items so HA will subscribe to the discovery topic and from this topic knows the devices where it will poll the configuration topics to get the information from state topic, right?

Ok, I will need an example for the discovery topic to be used. Or how to configure HA to subscribe to the discovery topic…

@TheHolyRoger : I checked your link but I have not seen (even not in the video) how to configure HA… I saw how to design the configuration and state topics. But this is currently not my issue. Thanks anyways, interesting link!

Thanks!

/KNEBB

The discovery topic is the config topic.

So you’ll only have:

  • Discovery/config topic
  • state topic

You might find the following post useful. I explained to another user how to use scripts to create MQTT Sensors via MQTT Discovery. It’s the technique I have used to create all of my MQTT-based entities (sensor, binary_sensor, lock, climate, etc).

I had the same problem with my miflora sensors with tasmota mqtt messages. so i wrote a samll phyton script that generates mqtt discovery messages. Works for me like a charm. maybe that is helpfull for you too.

TBail/miflora_tasmota2ha_discovery_messenger (github.com)

1 Like

Hi guys,

great thank to you all trying to explain it to me. Still, I feel a little bit dumb as it appears to be so easy for you guys and I still do not get it…

Oh, wait…while writing: may I have misunderstood the whole autodiscovery stuff? Is it HA sends configuration information to the device which configure themselves according to the payload in configuration topic???

I expected it the other way round- HA will poll (preconfigured) topics and based on this add the found devices.

So I need to tell my devices to listen to the configuration topic and update its state according to the configuration found there?

Still, I have to generate the script to define the topics and payloads for every 24 devices, right? My initial intension was so save typing and do some automated generation of the sensors, but the autodiscovery will not save typing, instead I will have more to type in the script…

If so I got it totally wrong and you can forget the following:

=====================snip===========
What did I get so far?

I need a script which generates some sort of entries in HA. There I have to define the configuration topics, right? So HA will then execute this script and the script will tell HA where to look for devices. Once a message has been posted in the configuration topic HA will poll the state topic for the sensor’s state and use this. Correct so far?

An example of such a script is this (thanks to @123 ):

script:
  create_sensors:
    alias: "Create sensors via MQTT Discovery"
    sequence:
      - service: mqtt.publish
        data:
          topic: homeassistant/sensor/temperatura_dnevna/config
          retain: true
          payload: >
            {
              "name": "Temperatura Dnevna",
              "unique_id": "esp9_temperature_dneva",
              "device_class": "temperature",
              "unit_of_measurement": "°C",
              "state_topic": "home/esp9/dnevna",
              "value_template": "{{ value_json.temperatura }}",
              "device": {
                  "identifiers": ["esp9"],
                  "name": "ESP9",
                  "model": "ESP",
                  "manufacturer": "Espressif",
                  "sw_version": "1.XX"
              }
            }

I am missing to things to understand completely:
First, where do I place the script and how is it executed? (may be a little bit off topic but will help)

Second, does the script mention all configuration topics? The above example will (as far as I understand) create a single entry to discover a sensor for temperature. So for a second sensor I have to add nearly the same to the above script (ie replace name and stuff in topic string), correct?

Or can I use some sort of variables in the script to tell them: “hey, do this twenty times and replace the variable for every run with the current run”? If not, I do not see advantage of the script compared to directly configure the sensors…

Ok, what Do I have so far? I will configure my script:

script:
  create_sensors:
    alias: "Create iforecast sensors via MQTT Discovery"
    sequence:
      - service: mqtt.publish
        data:
          topic: homeassistant/sensor/forecast_VARIABLE/config
          retain: true
          payload: >
            {
              "name": "Temperatura in VARIABLE hours",
              "unique_id": "forecast_temp_VARIABLE",
              "device_class": "temperature",
              "force_update": "true",
              "unit_of_measurement": "°C",
              "state_topic": "home/forecast/VARIABLE/",
              "value_template": "{{ value_json.temp }}",
              "device": {
                  "identifiers": ["forecastVARIABLE"],
                  "name": "Forecast Temp VARIABLE",
                }
            }

Am I right about the devices itself do not need to do anything with the configuration topic as it is solely for HA? From what I see in the script it appears to me HA executes the script and sends the payload to the configuration topic…

/KNEBB

Don’t worry i am just on step ahead :slight_smile:

Idea is: A device with mqtt send a discovery message thats defines an entity for the dvice. if there are more entities for that device these could be combined as a device. This is all done by using the discovery message(s). So the device announces its capabilities to HA.

If the device does not create the auto dicovery message on its own, you have the options to create entities in configuration.yaml or you create you own discovery message. Benefit of discovery message is that you are able to combin entities to a device. So it is more structures in the UI.

Creating discovery message could be done in some ways.

1.) Creat it manually in an mqqt Client like MQTT Explorer. Not very handy

2.) Write a python script like i did it. I like that

3.) Creat a scripte in HA as shown in you last post. This script could be executes in the dveloer areas in the services tab (Hopefully someone could comfirn that, because i have never done that)

At the end all the methods are doing the same. Sending a discovery Message to HA. If you find an error in the definition of you sensor, just send an correct message to the same topic and HA will pick it up. Deleting an sensor could be done using an mqtt Client linke mqtt explorer. You need to delete the Message with the client, because typically the discovery messages are retained by the broker. As soon as the message ist deleted the sensor will be deleted from you HA instance as well.

Hope this helps a little bit.

1 Like

Ok, that brings a little bit light here. Still not fully, sorry :wink:

Lets take the route of devices announcing capabilities, no script.

I can make my devices to send discovery messages to the discovery topic. But how do I tell HA to listen to this single (and in my case the other 23) topic?

Thanks
/KNEBB

OK, in HA you need the mqtt integration to be enables. With that HA could receive mqtt messages. The discovery topics are defined and HA will subscribe automatically the these messages.

And after receiving such a message the device/entity is created.

You keep asking this and it’s been answered. The discovery information tells HA this. I’m not sure how else to explain it.

It’s the same if you were to make a configuration by hand in yaml. That defines the entity and home assistant just knows what to do with it.

MQTT does not poll, it’s live reactions. AKA it listens to the topic and updates your sensor when changes occur to the topic.

It would be like if you handed a blueprint to a contractor. They just know how to build it after they get the blueprint. Discovery or MQTT Configuration do the same thing, they give home assistant the blueprint and home assistant runs with it.

It was really unclear how the stuff works. Thanks to @Dibbler I got it now. So the devices publish their configuration values through the discovery topic to HA. And HA uses these discovery topics to create its entities. Got it so far now. I am currently telling my devices to send these information.

I tried now with just the command line mosquitto tools and indeed it worked really nicely. I have the to-be-used- sensor available automatically in HA now and it it getting its proper state. Perfect! This will prevent me a lot of typing.
I did the following and the sensor magically appeared:

root@zentrale:/src/mqttd# mosquitto_pub -t homeassistant/sensor/wetter-forecast-1h-temp/config -m '{"device_class": "temperature", "name": "Temperatur", "state_topic": "homeassistant/sensor/wetter-forecast-1h/state", "unit_of_measurement": "°C", "value_template": "{{ value_json.temp}}" }'
root@zentrale:/src/mqttd# mosquitto_pub -t homeassistant/sensor/wetter-forecast-1h/state -m '{"temp":233.2, "bla": 5}'

BTW:
Just as a side note I am wondering howw HA recons the discovery topics- the number of possible topics is infitity…

Anyways, will work now! Thanks a lot for your patience!

/KNEBB

1 Like

the exact same way it knows when to update a sensor. It watches each discovery topic and acts upon it when changes occur.

Yes it’s infinite. Yes you can create an automation that automatically creates 100s or 1000s of sensors on home assistant startup. This is what many of us do when we want virtual sensors.

1 Like

Typically the config message has the retain flag that keeps the sensor available if your mqtt broker restarts.

1 Like

Finally I managed it!

It took me quite a while to tell my C programm how to generate and send the messages but I succeeded.
My program now initially sends a message to the topic homeassistant/sensor/wetter-vorhersage-24h-temp/config:

{
  "device_class": "temperature",
  "name": "wetter-vorhersage-24h-temp",
  "state_topic": "homeassistant/sensor/wetter-vorhersage-24h/state",
  "force_update": "yes",
  "unit_of_measurement": "°C",
  "value_template": "{{value_json.temp}}"
}

And in the above configured state topic of homeassistant/sensor/wetter-vorhersage-24h/state I send as non-retain:

{
   "clouds": 63,
   "humidity": 72,
   "pressure": 1017,
   "wind_deg": 268,
   "rain":   0.0,
   "snow":   0.0,
   "dew_point":   9.1,
   "temp":  14.2,
   "feels_like":  13.5,
   "wind_speed":   6.4
 }

Aaaand I can see the new sensors without further configuration in HA:

In the end it has been more work than just adding these sensors to HA manually. But the point is: I can simply change the sensors in my programm without the need of reconfiguring HA!

Thanks to all!
/KNEBB

1 Like

For this purpose I am using python and API - directly call sensor, however it has also its limitation.