Long setup time for MQTT integration

Restarts of HA are taking quite some time on my machine and I found out, that the main issue seems to be MQTT integration - it takes over 60 seconds to finish setup!

Granted, some integrations need more time than others (f.e. LocalTuya needs 15 secs, Mobile App needs 11 secs), but more than a minute seems a bit too much.
Anyone else with similar issue and/or advice hoiw to reduce it? Or is this normal?

Seems normal to me.
Hereā€™s my startup time for mgtt:
mqtt

Running on a I3 nuc. w/ssd.

EDIT:
After watching and reading this post evolve over time I remembered that I still had a vast majority of my Tasmota equipment still configured via yaml, wile running Tasmota FW 9.5 with the auto discovery on.
I guess it slipped my mind.

However after a few hours of reconfiguration my MQTT startup time is now down to 34.40 s

I still have a few hours of .yaml entries to go thru and remove and convert all automationā€™s associated with the .yaml entries.

Just a update for completeness sake.

12 year-old laptop with Core2 Duo and SSD.

Screenshot_20211110-180738~2

Intel NUC i3 with 128GB SSD, and my MQTT takes ~73 seconds to start up.

The actual restarts of HA itself are lightning quick though, but it does take a minute or so before it is fully up and running with all sensors reporting back correctly. The fact that is take a bit of time to fully display properly really hasnā€™t bothered me ā€¦ but then I only restart when I need to (upgrades, I notice something is playing up).

The MQTT integration is basically logging into the MQTT broker and subscribing to topics (for all the MQTT-based entities you have).

For the people with long MQTT startup times, how many MQTT entities do you have (41 in my case)?

16 devices and 86 entities.

Pi 4 , using a separate broker :

image

image

Well, thatā€™s weirdā€¦ no devices and no entities.

png

Iā€™m only using the integration (so it seems) to publish MQTT messages in certain cases - otherwise I use zigbee2mqtt for sensors (33 devices and 246 topics) and the broker runs on my Pi4 as a Supervisor addon.

It seems mqtt entities manually defined in .yaml donā€™t show up there. Just removed a switch I created yesterday on one of my HA instances to help someone on the forum, and the number of entities was not changed.
Devices can only be created by auto-discovery, so that is normal in that case.
Just learned myself I have more than 506 mqtt entities :slight_smile:

The fact that francispā€™s system subscribes to topics representing hundreds of entities in under 10 seconds suggests the process is quick and efficient. If it takes over a minute then either there are thousands of topics involved (rare) or thereā€™s something else responsible for the delay (likely). Iā€™d be interested to know what that ā€˜something elseā€™ could be.

In my case, the broker is the Mosquitto Add-on hosted by the same machine as Home Assistant. All entities are configured via MQTT Discovery. I am also using the Tasmota integration which also employs MQTT for communication. That has 260 entities and takes 0.57 seconds to load. :man_shrugging: It would appear that thereā€™s more to this puzzle than merely the number of entities involved.

The biggest difference between mine and your installation seems to be, that I donā€™t use solely MQTT Discovery, but have a bunch of MQTT sensors defined in sensor.yaml (140 to be exact). Iā€™m not sure if I dare removing them for a test, to see if it loads faster without them (donā€™t want to break something, that works :slight_smile: ).
Otherwise also my broker is an Addon runing on the same machine as HA. And besides the zigbe2mqtt (19 devices) I use MQTT for some other internal communication between HA and four Android phones and 11 ESP32 and ESP8266 microcontrolers.

OK, thatā€™s one possible explanation. Does your Pi4 use an SD card or SSD?


In the ā€œFWIWā€ department:

I have a test system hosted by an RPi3 with an SD card. Although it isnā€™t configured with MQTT, there are other integrations it has in common with my production system. The RPi3 is noticeably slower than the ancient Core2 Duo with SSD. Loading input_booleans takes over 10 seconds (which causes a warning message in the log noting the long delay).

SSD, much safer and faster :).

Btw, in the meantime I tried removing 80 out of 140 manualy defined entities in sensor.yamlā€¦ loading time reduced from 67 to 28 secondsā€¦ I guess I have my answer :).

Well, removing its workload is bound to reduce its loading time. What would be interesting to see is if you make them discoverable to MQTT Discovery and compare the loading time.

If you are interested, I explained how to use a script to create an entity via MQTT Discovery:

You can also define devices this way because manual MQTT configuration only allows you to define entities, not devices.

Tried turning on Autodiscovery and it found my 19 Aqara devices with corresponding 100 entities right away. Now, Iā€™m sure it would reduce loading time and once day I might even try converting everything, but right now Iā€™m not sure itā€™s really worth the effort as I would have to:

  1. Manually rename the entities, that autodiscovery found, to match the manualy defined entities - otherwise I must change all cards, automations etcā€¦
  2. Make also other (the missing 40) MQTT entities autodiscoverable (not even sure how to do that on ESP and android machines) and then repeat the process from #1.

So, if I find myself one day having too much free time and/or long setup time bothering me too much, I might try it, but for nowā€¦ well, one minute at rare restarts is not too annoying, is it? :smiley:

How much did it reduce loading time? What was reported for MQTT in Info?

Easily done using the script technique I described above.

Thatā€™s your call; you created this thread out of concern for the length of the loading time. Now you know itā€™s much longer than for users with far more MQTT-based entities.

Yes, I know, and Iā€™m really thankfull for all your inputs. What Iā€™m trying to say is, that at the moment Iā€™m not sure if solving this CF Iā€™ve created through time (as I didnā€™t know the right way of doing it) is worth the workload compared to waiting one minute at each restart (btw, I misscalculated before, autodiscovery found 100 entities, but I was only using 40 of them - so I would have to newly autodiscover the missing manually defined 100 of them and some of them are even templates, so I would have to have an additional sensor anyway).
Iā€™ve learned a lot from this and from now on Iā€™ll use autodiscovered entities when adding new stuffā€¦ but solving the legacy issues will wait for better times :).

Well, deep inside I know, Iā€™ll have to deal with it somedayā€¦ so if itā€™s not too much trouble for you, could you show me on my example, how would I make the esp9 autodiscoverable (the end result should be 6 entities - 4 from json and two separate ones)?
Iā€™m afraid Iā€™m unable to create the script myself from the example you providedā€¦

Are your ESP-based devices flashed with Tasmota or ESPHome?

If not, you can create a script that employs MQTT Discovery to create entities. Post your existing MQTT sensor (s) configuration for the ESP9 device. That information is needed in order to create the script.

Neitherā€¦ they were programmed using Arduino IDE. At the time of programming (when I just started my HA joruney) I had no knowledge about anything and therefore made some crappy decisions, which I can not easily correct now (re-programming of all ESPā€™s is not an option anymore). Therefore I assumed, that I can correct at least Autodiscovery in a way you suggested - by posting certain messages to ā€œconfigā€ topics for existing devices.

So, there are six sensors for esp9, and they are:

- platform: mqtt
  state_topic: 'home/esp9/dnevna'
  name: 'Temperatura Dnevna'
  unit_of_measurement: 'Ā°C'
  value_template: "{{ value_json.temperatura }}"

- platform: mqtt
  state_topic: 'home/esp9/dnevna'
  name: 'Vlažnost Dnevna'
  unit_of_measurement: '%'
  value_template: "{{ value_json.vlaznost }}"

- platform: mqtt
  state_topic: 'home/esp9/dnevna'
  name: 'Datum Dnevna'
  value_template: "{{ value_json.datum }}"

- platform: mqtt
  state_topic: 'home/esp9/dnevna'
  name: 'Čas Dnevna'
  value_template: "{{ value_json.cas }}"

- platform: mqtt
  state_topic: 'home/esp9/pinstatus'
  name: 'Senzor Dnevna'

- platform: mqtt
  state_topic: 'home/esp9/dummy'
  name: 'Status Dnevna'

Please note, that (again, based on my begginerā€™s mistakes) the first four sensors are sent in one topic as json and the last two are sent in separate topics as raw text - if that is relevant for the script.