2022.6: Gaining new insights!

I thought I would provide you with my experience…I have moved all my MQTT YAML configs to the new format and running 2022.6.2, the MQTT reload works. I will add that after about 20-30 seconds I do get a lot of errors but they are related to cover groups which seems to be an issue with the group utility code and not the MQTT reload specifically. I find also that my MQTT entities take about the same 20-30 seconds to show up again after reload, so I deduce that it takes this long for the reload to actually process. During this 20-30 seconds I now get an occasional warning about some other integration taking more than 10 seconds to do something. But nevertheless all works when finished reloading.

You might be interested in an another user’s comments regarding loading time. I had helped them convert from defining MQTT entities via YAML to MQTT Discovery. Basically, you make a script that publishes the configuration information in the format accepted by MQTT Discovery and it creates the entities. It also lets you define devices which is something you can’t currently do in YAML.

The user converted 130 entities, from YAML to MQTT Discovery, and reported the following observation:

Loading time for MQTT reduced from 68 to 5.9 seconds

An example of a script, to create an MQTT entity via MQTT Discovery, is posted here and is followed by the user’s reply.

2 Likes

Thanks. I actually spent a fair bit of time reading that post a few months ago proving the concept out (to myself), but never took the time to convert mine over. I had not picked up on the time savings. Again thanks for the tip :slight_smile:

You’re welcome!

FWIW, I have about four dozen MQTT entities and all are configured via MQTT Discovery (so I am unaffected by the recent changes to the format of YAML-based MQTT configuration). According to the System Health page, the MQTT integration took 2.65 seconds to load.

That’s pretty neat. I always used Mqtt discovery for almost everything so I never had a perf comparison.

Another cool thing with that route is you could put that script in a separate folder that folder watcher watches. Then you could make an automation that triggers when that script is modified and does:

  1. check config
  2. reload scripts
  3. run the script

Then you’ve got automatic reloading of Mqtt entities after a config change without a restart as well (another issue others have faced above from what I’ve been reading). And fully automated!

Setting up from YAML is usually slower for any integration. Setting up a lot of yaml platforms can painfully slow (as observed above).

That’s partially driven by python overhead (will be much better in python 3.11), and the yaml parser being slow because of the enhanced error reporting (we have to keep that). The new python version will improve the yaml performance as well but we are still likely 16-20 months away from being able to upgrade since it’s not even released yet.

Some light reading Python 3.11 Performance Benchmarks Are Looking Fantastic - Phoronix

Generally anything you can move out of yaml will improve your startup time.

1 Like

I’ll give that a try for YAML-based helpers because they take surprisingly long to load.

My Mqtt was always atop of the integration loading times (over 450 sec…) and now has sunk to a measly 7th spot with

all in Yaml… so other improvements have been made too.

Problems here with mqtt integration. I have a dozen devices that are in the mqtt integration through discovery and work well. There are three that are not. They are all 3-way switches with rules and use of EVENT message instead of power1.

I posted on this site in another area. It shows my configuration.yaml, tasmota setup, and the rules.

The problem is that HA will not send ON payloads via the UI until HA wakes up (for lack of a better term). I have to use an mqtt tool (like HA’s mqtt integration publish page or mqtt_lens). Once this happens, HA ‘wakes’. As a fix for now, I have an automation at HA startup that sends ON and OFF payloads to wake up HA.
The set up was working well for a couple of years under the old platform: mqtt setup but is now broken. Control of the switches works as expected through an mqtt tool.

That might be the incentive I need to move my many helpers to the UI.

Have you looked in the database to see if that weird value is in there? I am not seeing any such issues.

Yes it is a compelling reason, there doesn’t seem to be a good “yaml is best” argument for not using UI based helpers.

But doesn’t this create a problem with sharing code?

I have a package that I share and as far as I can tell has quite a few (by which I mean probably at least ten, maybe a lot more) people using it and it relies on a lot of helpers.

I don’t believe I could sensibly share it if I moved to UI based helpers.

I also have a severe problem with reloading MQTT.

When I click the UI I get an error both in the UI and in the log.

I saw the bug report Reload MQTT entities takes down system · Issue #73275 · home-assistant/core · GitHub

And added my log entries. And MQTT stops working totally for everything. I have to restart HA to get it alive again.

This is a very recent bug. I have loaded manual MQTT entities so many times before and also just weeks ago.

The only way I can reload MQTT is to restart Home Assistant. This is really nasty

1 Like

I have 2 counters in my system, both UI created Helpers

so one would expect that to happen in the blink of an eye. And yet:

is indicative of an issue?

Id be glad to move all of my 130 input_booleans from yaml to UI, but given:

I wonder if that would be worth the trouble…

That’s way too long for those. Something is either I/O bound (your disk is too slow), cpu bound (your cpu is too slow), blocking your event loop at startup (usually a bug in an integration), or you just have so many integrations the overhead of loading the python code slows the system down.

using an external SSD (only reads the startup from SD)

RPi4 4 Gb so that could be…?

that is logged isnt it if so? Not the case .

that might be it :wink:

I do also have a lot of entities, and even though I drastically cut my yaml config on them (eg template sensors) HA keeps adding them back because of all attributes turning into standalone sensors…

Anyways, other than global debug level (which isnt good for system performance here) could I up a single logger component to check any of the above?
Or send in a Profiler snapshot maybe?

to give you some more insight, the list is currently topped by template (yaml obviously) and Threshold (14 in all), the latter are all UI configured now…

ofc, these are based on other entities, mqtt sensors in my case, so it might be an internal dependency thingy too.

Only if they do the 2-3 most common mistakes. We catch new ones every month.

You probably need to do a py-spy at startup time.

We are getting off-topic from the release though so this should likely be a new thread.

I decided to check mine…

image

image

To be honest I don’t think I’d see much benefit :man_shrugging:

EDIT:

Yeah sorry, I was too quick to comment. Please link here if you continue elsewhere, I’d like to follow this.

Well maybe the last update/compress did the trick. It’s been running for 48 hours now with no problems. But to answer your question, yes the data was making it to the DB and once it goes weird it pretty much stayed weird. Will keep an eye on it. Thanks for being interested.

FYI, the 2022.6.6 release sped up my MQTT reload time down to around 5-6 seconds.
HA restart still has MQTT taking around 30-40 seconds in my case.

[EDIT] My mistake. It was release 2022.6.5