WTH why do we need to restart HA core so much?

[fodi666]

Hi,
I love HA but there is one thing that still bothers me: why do I need to restart the whole core if I add a new integration? I know that after last year’s WTH month quite some integrations got buttons to reload without a restart but it is not nearly all of them.
It would be great if we could restart every integration without restarting or even start an integration (especially when set through yaml) without restarting HA core.
Thanks

[frenck (Franck Nijhof)]

Which integrations are you referring too?

As in, could you give some examples?

[fodi666]

• for example the last one I defined was the GTFS integration and had to restart each time I made a change
• all the HACS integrations need a restart before using

[frenck (Franck Nijhof)]

@fodi666

for example the last one I defined was the GTFS integration and had to restart each time I made a change

That one specifically could use quite a bit of love. It is a very old-school integration. If it was modernized it would support it.

@fodi666

all the HACS integrations need a restart before using

We can’t control the features of third-party integrations. The capabilities are there, it is up to the creators of that custom integration to enable it.

…/Frenck

[fodi666]

@frenck

That one specifically could use quite a bit of love. It is a very old-school integration. If it was modernized it would support it.

That is true and I agree that it could be lot nicer to work with.

@frenck

We can’t control the features of third-party integrations. The capabilities are there, it is up to the creators of that custom integration to enable it.

Why should this capability be defined as a per-integration basis? Why should not the HA Core go through each loaded integration and allow reloading it and running the same stuff as it does during a restart? Or even why could it not load the integration from the UI with a click if there is a new one in the config or in a watched folder (custom integration for example) instead of loading the code only on restart?

[frenck (Franck Nijhof)]

@fodi666

Why should this capability be defined as a per-integration basis?

It’s a technical answer, but we can’t know what integration has to do to unload. They have to specify that. For example, disconnect from a device first (or not), maybe unregister a webhook remotely, clean up timers that are running, and whatever more.

[dawiyo]

Being able to reload sensors and covers would be a great addition.

[fodi666]

fair enough, but don’t you do the same checks when restarting the core? somehow the integrations need to quit

[frenck (Franck Nijhof)]

@fodi666

fair enough, but don’t you do the same checks when restarting the core?

Yup, the difference is that the whole process is shut down (meaning everything) instead of having to pick specific things. In such cases, there is no memory management or things like connections that stay open we have to take into account.

[frenck (Franck Nijhof)]

@dawiyo

Being able to reload sensors and covers would be a great addition.

Those are entity components, not integrations. Nothing creates them, so they cannot be set up or related. They are provided by others. Which integration specifically are you talking about?

[Cougar]

Can we can add a new optional “reload” hook for integrations and use it if available? Of course it needs to be well documented and user should see/know what changes can be hot reloaded and what are the ones that need core restart

[frenck (Franck Nijhof)]

@Cougar

Can we can add a new optional “reload” hook for integrations and use it if available?

I’m not sure what that means. If an integration is capable of unloading/restarting, it will already offer that possibility.

[dawiyo]

@frenck

@dawiyo

Being able to reload sensors and covers would be a great addition.

Those are entity components, not integrations. Nothing creates them, so they cannot be set up or related. They are provided by others. Which integration specifically are you talking about?

I’m guessing I’m not talking about an integration at all then. I was referring to things like the scrape sensor (which I would love to be able to add an ID to so I can hide it), and template covers.

[frenck (Franck Nijhof)]

@dawiyo

scrape sensor

That indeed doesn’t have that ability

@dawiyo

and template covers.

All templates can be live-reloaded.

[Cougar]

What about custom integrations (coming via HACS)? Is there any way to simply reload them after upgrade and can be implemented by integration devs or is it still HA limitation that the whole core needs to be reloaded?

[frenck (Franck Nijhof)]

@Cougar

What about custom integrations (coming via HACS)?

That I already answered above:

@frenck

We can’t control the features of third-party integrations. The capabilities are there, it is up to the creators of that custom integration to enable it.

[frenck (Franck Nijhof)]

@Cougar

Is there any way to simply reload them after upgrade

That is a different thing btw… An upgrade is change of source code, only a restart can solve that.

[e-raser]

Exactly what came to my mind at 1st when reading the WTH month blog post! It even is my number 1 point in my personal HA documentation I started two years ago when starting to use HA, section “What really annoys me”:

Need for restarting HA Core for reloading components (integrations, alert, many YAML based stuff)

• I really hated this right from the beginning. I really appreciate that a lot of things can be reloaded meanwhile using the dev-tools yaml section. But not everything, not at all. Think about the famous, so powerful and so little love-receiving alert integration.
• And some things (like “command_line sensors”) create issues: once I reload command line sensors, they are reloaded - but at the same time a duplicate (shadow entity) is created, e. g. “sensor.thing_a” will have a “sensor.thing_a_2” brother, after another reload another sibling “sensor.thing_a_3” joyns the party and so on. (see WTH why do we need to restart HA core so much? - #19 by petro) The “fix” is: a full HA restart.
• Back to the root cause, need for restarting HA:
  • restarting creates a downtime (“single unplanned/planned maintenance window”) of at least 6 minutes in my case (Pi 4, SD card, running HA OS, few integrations taking quite some time to load). Which is pretty much if you are
    • a) working heavyly on your configuration and have
    • b) many systems integrated which all now NEED HA to be alive/online, otherwise the smart home turns into a pretty dumb home (wall switches not working, lights not being turned on by motion sensors, data not collecting, automations not running, notifictions not being sent, … - basically feels like there’s a power outage). Creating follow-up issues like
    • missed automations which should have run meanwhile etc.
    • “wrong” last-updated / last-changed states of entities - yes, I’ve restarted HA, but I didn’t turn on or off this light/switch/… (all entities are initialized and get a new state update in the recorder so basically they are “reset” to the HA restart timestamp)
    • …

So many things which could be avoided if we wouldn’t need to restart HA that often.

---

Bit of reality maths:
I’m using HA in production state roughly since december 2020. Using SQL - Home Assistant with a simple SELECT run_id AS "Anzahl HA-Starts" FROM "recorder_runs" ORDER BY "created" DESC LIMIT 1; statement I created a sensor.ha_starts_count sensor, currently showing:

So roughly 639 days since starting using HA and that many restarts (since few months I made a big step back and only update HA Core once a month cause there’s usually no need to update every 20yy.mm.XX release if XX does not contain an important fix - since then I quite often have uptimes of 14 to 21 days, not adding or configuring anything) makes a mean run time of amazing:

[petro]

@e-raser

And some things (like “command_line sensors”) create issues: once I reload command line sensors, they are reloaded - but at the same time a duplicate (shadow entity) is created, e. g. “sensor.thing_a” will have a “sensor.thing_a_2” brother, after another reload another sibling “sensor.thing_a_3” joyns the party and so on. The “fix” is: a full HA restart.

Use a unique_id. Otherwise the system doesn’t know it’s the same entity because it already exists when you reload. Providing a unique_id to your command line lets the system know “Hey this is that sensor, don’t make a new one”.

[e-raser]

Uff, thanks for reminding me of Command Line - Home Assistant ! Started using CL sensors before learning to read the existing docs first…

I’m going to over all my CL sensors now, let’s see what happens when reloading then without a HA Core restart.

I guess: finally “fixed” this little piece, so substract that minor pain point from the list and focus on the remaining ones (only examples…)

Update on this: for bad news see

WTH why do we need to restart HA core so much?

Update few days and HA restarts later @petro regarding WTH why do we need to restart HA core so much? - #20 by e-raser: Unfortunately when setting a unique_id for (command_line) sensors after using them already, it seems one ends up with a lost history from before that change. An entity now having a unique_id can not be mapped to the former one with the same name but without the unique_id. Probably “works as intended” but a big downside for me as I use some sensors for long term statistics - …