After 2023.8.1 HA fails to start with an error that it can't find some "integrations"?

whd · September 9, 2023, 9:56pm

I am running HA in a container and starting with HA version 2023.8.1 (and with every version afterwards) I unfortunately get an error message when starting HA that some integrations cannot be found and HA gets stuck and does not load afterwards.

Example error when trying to start with HA Version 2023.8.1:

2023-08-27 16:11:49.172 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [548237125696] Error handling message: Unknown error (unknown_error) Display Panel from 192.168.2.100 (Mozilla/5.0 (X11; CrOS aarch64 13597.84.0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.5672.95 Safari/537.36)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 417, in handle_get_services
    payload = await _async_get_all_descriptions_json(hass)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 398, in _async_get_all_descriptions_json
    descriptions = await async_get_all_descriptions(hass)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 612, in async_get_all_descriptions
    translations = await translation.async_get_translations(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 318, in async_get_translations
    cached = await cache.async_fetch(language, category, components)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 209, in async_fetch
    await self._async_load(language, components_to_load)
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 230, in _async_load
    raise int_or_exc
homeassistant.loader.IntegrationNotFound: Integration 'mennekes' not found.
2023-08-27 16:11:55.575 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [548184098240] Error handling message: Unknown error (unknown_error) Home Assistant Cast from 127.0.0.1 (Mozilla/5.0 (Fuchsia) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36 CrKey/1.56.500000)
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/decorators.py", line 26, in _handle_async_response
    await func(hass, connection, msg)
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 417, in handle_get_services
    payload = await _async_get_all_descriptions_json(hass)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 398, in _async_get_all_descriptions_json
    descriptions = await async_get_all_descriptions(hass)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 612, in async_get_all_descriptions
    translations = await translation.async_get_translations(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 318, in async_get_translations
    cached = await cache.async_fetch(language, category, components)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 209, in async_fetch
    await self._async_load(language, components_to_load)
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 230, in _async_load
    raise int_or_exc
homeassistant.loader.IntegrationNotFound: Integration 'mennekes' not found.

Example error when trying to start with HA Version 2023.9.1:

2023-09-09 23:06:56.040 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/pyscript/__init__.py", line 330, in hass_started
    await State.get_service_params()
  File "/config/custom_components/pyscript/state.py", line 80, in get_service_params
    all_services = await async_get_all_descriptions(cls.hass)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 612, in async_get_all_descriptions
    translations = await translation.async_get_translations(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 318, in async_get_translations
    cached = await cache.async_fetch(language, category, components)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 209, in async_fetch
    await self._async_load(language, components_to_load)
  File "/usr/src/homeassistant/homeassistant/helpers/translation.py", line 230, in _async_load
    raise int_or_exc
homeassistant.loader.IntegrationNotFound: Integration 'studer' not found.

Some odd observation, I also have the custom integration pyscript installed and actually in pyscript I do have a python module “mennekes” and a python module “studer”, and now both of these strings are flagged by HA as “Integration ‘xxx’ not found.” after 2023.8.1 with the translation, but this should really be completely independent and should not be triggering such an integration not found error?

I do not have a custom integration with either of those names and I also can’t find a file with “studer” or “mennekes” in its name (with find . -name “studer” or find . -name “mennekes”`) or even a file with the string "studer" or “mennekes” (with fgrep -r “studer”orfgrep -r “mennekes”```) in my HA/config directory (outside the pyscript directory)? But HA somewhere still thinks that integrations with the names “studer” or “mennekes” exist and with the new translated services in HA 2023.8.1 it tries to find them to apply the new translation services but then gets stuck and doesn’t load?

So unfortunately I am currently stuck with 2023.7.3 (where everything works just fine). Something must have changed in 2023.8.1 that leads to this error but I actually did not change anything on my side so could this be a bug introduced with the translation services starting with 2023.8.1?

See also discussion after my post here.

starob · September 10, 2023, 9:41am

Maybe it’s related to this?
https://github.com/custom-components/pyscript/issues/510

whd · September 10, 2023, 12:24pm

no, unfortunately this issue was unrelated, I am running pyscript (manifest.json: "version": "1.5.0") already since it was released (and before HA 2023.8.1 was released) and also, the bug you referred to turned out to not be a bug but a misconfiguration. @frenck, any chance that you could help me identify where that error is originated from after the 2023.8.1 release? Thanks!

123 · September 10, 2023, 12:46pm

Have you uninstalled pyscript to confirm it eliminates the error messages?

whd · September 10, 2023, 8:02pm

yes, I did, but it unfortunately that didn’t help. If I knew where HA actually retrieves the information from that such an integration exist, we could probably chade it down.

pedolsky · September 10, 2023, 8:37pm

Could this be a REST sensor? Studer-Innotec solar components integration - #5 by JeffersM

whd · September 10, 2023, 9:29pm

unfortunately not, no such REST sensors either?

whd · September 11, 2023, 9:46pm

\o/ after 5+weeks, I finally found and fixed the issue! It was caused by the pyscript custom integration. I had services that had @service decorators with a service_name DOMAIN.SERVICE (which is explicitly allowed by pyscript, see here), but starting with HA 2023.8.1 HA interprets the DOMAIN part in the service_name as an integration and is looking for a translation, that it can’t find and then it fails to start. Puh, that was a difficult one… I documented it in more detail in a pyscript bug here.

123 · September 11, 2023, 10:42pm

If it was caused by the pyscript integration, how was it possible for the error message to appear after you had uninstalled the pyscript integration?

The service decorator appears in files that Home Assistant wouldn’t normally access without the pyscript integration, does it not?

whd · September 12, 2023, 9:42am

I did not fully uninstall pyscript. But when you asked, I literally just rebooted after disabling the integration and that didn’t help. So to give you even more context, I started by changing the filenames of the pyscript modules that were called “mennekes.py” and “studer.py” to exclude that those were causing the issue (it was sort of the most obvious suspicion, that these pyscript module names would be interpreted as HA integrations). But since that didn’t resolve the issue, it was clear that the pyscript modules (or the module names) were not the issue. I then disabled the pyscript integration in HA (not fully uninstalled it), which actually created some other issues when booting up HA since I have quite a few dependencies to pyscript for the rest of HA so it didn’t help much (and that was exactly when you had asked). But I still figured it must have something to do with pyscript (where I do most of the energy management including managing my Studer inverters and managing the Mennekes wallbox among other things) and I therefore decided that completely uninstalling pyscript would not actually help to get to the bottom of it. So I continued with the Jupiter Debug Notebook for pyscript and there then realized that the mennekes and studer services I had exposed to HA via pyscript were treated similar to other integrations and then I finally realized that this error could actually be caused by these service descriptions. But obviously the fact that the DOMAIN in the service_name of the @service descriptor is identified as an integration wasn’t obvious, especially since the error appeared only with 2023.8.1 without me actually changing anything on mys side and these service descriptors were there for ages. Also such a service description is not like a classical HA integration and HA doesn’t actually read that information from a file like the mennekes.py or studer.py files (and as I verified, it is also not related to the filenames of the modules which is how you refer to the modules from within other pyscript code) but I believe what happens is that the pyscript integration reads the service descriptor and then registers the service with that descriptor (and that DOMAIN name) directly in the HASS Kernel which makes it available in some internal data structure to HA where the DOMAIN then apparently is interpreted as an integration and then HA looks for a translation for that “integration” and that fails.

I hope that helps explain it. I am glad that the issue is now identified and there is an easy workaround for HA versions >= 2023.8.1 (i.e. by just avoiding specifying a service_name with a DOMAIN.service in an @service descriptor in pyscriipt) and hopefully it will also eventually be fixed in the code so that it doesn’t cause the error in the first place. I am actually not sure if this is a pyscript bug injecting the information the wrong way into the HA data structure directly via the kernel or if HA is just misinterpreting the entry in that data structure as an integration but it shouldn’t, or if the way pyscript has allowed specifying DOMAIN.service in the service descriptor to then expose this to HA as a HA service was never meant to be done that way, but just happened to work until 2023.8.1 or if pyscript should provide a mandatory mechanism to provide a translation when specifying a DOMAN.service in a @service descriptor.

123 · September 12, 2023, 12:12pm

Yet when I asked if you uninstalled it, you said you did.

For future reference, when people take the time to try to help you, it’s important to answer simple questions accurately.

Q: Did you uninstall pyscript?
A: No, I just disabled it.

Otherwise they’re likely to ignore your requests for help in the future.

whd · September 12, 2023, 2:24pm

fully appreciate the feedback @123 and yes, I should have been clearer, mea culpa, I was in the middle of it and the answer was not concise. And I hope that if you look at the level of detail that I shared back with the community in this case, you hopefully also see that I do care a lot about the community. This is the most amazing project and it has the most amazing people working on it. This issue was a tricky one and I hope finding the solution and sharing it back actually helps others as well. I also bought you some coffee so hopefully you can forgive me this time

123 · September 12, 2023, 3:22pm

No problem. This problem had me stumped; glad you were able to find the cause of the error message.