Urgent help, please, I think my Home Assistant is in a 'race' state - freezing and restarting

daneboom · March 11, 2024, 10:49am

I’m having lots of problems with icloud3 which seems to be causing my Pi to hang then I guess the watchdog catches it and reboots it.

Whilst the (wonderful) developer is investigating, I disabled the integration and now Home Assistant just seems to be in a restart loop, my log is growing a megabyte a minute and is filled with:

I DO have SSH access which is working and is quick. Anything I can do?! Thanks


2024-03-11 10:47:55.610 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.617 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.623 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.629 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.635 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.641 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.648 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.654 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.661 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None
AssertionError
2024-03-11 10:47:55.667 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 637, in async_trigger
    and not self._cond_func(variables)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/automation/__init__.py", line 1014, in if_action
    if check(hass, variables) is False:
       ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 179, in wrapper
    result = condition(hass, variables)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 787, in template_if
    return async_template(hass, value_template, variables)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/condition.py", line 767, in async_template
    info = value_template.async_render_to_info(variables, parse_result=False)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/template.py", line 687, in async_render_to_info
    assert self.hass and _render_info.get() is None

daneboom · March 11, 2024, 12:00pm

Managed to stop it (eventually) by clearing my automations.yaml. I think it was a template condition or trigger linked to icloud3 that was causing the race

bdraco · April 21, 2024, 10:12pm

The AssertionError should normally not be reachable, which indicates you have an integration doing non-threadsafe operations on asyncio objects which is corrupting the internal asyncio state.

When an integration modifies the event loop internals from another thread the behavior is undefined and will usually eventually lead to a crash.

If you put asyncio in debug mode, it will be able to detect some of these bad calls and maybe even block them. You can use the profiler service to do that:
https://www.home-assistant.io/integrations/profiler/#service-profilerset_asyncio_debug

If you can’t get far enough along you can also enable debugpy: in configuration.yaml instead (but its much slower)
https://www.home-assistant.io/integrations/debugpy/

Its likely the cloud3 integration needs to be audited for unsafe thread operations.