Home Assistant losing internet connection first, slowly starts failing until crash 1.5h later

ilovezigbee · February 14, 2022, 4:15pm

This one has been troubling me for months now and I can’t figure it out. Every since I set up Home Assistant at my old folk’s home this has been happening.

TL;DR: Crashes happen predominantly during the weekend, almost always overnight. First the internet connection drops (but local connections stay online), 10 minutes later Zigbee devices go offline one by one, internal sensors and Shelly readings go last. No sign of life after that until power cycling the Pi.

Raspberry Pi 4B
core-2022.2.0
supervisor-2022.01.1
Home Assistant OS 7.2

network_status
Several binary sensors monitor the network status of the Pi.

Internet access is lost at 23:28 on Feb13 and restored upon power cycling the Pi at 10:04 on Feb14. Note that the modem and router ping are fine, meaning the network stayed up.

Naturally, HASS was not reachable from outside of the network.

If the internet connection would actually be the problem, the Zigbee network (ZHA) should have kept functioning just fine. But it didn’t.

Several temperature sensors stopped functioning at different times until the PI was power cycled. Offline times:

Kantoor: 23:40
Gastenruimte: 00:52
Badkamer: 00:48
Werkhok: 00:37

The humidity values (provided by the same sensors) stopped at different times. The last one was at 00:58.

The among the last sensors to cut out were:

Memory, CPU%, and CPU temperature: 1:05 (all values were fine)
Shelly 1PM energy data: 1:05

Last 10 minutes of Log

022-02-14 01:54:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:54:40 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:55:40 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:55:50 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 01:55:50 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 01:56:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:57:05 WARNING (MainThread) [dsmr_parser.clients.protocol] Invalid telegram. The CRC checksum '16388' does not match the expected '5672'
2022-02-14 01:57:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:57:55 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 01:57:55 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 01:58:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:58:17 ERROR (MainThread) [vw_connection] Login failed for VW account, Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
2022-02-14 01:58:17 ERROR (MainThread) [vw_connection] Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/vw_connection.py", line 157, in _login
    req = await self._session.get(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
2022-02-14 01:58:17 WARNING (MainThread) [vw_connection] Login for VW account failed!
2022-02-14 01:58:17 WARNING (MainThread) [custom_components.volkswagencarnet] Could not query update from volkswagen WeConnect
2022-02-14 01:59:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 01:59:40 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 02:00:00 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:00:00 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:00:40 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 02:00:55 ERROR (Thread-4) [root] Uncaught thread exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn
    conn = connection.create_connection(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/connection.py", line 72, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 386, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 358, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn
    raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x7f78f04b50>: Failed to establish a new connection: [Errno -3] Try again

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 440, in send
    resp = conn.urlopen(
  File "/usr/local/lib/python3.9/site-packages/urllib3/connectionpool.py", line 785, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='openapi.tuyaeu.com', port=443): Max retries exceeded with url: /v1.0/token/d4d4346456c36b60cd25f2ed6719955e (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f78f04b50>: Failed to establish a new connection: [Errno -3] Try again'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openmq.py", line 158, in run
    self.__run_mqtt()
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openmq.py", line 164, in __run_mqtt
    mq_config = self._get_mqtt_config()
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openmq.py", line 62, in _get_mqtt_config
    response = self.api.post(
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openapi.py", line 316, in post
    return self.__request("POST", path, None, body)
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openapi.py", line 238, in __request
    self.__refresh_access_token_if_need(path)
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openapi.py", line 164, in __refresh_access_token_if_need
    response = self.get(
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openapi.py", line 302, in get
    return self.__request("GET", path, params, None)
  File "/usr/local/lib/python3.9/site-packages/tuya_iot/openapi.py", line 266, in __request
    response = self.session.request(
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 529, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/sessions.py", line 645, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/requests/adapters.py", line 519, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='openapi.tuyaeu.com', port=443): Max retries exceeded with url: /v1.0/token/d4d4346456c36b60cd25f2ed6719955e (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f78f04b50>: Failed to establish a new connection: [Errno -3] Try again'))
2022-02-14 02:01:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 02:02:05 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:02:05 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:02:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 02:03:22 ERROR (MainThread) [vw_connection] Login failed for VW account, Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
2022-02-14 02:03:22 ERROR (MainThread) [vw_connection] Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1154, in _create_direct_connection
    hosts = await asyncio.shield(host_resolved)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 880, in _resolve_host
    addrs = await self._resolver.resolve(host, port, family=self._family)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/resolver.py", line 33, in resolve
    infos = await self._loop.getaddrinfo(
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 856, in getaddrinfo
    return await self.run_in_executor(
  File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.9/socket.py", line 954, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Try again

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/vw_connection.py", line 157, in _login
    req = await self._session.get(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/client.py", line 535, in _request
    conn = await self._connector.connect(
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 542, in connect
    proto = await self._create_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 907, in _create_connection
    _, proto = await self._create_direct_connection(req, traces, timeout)
  File "/usr/local/lib/python3.9/site-packages/aiohttp/connector.py", line 1166, in _create_direct_connection
    raise ClientConnectorError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host identity.vwgroup.io:443 ssl:default [Try again]
2022-02-14 02:03:22 WARNING (MainThread) [vw_connection] Login for VW account failed!
2022-02-14 02:03:22 WARNING (MainThread) [custom_components.volkswagencarnet] Could not query update from volkswagen WeConnect
2022-02-14 02:03:27 ERROR (SyncWorker_7) [stookalert.rivm_stookalert] Error getting Stookalert data
2022-02-14 02:03:28 ERROR (MainThread) [custom_components.hacs] Request exception for 'https://api.github.com/rate_limit' with - Cannot connect to host api.github.com:443 ssl:default [Try again]
Traceback (most recent call last):
  File "/config/custom_components/hacs/base.py", line 428, in async_can_update
    response = await self.async_github_api_method(self.githubapi.rate_limit)
  File "/config/custom_components/hacs/base.py", line 479, in async_github_api_method
    raise HacsException(_exception)
custom_components.hacs.exceptions.HacsException: Request exception for 'https://api.github.com/rate_limit' with - Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:03:38 INFO (MainThread) [custom_components.hacs] Loading known repositories
2022-02-14 02:03:43 ERROR (MainThread) [homeassistant] Error doing job: Task exception was never retrieved
Traceback (most recent call last):
  File "/config/custom_components/hacs/base.py", line 712, in recurring_tasks_all
    await self.async_load_default_repositories()
  File "/config/custom_components/hacs/base.py", line 724, in async_load_default_repositories
    for item in await self.async_github_get_hacs_default_file(HacsCategory.REMOVED):
  File "/config/custom_components/hacs/base.py", line 442, in async_github_get_hacs_default_file
    response = await self.async_github_api_method(
  File "/config/custom_components/hacs/base.py", line 479, in async_github_api_method
    raise HacsException(_exception)
custom_components.hacs.exceptions.HacsException: Request exception for 'https://api.github.com/repos/hacs/default/contents/removed' with - Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:03:49 WARNING (MainThread) [dsmr_parser.clients.protocol] Invalid telegram. The CRC checksum '10167' does not match the expected '17230'
2022-02-14 02:04:04 ERROR (MainThread) [custom_components.hacs] <Theme bbbenji/synthwave-hass> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:04 ERROR (MainThread) [custom_components.hacs] <Theme 3ative/3ative-blue-theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:04 ERROR (MainThread) [custom_components.hacs] <Theme basnijholt/lovelace-ios-dark-mode-theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:04 ERROR (MainThread) [custom_components.hacs] <Theme basnijholt/lovelace-ios-light-mode-theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:04 ERROR (MainThread) [custom_components.hacs] <Theme awolkers/home-assistant-themes> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:04:10 WARNING (MainThread) [homeassistant.components.buienradar.util] Unable to retrieve json data from Buienradar(Msg: Cannot connect to host data.buienradar.nl:443 ssl:default [Try again], status: None,)
2022-02-14 02:04:10 ERROR (MainThread) [custom_components.hacs] <Theme chaptergy/noctis-grey> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 ERROR (MainThread) [custom_components.hacs] <Theme aFFekopp/dark_teal> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 ERROR (MainThread) [custom_components.hacs] <Integration hacs/integration> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 ERROR (MainThread) [custom_components.hacs] <Theme 78wesley/Home-Assistant-Darkish-Theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 ERROR (MainThread) [custom_components.hacs] <Theme arsaboo/oxford_blue_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:10 WARNING (MainThread) [homeassistant.helpers.entity] Update of sensor.tz3000_8nkb7mof_ts0121_electrical_measurement is taking over 10 seconds
2022-02-14 02:04:15 ERROR (MainThread) [homeassistant.components.forecast_solar] Unexpected error fetching forecast_solar data: Error while resolving Forecast.Solar API address
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 72, in _request
    result = await dns.query("api.forecast.solar", "A")
aiodns.error.DNSError: (11, 'Could not contact DNS servers')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 138, in estimate
    data = await self._request(
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 74, in _request
    raise ForecastSolarConnectionError(
forecast_solar.exceptions.ForecastSolarConnectionError: Error while resolving Forecast.Solar API address
2022-02-14 02:04:30 ERROR (MainThread) [homeassistant.components.forecast_solar] Unexpected error fetching forecast_solar data: Error while resolving Forecast.Solar API address
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 72, in _request
    result = await dns.query("api.forecast.solar", "A")
aiodns.error.DNSError: (11, 'Could not contact DNS servers')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 138, in estimate
    data = await self._request(
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 74, in _request
    raise ForecastSolarConnectionError(
forecast_solar.exceptions.ForecastSolarConnectionError: Error while resolving Forecast.Solar API address
2022-02-14 02:04:32 ERROR (MainThread) [custom_components.hacs] <Theme basnijholt/lovelace-ios-themes> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:32 ERROR (MainThread) [custom_components.hacs] <Theme einschmidt/github_dark_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:32 ERROR (MainThread) [custom_components.hacs] <Theme am80l/sundown> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme aFFekopp/noctis> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme DickSwart/swart_ninja_dark_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme hekm77/reeder_dark_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme Banditen01/vintage_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme estiens/sweet_pink_hass_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:37 ERROR (MainThread) [custom_components.hacs] <Theme einschmidt/github_light_theme> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:42 ERROR (MainThread) [custom_components.hacs] <Theme frenck/home-assistant-theme-outline> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:58 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/aqua-fiesta> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:58 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/amoled> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:04:58 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/blackened> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/grey-night> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/blue-night> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/dark-orange> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/christmas> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/halloween> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:03 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/dark-mint> Cannot connect to host api.github.com:443 ssl:default [Try again]
2022-02-14 02:05:10 ERROR (MainThread) [homeassistant.components.forecast_solar] Unexpected error fetching forecast_solar data: Error while resolving Forecast.Solar API address
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 72, in _request
    result = await dns.query("api.forecast.solar", "A")
aiodns.error.DNSError: (11, 'Could not contact DNS servers')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 187, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 147, in _async_update_data
    return await self.update_method()
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 138, in estimate
    data = await self._request(
  File "/usr/local/lib/python3.9/site-packages/forecast_solar/__init__.py", line 74, in _request
    raise ForecastSolarConnectionError(
forecast_solar.exceptions.ForecastSolarConnectionError: Error while resolving Forecast.Solar API address
2022-02-14 02:05:14 ERROR (MainThread) [custom_components.hacs] <Theme home-assistant-community-themes/material-dark-green> Cannot connect to host api.github.com:443 ssl:default [Try again]

Last 1.5h of the log (link needed as it’s too long for a post)

When reading the log, note:

I am in a different time zone. 02:05 in the log is actually 01:05, which is the time of the crash.
I think you can ignore the vw_connection related log entries.

The last line of the log is visualised by VScode as NULNULNULNUL

Any idea what might be causing this issue? Any advice on how I can figure out what’s causing the issue?

gmlupatelli · February 14, 2022, 4:45pm

Hi @ilovezigbee. I had some problems with USB devices connected to the PI stopping working, and it was a power supply issue. It wasn’t providing the PI with enough power and would trigger some strange behavior.

Give your description of the problem it could also be a Power Supply issue caused by grip power fluctuations that the power supply can’t handle or even a PI hardware problem.

Try switching your power supply with theirs and see if you still have problems. The next troubleshooting step would be switching the PI’s. (For both cases I am assuming you are also running HA in a Raspberry PI).

Another thing you may try, if you are using a UPS (Uninterrupted Power Supply), is removing it, or if you are not using one, try adding one.

ilovezigbee · February 14, 2022, 5:19pm

@gmlupatelli I am using the official EU Pi 4 Power supply and no UPS. The only USB devices attached are an extension cord with a CC2531 and an SSD. Do you still think power supply issues could be the issue?

gmlupatelli · February 15, 2022, 2:50am

By the log file, it doesn’t seem to be a software issue, probably you can simulate the same errors if you manually disconnect the Zigbee dongle.

It is an intermittent problem that affects your PI peripherals, and it usually happens at night, so my hypothesis is that the power grid has some voltage and frequency instability at that time and it is impacting the PI due to some characteristics of your power adapter.

It could also be a processor or memory problem, but the power supply is an easier and cheaper thing to test. You may also try reducing the PI power load, removing the external SSD, and running HA from a memory card, but moving HA from the SSD to a memory card is a bigger hassle than just replacing a power supply.

packetinspector · February 28, 2022, 4:47pm

I’ve been having similar issues. I start getting timeouts to sensors, media players, etc…HA slows down a little bit too. Only fix is to restart, happens almost every 24 hours.

System is rpi4, HA running in docker 2022.2.9. All software OS, docker, etc… is up to date. Been happening since 2022.10.x

I’ve tried two different power supplies, 3 SD cards, and an SSD drive. Nothing has worked.

What’s odd is if I get bash prompt in the container I can curl or run a python script to anything that fails in HA and they work without any delay. Nor do I have any problems outside docker.

HA only complains about timeouts, haven’t seen any logs pointing to an issue even with debug.

I’ve been suspicious that something is amiss with aiohttp and its client pool, but I have no way to debug that.

ilovezigbee · February 28, 2022, 5:50pm

I think my log file also hinted at issues with aiohttp.

In the meantime I’ve given up and bought an Intel NUC Kit NUC6CAYH and a bunch of Zigbee routers so that I could move the physical Home Assistant server to my main router.

My thinking was that since I didn’t have a spare SSD adapter, Pi, or PSU lying around, buying a second hand NUC was the safest option. Otherwise I might have ended up realising after buying a bunch of hardware parts that it was a software problem after all.

Previously it was connected to a Asus AImesh node (wired backhaul). Now it’s connected to the main router. But other than that it’s running HASS OS with exactly the same configuration. Since it likely wasn’t a networking issue, reporting back to you whether it’s stable now might still be valuable. I’ll report back in a few weeks if the problem is gone (if I remember).

packetinspector · February 28, 2022, 8:57pm

Been thinking of the same approach, found some with mem/SSD already installed.

The only issue on my rpi is Home Assistant, I’m just worried that I move the problem from one hardware device to another.

ilovezigbee · March 11, 2022, 7:30pm

Update: (@packetinspector )

Problem is totally fixed. I can’t be sure that my router wasn’t the issue, but I haven’t had a single crash yet. So compared to before, it’s safe to conclude that the issue is fixed. Plus, I’m happy with the improved performance.

packetinspector · March 11, 2022, 7:51pm

I took the hardware plunge too, but my fear came true and carried the problem with me. I think I found the issue with the Kodi integration…crossing fingers…

ilovezigbee · March 11, 2022, 8:20pm

That’s too bad to hear. Hopefully I’ll see your solution appear here.

If your new hardware was second hand you can always sell it and accept the shipping costs as loss. Although the shorter boot and loading times definitely make it worth keeping in my opinion.

pmcenaney · January 16, 2024, 11:40pm

Did you ever find a solution to your issue? It sounds similar to mine.

For the past few months or so I’ve noticed that every once in a while all of my sensors, addons, etc were unavailable and after trying different approaches the only way for me to get them back was to run ‘ha core rebuild’ from the CLI. Restarting HA or the host machine (I have HA OS running in Proxmox) had no affect.

I also have the same issue any time I manually restart HA (after an update, etc), and it seems the culprit is Supervisor isn’t loading correctly; although restarting Supervisor doesn’t fix it either.

I hadn’t had an issue for a few days until today, and then I remembered that my internet connection dropped briefly this afternoon, so I’m curious if that has something to do with it as well…

ilovezigbee · January 17, 2024, 10:52am

This sounds quite similar to the problems I was experiencing.

While your use of a hypervisor complicates things a bit, I’m still curious what your network infrastructure looks like.

Is your log file similar to what mine looked like?

It is now solved, but I did all of the following:

Replace Asus network infrastructure with a TP-link er605 plus and Omdat switches and APs (I would be hesitant to recommend it though, my UniFi experience is better)
Purchase a voltage regulator
Replace the Pi with a NUC
Install additional fans in the cupboard where it’s installed
Purchase USB hub with external power supply

pmcenaney · January 17, 2024, 1:29pm

I’ll have to wait until I get home to check my logs, but as far as network infrastructure I’m running Unifi Controller as an HA Addon, have a USG3, Unifi AC-LR, Proxmox on a custom built server, and all my network equipment (including HA) is connected to a UPS.