Nabu.Casa being offline brought Home Assistant OS to its knees

ricardoruiz · November 25, 2020, 5:59pm

I just started migrating from Smartthings + Webcore to Home Assistant OS on a NUC. Absolute noob here. The primary reason for this switch is because all my automation through Smartthings is tied to the cloud. I wanted HA to provide me more in-house direct control of my home automation system with less cloud dependency.

That said, I am trying out the Nabu Casa cloud service just for Alexa integration. I already setup my system with DuckDNS for remote access, so don’t need that. Here is my question/problem:

Nabu Casa seems to have gone down. Their login portal is giving me a 504. But here is the big problem, the lack of Nabu Casa seems to have brought Home Assistant OS to its knees. It will not complete the load of HA. Barely anything is running. I keep getting the: “Home Assistant is starting, not everything will be available until it is finished.”

Obviously, my big issue with this is that this is the kind of situation I’m trying to move away from. I would expect the Nabu Casa integration to fail and the system to just move on. I should be able to use the rest of my system. As it is, that integration absolutely impairs the stability of my system. I rather loose Alexa integration and keep everything else.

System Health:

System Health

Home Assistant Core Integration

version: 0.118.0
installation_type: Home Assistant OS
dev: false
hassio: true
docker: true
virtualenv: false
python_version: 3.8.6
os_name: Linux
os_version: 5.4.63
arch: x86_64
timezone: America/Chicago

Hass.io

host_os: HassOS 5.5
update_channel: stable
supervisor_version: 2020.11.0
docker_version: 19.03.12
disk_total: 468.8 GB
disk_used: 2.7 GB
healthy: true
supported: true
board: intel-nuc
supervisor_api: ok
version_api: ok
installed_addons: Visual Studio Code (2.8.2), File editor (5.2.0), Duck DNS (1.12.4), Terminal & SSH (8.9.1)

Lovelace

dashboards: 2
mode: auto-gen
resources: 0

Here is my home-assistant.log:

2020-11-25 11:34:21 WARNING (Recorder) [homeassistant.components.recorder] Ended unfinished session (id=40 from 2020-11-25 17:23:56.455984)
2020-11-25 11:34:31 WARNING (MainThread) [homeassistant.setup] Setup of cloud is taking over 10 seconds.
2020-11-25 11:34:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:35:21 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: cloud
2020-11-25 11:35:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:36:10 ERROR (MainThread) [hass_nabucasa.iot] Unexpected error
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/local/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/local/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/local/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/local/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
  File "/usr/local/lib/python3.8/ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "/usr/local/lib/python3.8/ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 254, in send
    urllib_response = conn.urlopen(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
    retries = retries.increment(
  File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 506, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/site-packages/urllib3/packages/six.py", line 735, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
    httplib_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 447, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 336, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: AWSHTTPSConnectionPool(host='cognito-idp.us-east-1.amazonaws.com', port=443): Read timed out. (read timeout=60)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/hass_nabucasa/iot_base.py", line 108, in connect
    await self._handle_connection()
  File "/usr/local/lib/python3.8/site-packages/hass_nabucasa/iot_base.py", line 147, in _handle_connection
    await self.cloud.auth.async_check_token()
  File "/usr/local/lib/python3.8/site-packages/hass_nabucasa/auth.py", line 172, in async_check_token
    await self._async_renew_access_token()
  File "/usr/local/lib/python3.8/site-packages/hass_nabucasa/auth.py", line 199, in _async_renew_access_token
    await self.cloud.run_executor(cognito.renew_access_token)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/pycognito/__init__.py", line 617, in renew_access_token
    refresh_response = self.client.initiate_auth(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 647, in _make_api_call
    http, parsed_response = self._make_request(
  File "/usr/local/lib/python3.8/site-packages/botocore/client.py", line 667, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 136, in _send_request
    while self._needs_retry(attempts, operation_model, request_dict,
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 228, in _needs_retry
    responses = self._event_emitter.emit(
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 183, in __call__
    if self._checker(attempts, response, caught_exception):
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 250, in __call__
    should_retry = self._should_retry(attempt_number, response,
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 277, in _should_retry
    return self._checker(attempt_number, response, caught_exception)
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 316, in __call__
    checker_response = checker(attempt_number, response,
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 222, in __call__
    return self._check_caught_exception(
  File "/usr/local/lib/python3.8/site-packages/botocore/retryhandler.py", line 359, in _check_caught_exception
    raise caught_exception
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 200, in _do_get_response
    http_response = self._send(request)
  File "/usr/local/lib/python3.8/site-packages/botocore/endpoint.py", line 244, in _send
    return self.http_session.send(request)
  File "/usr/local/lib/python3.8/site-packages/botocore/httpsession.py", line 289, in send
    raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://cognito-idp.us-east-1.amazonaws.com/"
2020-11-25 11:36:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:36:21 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: cloud
2020-11-25 11:36:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:37:09 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:37:21 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: cloud
2020-11-25 11:37:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:38:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:38:21 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: cloud
2020-11-25 11:38:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:39:19 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:39:21 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: cloud
2020-11-25 11:39:22 ERROR (MainThread) [homeassistant.setup] Setup of cloud is taking longer than 300 seconds. Startup will proceed without waiting any longer
2020-11-25 11:39:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:40:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:40:29 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:40:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:41:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:41:57 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:42:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:42:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:43:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:44:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:44:43 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:44:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:45:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:46:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:46:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:47:13 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:47:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:48:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:48:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:49:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:50:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:50:35 WARNING (MainThread) [hass_nabucasa.iot] Cannot connect because unable to refresh token: Gateway Timeout
2020-11-25 11:50:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:51:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:52:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:52:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:53:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:54:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:54:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:55:32 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:56:12 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud
2020-11-25 11:56:52 ERROR (MainThread) [hass_nabucasa.remote] Can't update remote details from Home Assistant cloud

franciszeks · November 25, 2020, 6:05pm

It’s a huge bummer! Having the same - cannot load HA properly with Nabu Casa down. Hope this dependency will be dropped in the next HA update.

Nickolaos · November 25, 2020, 6:11pm

I am not leveraging Nabu Casa for my deployment and I just rebooted my instance and it booted fine without incident.

As a test, have you thought of temporarily removing your credentials for Home Assistant Cloud and see if that allows you to restart your instance without incident?

I know it is not a permanent fix, but at least will allow you to continue leveraging local control until the situation is resolved.

ricardoruiz · November 25, 2020, 6:14pm

Well… wish I could, but the UI is not showing the cloud integration.

Probably because the integration hasn’t loaded completely yet. I am currently trying to figure out if there is a way to do it from command line.

Nickolaos · November 25, 2020, 6:18pm

Try disconnecting your Home Assistant instance (Your RPi from the Internet). I believe there is some built in error handling should the Internet not be available.

franciszeks · November 25, 2020, 6:20pm

Commenting out cloud in configuration.yaml should do the trick.

#cloud:

Yoinkz · November 25, 2020, 6:24pm

Seems to be same here…

Logger: hass_nabucasa.remote
Source: /usr/local/lib/python3.8/site-packages/hass_nabucasa/remote.py:136
First occurred: 19.15.47 (12 occurrences)
Last logged: 19.23.07

Can’t update remote details from Home Assistant cloud

And on my lovelace it just keeps saying Home Assistant starting… Some functions…

ricardoruiz · November 25, 2020, 6:29pm

Neither unplugging the ethernet cable nor commenting out the cloud entry in config did the trick. The integration is still trying to load. Might try using my router to block access to the internet to my NUC, maybe it reacts differently. Also, still trying to figure out how to disable an integration over command line.

AGW · November 25, 2020, 6:30pm