Multiple devices with different integrations are becoming unavailable

bphillips921 · June 4, 2021, 7:52pm

I’ve been having an odd daily issue. At random times I will have a bunch of entities go unavailable. Some of the entities:

Battery sensor for my wall tablet via MQTT
Chromecast devices connected to WiFi
Light switches and plugs that run Tasmota
Lutron bridge connected via Wifi.

There doesn’t seem to be any single variable pointing me to the problem. Here are my logs. I noticed the issue at 10:51

A reboot solves the issue.

Any ideas what is causing this?

2021-06-04 10:50:05 WARNING (MainThread) [homeassistant.components.http.forwarded] A request from a reverse proxy was received from 172.30.33.7, but your HTTP integration is not set-up for reverse proxies; This request will be blocked in Home Assistant 2021.7 unless you configure your HTTP integration to allow this header
2021-06-04 10:50:05 ERROR (MainThread) [homeassistant.components.mobile_app.webhook] Refusing to update homeassistant's SM-G975U non-registered sensor: 6eba9733fea85ca2c43ff3a4f0d3dc3c75f69566f8718075d402555bd0f01db4_next_alarm
2021-06-04 10:51:51 WARNING (Thread-8) [pychromecast.socket_client] [Nursery speaker(192.168.1.23):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:51 WARNING (Thread-17) [pychromecast.socket_client] [Basement TV(192.168.1.244):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:51 WARNING (Thread-12) [pychromecast.socket_client] [Upstairs Speakers(192.168.1.25):32154] Heartbeat timeout, resetting connection
2021-06-04 10:51:52 WARNING (MainThread) [homeassistant.components.denonavr.media_player] Timeout connecting to Denon AVR receiver at host 192.168.1.179. Device is unavailable
2021-06-04 10:51:53 ERROR (SyncWorker_27) [homeassistant.components.life360.device_tracker] get_circles: CommError: HTTPSConnectionPool(host='api.life360.com', port=443): Max retries exceeded with url: /v3/circles.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xa2fe1f40>: Failed to establish a new connection: [Errno -3] Try again'))
2021-06-04 10:51:55 WARNING (Thread-15) [pychromecast.socket_client] [Living Room TV(192.168.1.242):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:55 WARNING (Thread-19) [pychromecast.socket_client] [All Speakers(192.168.1.20):32223] Heartbeat timeout, resetting connection
2021-06-04 10:51:55 WARNING (Thread-20) [pychromecast.socket_client] [Basement speaker(192.168.1.20):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-11) [pychromecast.socket_client] [Main Floor(192.168.1.24):32208] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-13) [pychromecast.socket_client] [Living room and kitchen(192.168.1.24):32201] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-16) [pychromecast.socket_client] [Whole house(192.168.1.24):32037] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-18) [pychromecast.socket_client] [Kitchen display(192.168.1.24):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-10) [pychromecast.socket_client] [Master Bedroom speaker(192.168.1.25):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-9) [pychromecast.socket_client] [Beyer's Room speaker(192.168.1.245):8009] Heartbeat timeout, resetting connection
2021-06-04 10:51:56 WARNING (Thread-14) [pychromecast.socket_client] [Living Room speaker(192.168.1.237):8009] Heartbeat timeout, resetting connection
2021-06-04 10:52:01 ERROR (Thread-13) [pychromecast.socket_client] [Living room and kitchen([::ffff:c0a8:118]):32201] Failed to connect to service ServiceInfo(type='host', data=('[::ffff:c0a8:118]', 32201)), retrying in 5.0s
2021-06-04 10:52:01 ERROR (SyncWorker_6) [homeassistant.components.life360.device_tracker] Suppressing further errors until OK: get_circles: CommError: HTTPSConnectionPool(host='api.life360.com', port=443): Max retries exceeded with url: /v3/circles.json (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0xa2d1ecd0>: Failed to establish a new connection: [Errno -3] Try again'))
2021-06-04 10:52:04 ERROR (MainThread) [homeassistant.components.brother] Error fetching brother data: No SNMP response received before timeout
2021-06-04 10:52:13 WARNING (MainThread) [hass_nabucasa.google_report_state] Unable to connect: Cannot connect to host remotestate.nabucasa.com:443 ssl:default [Try again]
2021-06-04 10:52:14 WARNING (MainThread) [pylutron_caseta.smartbridge] ping was not answered. closing connection.
2021-06-04 10:52:17 WARNING (MainThread) [homeassistant.components.esphome] Can't connect to ESPHome API for kass_noise_sensor (192.168.1.32): Timeout while connecting to ('192.168.1.32', 6053)
2021-06-04 10:52:21 ERROR (Thread-8) [pychromecast.socket_client] [Nursery speaker(192.168.1.23):8009] Failed to connect to service ServiceInfo(type='mdns', data='Google-Home-Mini-d76804d4808009462d1b1806a8e38a31._googlecast._tcp.local.'), retrying in 5.0s
2021-06-04 10:52:21 ERROR (Thread-17) [pychromecast.socket_client] [Basement TV(192.168.1.244):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.244', 8009)), retrying in 5.0s
2021-06-04 10:52:21 ERROR (Thread-12) [pychromecast.socket_client] [Upstairs Speakers(192.168.1.25):32154] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.25', 32154)), retrying in 5.0s
2021-06-04 10:52:25 ERROR (Thread-15) [pychromecast.socket_client] [Living Room TV(192.168.1.242):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.242', 8009)), retrying in 5.0s
2021-06-04 10:52:25 ERROR (SyncWorker_1) [homeassistant.components.life360.device_tracker] get_circles: OK again
2021-06-04 10:52:25 ERROR (Thread-19) [pychromecast.socket_client] [All Speakers(192.168.1.20):32223] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.20', 32223)), retrying in 5.0s
2021-06-04 10:52:25 ERROR (Thread-20) [pychromecast.socket_client] [Basement speaker(192.168.1.20):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.20', 8009)), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-11) [pychromecast.socket_client] [Main Floor(192.168.1.24):32208] Failed to connect to service ServiceInfo(type='mdns', data='Google-Cast-Group-a69b7a49f2674462976e0bb3de34e3aa._googlecast._tcp.local.'), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-16) [pychromecast.socket_client] [Whole house(192.168.1.24):32037] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.24', 32037)), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-18) [pychromecast.socket_client] [Kitchen display(192.168.1.24):8009] Failed to connect to service ServiceInfo(type='mdns', data='Google-Nest-Hub-Max-6090d21580b30f275c7ae1d2b43c6789._googlecast._tcp.local.'), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-10) [pychromecast.socket_client] [Master Bedroom speaker(192.168.1.25):8009] Failed to connect to service ServiceInfo(type='mdns', data='Google-Home-Mini-bb4bcfd9b5561ec9c57efa31f9036c24._googlecast._tcp.local.'), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-9) [pychromecast.socket_client] [Beyer's Room speaker(192.168.1.245):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.245', 8009)), retrying in 5.0s
2021-06-04 10:52:26 ERROR (Thread-14) [pychromecast.socket_client] [Living Room speaker(192.168.1.237):8009] Failed to connect to service ServiceInfo(type='host', data=('192.168.1.237', 8009)), retrying in 5.0s
2021-06-04 10:52:30 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:53:01 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:53:34 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:53:56 ERROR (MainThread) [snitun.multiplexer.core] Ping fails, no response from peer
2021-06-04 10:54:08 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:54:41 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:55:14 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:55:48 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:56:21 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:56:54 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:57:27 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:58:01 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:58:34 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:59:07 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 10:59:41 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 11:00:14 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 11:00:47 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 11:01:20 WARNING (MainThread) [haffmpeg.core] FFmpeg isn't running!
2021-06-04 11:01:23 WARNING (MainThread) [pylutron_caseta.smartbridge] LEAP session ended. Reconnecting...
2021-06-04 11:01:26 WARNING (MainThread) [pylutron_caseta.smartbridge] Reconnecting...
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pylutron_caseta/smartbridge.py", line 421, in _monitor_once
    self._leap = await self._connect()
  File "/usr/local/lib/python3.8/site-packages/pylutron_caseta/smartbridge.py", line 112, in _connect
    res = await open_connection(
  File "/usr/local/lib/python3.8/site-packages/pylutron_caseta/leap.py", line 178, in open_connection
    reader, writer = await asyncio.open_connection(host, port, limit=limit, **kwds)
  File "/usr/local/lib/python3.8/asyncio/streams.py", line 52, in open_connection
    transport, _ = await loop.create_connection(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1025, in create_connection
    raise exceptions[0]
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 1010, in create_connection
    sock = await self._connect_sock(
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 924, in _connect_sock
    await self.sock_connect(sock, address)
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 496, in sock_connect
    return await fut
  File "/usr/local/lib/python3.8/asyncio/selector_events.py", line 528, in _sock_connect_cb
    raise OSError(err, f'Connect call failed {address}')
OSError: [Errno 113] Connect call failed ('192.168.1.134', 8081)

duceduc · June 4, 2021, 8:05pm

HTTP integration is not set-up for reverse proxies

I have the same error as you and this post removed my warning.

bphillips921 · June 6, 2021, 3:01am

Thanks for the tip. It looks like this just became an issue with 2021.6.1. I was having this issue prior to that upgrade. I also tried going back to 2021.3.1 but it didn’t fix it. I’m moving into my new house in 2 months and I’m starting from scratch with HA, so I just want to patch this one to get it working for two more months.

bphillips921 · June 7, 2021, 4:46am

My issue must have been something with my WiFi network. I restarted it and haven’t had the issue since.

dib0 · January 24, 2023, 8:47am

I seem to have the same issue since updating to 2023.1.1. Several (not all) integrations top working after around 2.5 hours.

I have Home Assistant running in a docker on my NAS (behind a Nginx reverse proxy). It has been working like this for over a year (ran hass on a raspberry pi before that). Not sure what is happening. If I reboot the container all integrations work. But after 2.5 hours some custom integrations go to the status “unavailable”, it also happens with one not-custom integration. I tried going back to the last 2022.12.x version, but that didn’t solve the issue.

I get messages like these:

Logger: surepy.client
Source: components/surepetcare/__init__.py:190
First occurred: January 23, 2023 at 17:11:38 (238 occurrences)
Last logged: 08:59:38

 Logger: metno
Source: components/met/__init__.py:186
First occurred: January 23, 2023 at 17:31:16 (16 occurrences)
Last logged: 08:08:46
Access to https://aa010h7butvih86i1.api.met.no/weatherapi/locationforecast/2.0/complete returned error 'TimeoutError'

Logger: snitun.multiplexer.core
Source: runner.py:120
First occurred: 07:16:08 (1 occurrences)
Last logged: 07:16:08

Ping fails, no response from peer

Logger: homeassistant.components.homewizard.coordinator
Source: helpers/update_coordinator.py:168
Integration: HomeWizard Energy (documentation, issues)
First occurred: January 23, 2023 at 17:09:30 (1 occurrences)
Last logged: January 23, 2023 at 17:09:30

Error fetching homewizard data: Timeout occurred while connecting to the HomeWizard Energy device

In de detailes logs I find the following after 2-3 hours:

2023-01-23 16:09:30.348 ERROR (MainThread) [homeassistant.components.homewizard.coordinator] Error fetching homewizard data: Timeout occurred while connecting to the HomeWizard Energy device
2023-01-23 16:10:14.107 ERROR (MainThread) [homeassistant.components.sonarr] Error fetching sonarr data: Request timeout for 'http://x.x.x.x:yy/api/v3/calendar'
2023-01-23 16:10:15.337 ERROR (MainThread) [homeassistant.components.nextdns] Timeout fetching nextdns data

Just restarting Home Assistant from the browser is enough to let it all work again. No need to restart the Docker image.

Any ideas?

dib0 · January 24, 2023, 1:57pm

As additional information:
After a restart is takes exactly 2.5 hours for these integrations to become unavailable. Not sure why that is.

Troon · January 24, 2023, 2:40pm

DHCP or some other router change? Your errors all seem to relate to network connections. Is this the standard homeassistant Docker image running in host network mode? Is it behind the reverse proxy even on your LAN?

dib0 · January 24, 2023, 2:50pm

No DHCP or router changes for a few years at the moment. I run the standard homeassistant docker image. It is running in bridged mode (If I recall correctly) Has been working fine for at least a year now. It is not behind a reverse proxy, I use personal .ui.nabu.casa address to access my home assistant installation.

Like I said: After a restart of home assistant (also a restart of the docker image) it all works. After 2.5 hours some (about 4) integrations stop working and show as unavailable. But others (including MQTT) keep working. Also home asistant seems to be able to reach internet, since the SpeedTest integration keeps adding new speeds.
Pings to mobile phones (to check if someone is home) also work.

I use watchtower to update my images every week (on friday).

Troon · January 24, 2023, 3:08pm

That aside, and shooting in the dark here, have you tried turning off your NextDNS service? Also, is the Sonarr server on the same NAS?

dib0 · January 24, 2023, 4:04pm

Sorry for that. I have been running home assistant behind a reverse proxy for years. But since I migrated to docker I stopped using Nginx for home assistant (still use it for around 40 other containers). Lets say that at my age these things get fuzzy…

Sonarr is also a docker container on the same NAS. Sonarr is in host network mode, where home assistant (like most of my docker containers) are bridged (is several networks).

So I’m also in the dark… I just disabled the nextDNS service. I’ve set one of the integrations to provide debug logging.

Any ideas are welcome. If I have more information I’ll post that.

dib0 · January 24, 2023, 4:10pm

I just saw in the state history of several integrations that the issues started after Watchtower updated the docker container to 2023.1.6 last Friday night. Before that it was working fine.

I didn’t see any related points in de release notes. Last Sunday I tried to go back to older versions up to 2022.12.6 and they all showed the same problem.

dib0 · January 25, 2023, 5:23am

Thank you for your thoughts. I’ve found the issue. It was a custom component called eufy_security.

I just went though all the components with issues and disabled them one by one. At last this one seems to be affecting the others. Home assistant worked well overnight with eufy_security disabled.

Thanks again!