InfluxDB sensor failing after reboot

truglodite · December 31, 2019, 8:27pm

I’m running the latest hassos+hassio on a pi3 w/ ssd. I’ve been using influxdb without issues for a little while. Recently I added a database file size sensor using the influxdb platform with the following config:

  - platform: influxdb
    scan_interval: 1200
    host: 192.168.22.6
    port: 8086
    username: !secret influxdb_username2
    password: !secret influxdb_password2
    queries:
      - name: InfluxDB Size
        unit_of_measurement: MB
        value_template: '{{ (value | float / 1024 /1024) | round(1) }}'
        group_function: mean
        measurement: '"monitor"."shard"'
        database: _internal
        where: 'time > now() - 1m AND "database"=''homeassistant'''
        field: diskBytes

After a reboot my logs show this error first:

2019-12-31 11:09:28 WARNING (SyncWorker_10) [homeassistant.components.influxdb] Database host is not accessible due to 'HTTPConnectionPool(host='a0d7b954-influxdb', port=8086): Max retries exceeded with url: /write?db=homeassistant (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x72380690>: Failed to establish a new connection: [Errno 111] Connection refused'))', please check your entries in the configuration file (host, port, etc.) and verify that the database exists and is READ/WRITE. Retrying again in 60 seconds.

…and (I believe) more related errors:

2019-12-31 11:09:44 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up platform influxdb
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 157, in _new_conn
    (self._dns_host, self.port), self.timeout, **extra_kw
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 84, in create_connection
    raise err
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 74, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 672, in urlopen
    chunked=chunked,
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 387, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/lib/python3.7/http/client.py", line 1252, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1298, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1247, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.7/http/client.py", line 966, in send
    self.connect()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 184, in connect
    conn = self._new_conn()
  File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn
    self, "Failed to establish a new connection: %s" % e
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x714c5f10>: Failed to establish a new connection: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 720, in urlopen
    method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
  File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 436, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='192.168.22.6', port=8086): Max retries exceeded with url: /query?q=SHOW+SERIES+LIMIT+1%3B&db=_internal (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x714c5f10>: Failed to establish a new connection: [Errno 111] Connection refused'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 150, in _async_setup_platform
    await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT)
  File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for
    return fut.result()
  File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/src/homeassistant/homeassistant/components/influxdb/sensor.py", line 85, in setup_platform
    sensor = InfluxSensor(hass, influx_conf, query)
  File "/usr/src/homeassistant/homeassistant/components/influxdb/sensor.py", line 124, in __init__
    influx.query("SHOW SERIES LIMIT 1;")
  File "/usr/local/lib/python3.7/site-packages/influxdb/client.py", line 450, in query
    expected_response_code=expected_response_code
  File "/usr/local/lib/python3.7/site-packages/influxdb/client.py", line 283, in request
    timeout=self._timeout
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='192.168.22.6', port=8086): Max retries exceeded with url: /query?q=SHOW+SERIES+LIMIT+1%3B&db=_internal (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x714c5f10>: Failed to establish a new connection: [Errno 111] Connection refused'))

…and the sensor remains “unavailable” until I restart ha. After restarting ha, the sensor works as expected. This is of course a problem anytime I have to reboot. I read that the actual influxdb integration used to have similar problems with the db not being available when ha was starting up; there was some patches that have since fixed that issue. I wonder if this issue with the influxdb sensor is related? I’m guessing it’s not misconfiguration since all works fine after restart, but not sure if it’s a bug in ha, the add-on, or what. So I figured I should post here before chiming the discord add-on support. Anyone have an idea how to go about fixing this?

[edit: On an likely unrelated side note, it seems this influxdb sensor may be excessively resource intensive, no? My load averages without the sensor are usually 0.1-0.2ish, and with the influxdb sensor running it stays around 0.5ish. Is this expected, or is there something I can maybe optimize? I did slow down the scan_interval since I don’t need real time updates really, and that has helped a little, but it still runs ~0.5 even when it isn’t querying.

sorry for the TLDR…]

truglodite · January 1, 2020, 9:22pm

Well I guess disregard my above post… for some automagic reason the sensor is now working as expected after reboots. Now at first the sensor just reads 0MB instead of unavailable, because the the db is not available on the first query after reboot. After 20min the sensor refreshes and shows the proper value (mine is set to refresh every 20min). So whatever fixed it I’m not sure, but perhaps something properly configured itself after that first failed reboot that led me to post.

truglodite · January 11, 2020, 6:22pm

^bump^

I don’t know what I drank before writing my last post, but I must have just did an HA restart, not a hassos reboot. I still am not able to get the influxdb sensor working after reboot. In following up on this, I found some other info on this issue:

github.com/home-assistant/core

InfluxDB sensor does not recover if InfluxDB was unreachable during HA startup

opened 12:27PM - 07 Sep 19 UTC

closed 09:27PM - 13 Dec 19 UTC

schwma

integration: influxdb stale

**Home Assistant release with the issue:**  0.98.4 **Last working Home Assistant release (if known):** **Operating environment (Hass.io/Docker/Windows/etc.):**  Docker **Component/platform:**  InfluxDB **Description of problem:** Currently, if the InfluxDB server is unreachable while Home Assistant is starting up, the sensor badges do not show up in my auto-gen Lovelace UI. After failing to reach InfluxDB during startup, Home Assistant will not recover the InfluxDB sensor even if the InfluxDB server is reachable again. I personally would expect Home Assistant to periodically retry to reconnect to the InfluxDB server. This issue is similar to #18094, but it concerns polling sensor data from InfluxDB rather than persisting data. Might be the same issue as #25038, but could be different since connection can't be established at startup. **Problem-relevant `configuration.yaml` entries and (fill out even if it seems unimportant):** ```yaml influxdb: sensor: - platform: influxdb host: myhost queries: - name: Temperature unit_of_measurement: °C value_template: '{{ value | round(1) }}' group_function: last where: '"location" = ''inside''' measurement: '"weather_data"' field: temperature database: weatherstation ``` **Traceback (if applicable):** ``` 2019-09-07 14:13:35 ERROR (MainThread) [homeassistant.components.sensor] Error while setting up platform influxdb Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 160, in _new_conn (self._dns_host, self.port), self.timeout, **extra_kw) File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection raise err File "/usr/local/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection sock.connect(sa) ConnectionRefusedError: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 603, in urlopen chunked=chunked) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 355, in _make_request conn.request(method, url, **httplib_request_kw) File "/usr/local/lib/python3.7/http/client.py", line 1244, in request self._send_request(method, url, body, headers, encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1290, in _send_request self.endheaders(body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1239, in endheaders self._send_output(message_body, encode_chunked=encode_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1026, in _send_output self.send(msg) File "/usr/local/lib/python3.7/http/client.py", line 966, in send self.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 183, in connect conn = self._new_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 169, in _new_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7ffaf5f9b610>: Failed to establish a new connection: [Errno 111] Connection refused During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send timeout=timeout File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 641, in urlopen _stacktrace=sys.exc_info()[2]) File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment raise MaxRetryError(_pool, url, error or ResponseError(cause)) urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='myhost', port=8086): Max retries exceeded with url: /query?q=SHOW+SERIES+LIMIT+1%3B&db=weatherstation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffaf5f9b610>: Failed to establish a new connection: [Errno 111] Connection refused')) During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/src/homeassistant/homeassistant/helpers/entity_platform.py", line 150, in _async_setup_platform await asyncio.wait_for(asyncio.shield(task), SLOW_SETUP_MAX_WAIT) File "/usr/local/lib/python3.7/asyncio/tasks.py", line 442, in wait_for return fut.result() File "/usr/local/lib/python3.7/concurrent/futures/thread.py", line 57, in run result = self.fn(*self.args, **self.kwargs) File "/usr/src/homeassistant/homeassistant/components/influxdb/sensor.py", line 85, in setup_platform sensor = InfluxSensor(hass, influx_conf, query) File "/usr/src/homeassistant/homeassistant/components/influxdb/sensor.py", line 124, in __init__ influx.query("SHOW SERIES LIMIT 1;") File "/usr/local/lib/python3.7/site-packages/influxdb/client.py", line 416, in query expected_response_code=expected_response_code File "/usr/local/lib/python3.7/site-packages/influxdb/client.py", line 267, in request timeout=self._timeout File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request resp = self.send(prep, **send_kwargs) File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 646, in send r = adapter.send(request, **kwargs) File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) requests.exceptions.ConnectionError: HTTPConnectionPool(host='myhost', port=8086): Max retries exceeded with url: /query?q=SHOW+SERIES+LIMIT+1%3B&db=weatherstation (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ffaf5f9b610>: Failed to establish a new connection: [Errno 111] Connection refused')) ``` **Additional information:**

So it’s well known that the problem is caused by influxdb not being ready when HA starts the sensor up. The sensor init ‘crashes’, leaving a ‘yellow triangle’ in lovelace. HA does not later try to recover the sensor again when influx becomes available. Non-hassio folks will find a solution to this problem in the second link. However it looks like those using the hassio add-on are kinda screwed at the moment.