No cloud polling sensor data on hassos for Odroid C2 (beta)

aetjansen · August 31, 2018, 9:36am

Hi all,

I am giving the hassos image for Odroid C2 a go and am running into a couple issues. Since the image is a beta, there may be an issue with the way the image is setup, but I guess that most likely I am doing something wrong:

Right now, I notice that the yr sensor platform for weather data is not pulling in any data. Nor is the wunderground or buienradar sensors I tried. For wunderground I see these errors in the log (repeating every x minutes):
Platform wunderground not ready yet. Retrying in 180 seconds. 11:26 helpers/entity_platform.py (WARNING) Error fetching WUnderground data: ClientConnectorError(-3, 'Try again') 11:26 components/sensor/wunderground.py (ERROR)

Additionally, this error continuously repeats in the log a few times per minute:
Error fetching Cognito keyset: Cannot connect to host cognito-idp.us-east-1.amazonaws.com:443 ssl:None

From some searching I believe this has to do with home assistant trying to connect to home-assistant-cloud and should resolve itself over time, but it has not yet on my instance over the past few hours.

The YR weather plaform generates a similar error:
Retrying in 15 minutes: Cannot connect to host aa015h6buqvih86i1.api.met.no:443 ssl:None [Try again]

So, it seems to me that home assistant cannot reach out to the internet, except that it did throw me a card that an update to 0.77.1 was available and update was succesful. Additionally, the sun component seems to work fine, but not sure if that calculated everything internally based on data in the configuration.yaml.

As said, I am running the latest hassos version available for Odroid-C2 on a 64Gb emmc card. I have forwarded ports 80,443, 8123 and some others for add-ons to the instance and am using a static subdomain and have a static IP address. I have used the lets-encrypt add-on to get a certificate and there is nothing else responding on port 80.

Hass.io supervisor version 128 (latest currently)
System: HassOS 2.2
HA version: 0.77.1

The odroid/hass instance clearly has internet connectivity, because it did succesfully update the HA version, but other sources cannot be connected to and it seems unlikely that all those different platforms have separate disruptions.

Anybody have a clue what I may be doing wrong, or how to figure out if this may indeed be an issue with the image/setup?

aetjansen · August 31, 2018, 1:21pm

Found this post indicating something may need tweaking with network settings: Cloud Update

Also it was suggested to follow these instructions to enable internet access: https://docs.resin.io/reference/OS/network/2.x/#setting-a-static-ip

But, those names and folders mentioned there are not present in the odroid C2 image that I am using?

aetjansen · August 31, 2018, 5:59pm

Odd, most sensors do have a value now, but it updated only once, 4 hours ago. I have not been touching anything on the odroid or the network and we have had no connection issues, so I am currently clueless as to what is happening or how to debug.

aetjansen · August 31, 2018, 6:26pm

Also noticed several SSL handshake errors in the log, which led me to this issue: https://github.com/home-assistant/home-assistant/issues/15579#

But that is closed already, so not sure if this is something that is still being worked on or I missed a fix/workaround…

rbray89 · September 1, 2018, 2:36am

Do you have any logs?

aetjansen · September 1, 2018, 4:58am

not yet/anymore, in trying to debug and fix, I must have messed something up, because there is no way I can get logged in now as the new auth system only presents a “oops something went wrong” message on all configured authentication methods. I am reflashing the image and will start from scratch to see where it goes wrong. please stay tuned

aetjansen · September 1, 2018, 7:47pm

Well, I did 2 fresh installs today, both without any hiccups. I do seem to get SSL handshake errors the first time after I get a certificate with letsencrypt and then restart the hass instance. I have to login twice and that’s it. no further issues.

I am thinking there could be two things I haven’t tried that may have caused this earlier:

When I got the connection issues, I was running 0.77.1, the new installs all were on 0.77.2, but I did not find anything in the release notes indicating a fix for this issue or something related.
The order of installation: when I had the issues, I had setup most everything without ssl, then added letsencrypt and changed the add-ons, etc. The fresh installs today, I started with ssl after the basic install, then did the rest.

But, since it seems to work fine now, I am reluctant to tear it down again to see if those made a difference, since I have only one instance and not a separate test one. (and my wife really misses the smart lighting during the day I had setup before moving to hassio

Thanks to all involved for all the amazing work being done on this amazing platform!!

aetjansen · September 2, 2018, 12:16pm

@rbray89 let the install run overnight and just checked the log and found a bunch of the same errors:

2018-09-01 22:57:47 ERROR (MainThread) [homeassistant.core] Error doing job: Fatal error on transport TCPTransport (error status in uv_stream_t.read callback)
TimeoutError: [Errno 110] Operation timed out
2018-09-01 22:58:09 ERROR (MainThread) [homeassistant.components.updater] Got unexpected response: None
2018-09-01 22:58:40 ERROR (MainThread) [homeassistant.core] Error doing job: Fatal error on transport TCPTransport (error status in uv_stream_t.read callback)
TimeoutError: [Errno 110] Operation timed out
2018-09-01 23:09:01 WARNING (Thread-5) [pychromecast.socket_client] Heartbeat timeout, resetting connection
2018-09-01 23:09:16 ERROR (Thread-5) [pychromecast.socket_client] Failed to connect, retrying in 5.0s
2018-09-01 23:35:28 ERROR (MainThread) [homeassistant.core] Error doing job: SSL handshake failed
Traceback (most recent call last):
  File "uvloop/handles/stream.pyx", line 609, in uvloop.loop.UVStream._on_eof
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:841)
2018-09-01 23:35:28 ERROR (MainThread) [homeassistant.core] Error doing job: SSL error errno:1 reason: UNKNOWN_PROTOCOL
Traceback (most recent call last):
  File "uvloop/sslproto.pyx", line 504, in uvloop.loop.SSLProtocol.data_received
  File "uvloop/sslproto.pyx", line 204, in uvloop.loop._SSLPipe.feed_ssldata
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: UNKNOWN_PROTOCOL] unknown protocol (_ssl.c:841)
2018-09-01 23:46:07 ERROR (InfluxDB) [homeassistant.components.influxdb] Write error
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/homeassistant/components/influxdb.py", line 316, in write_to_influxdb
    self.influx.write_points(json)
  File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 468, in write_points
    tags=tags, protocol=protocol)
  File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 532, in _write_points
    protocol=protocol
  File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 312, in write
    headers=headers
  File "/usr/local/lib/python3.6/site-packages/influxdb/client.py", line 271, in request
    raise InfluxDBClientError(response.content, response.status_code)
influxdb.exceptions.InfluxDBClientError: 400: {"error":"partial write: unable to parse '°C,domain=sensor,entity_id=bijkeuken_gevoelstemperatuur friendly_name_str=\"Bijkeuken gevoelstemperatuur\",value=nan 1535838306050802944': invalid number dropped=0"}

2018-09-01 23:46:08 ERROR (InfluxDB) [homeassistant.components.influxdb] Resumed, lost 4 events
2018-09-02 00:00:00 ERROR (MainThread) [homeassistant.core] Invalid service data for hassio.addon_restart: invalid slug core_letsencrypt[] (try core_letsencrypt) for dictionary value @ data['addon']. Got 'core_letsencrypt[]'
2018-09-02 00:20:05 WARNING (MainThread) [buienradar.buienradar] Missing section in Buienradar xmldata (weerstation).Can happen 00:00-01:00 CE(S)T
2018-09-02 00:22:08 WARNING (MainThread) [buienradar.buienradar] Missing section in Buienradar xmldata (weerstation).Can happen 00:00-01:00 CE(S)T
2018-09-02 01:44:48 WARNING (MainThread) [homeassistant.components.sensor.buienradar] Unable to retrieve raindata from Buienradar.(Msg: , status: None,)
2018-09-02 03:26:23 WARNING (Thread-5) [pychromecast.socket_client] Heartbeat timeout, resetting connection
2018-09-02 03:26:30 ERROR (Thread-5) [pychromecast.socket_client] Failed to connect, retrying in 5.0s
2018-09-02 03:29:35 WARNING (Thread-4) [pychromecast.socket_client] Heartbeat timeout, resetting connection
2018-09-02 03:29:50 ERROR (Thread-4) [pychromecast.socket_client] Failed to connect, retrying in 5.0s
2018-09-02 04:07:59 WARNING (Thread-3) [pychromecast.socket_client] Heartbeat timeout, resetting connection
2018-09-02 04:08:14 ERROR (Thread-3) [pychromecast.socket_client] Failed to connect, retrying in 5.0s
2018-09-02 05:33:21 ERROR (MainThread) [homeassistant.core] Error doing job: SSL handshake failed
Traceback (most recent call last):
  File "uvloop/handles/stream.pyx", line 609, in uvloop.loop.UVStream._on_eof
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: NO_SHARED_CIPHER] no shared cipher (_ssl.c:841)
2018-09-02 05:33:21 ERROR (MainThread) [homeassistant.core] Error doing job: SSL error errno:1 reason: NO_SHARED_CIPHER
Traceback (most recent call last):
  File "uvloop/sslproto.pyx", line 504, in uvloop.loop.SSLProtocol.data_received
  File "uvloop/sslproto.pyx", line 204, in uvloop.loop._SSLPipe.feed_ssldata
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: NO_SHARED_CIPHER] no shared cipher (_ssl.c:841)
2018-09-02 07:39:00 ERROR (MainThread) [homeassistant.core] Error doing job: SSL handshake failed
Traceback (most recent call last):
  File "uvloop/handles/stream.pyx", line 609, in uvloop.loop.UVStream._on_eof
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:841)
2018-09-02 07:39:00 ERROR (MainThread) [homeassistant.core] Error doing job: SSL error errno:1 reason: WRONG_VERSION_NUMBER
Traceback (most recent call last):
  File "uvloop/sslproto.pyx", line 504, in uvloop.loop.SSLProtocol.data_received
  File "uvloop/sslproto.pyx", line 204, in uvloop.loop._SSLPipe.feed_ssldata
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:841)
2018-09-02 09:08:31 ERROR (MainThread) [homeassistant.core] Error doing job: SSL handshake failed
Traceback (most recent call last):
  File "uvloop/handles/stream.pyx", line 609, in uvloop.loop.UVStream._on_eof
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:841)
2018-09-02 09:08:31 ERROR (MainThread) [homeassistant.core] Error doing job: SSL error errno:1 reason: HTTP_REQUEST
Traceback (most recent call last):
  File "uvloop/sslproto.pyx", line 504, in uvloop.loop.SSLProtocol.data_received
  File "uvloop/sslproto.pyx", line 204, in uvloop.loop._SSLPipe.feed_ssldata
  File "uvloop/sslproto.pyx", line 171, in uvloop.loop._SSLPipe.feed_ssldata
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: HTTP_REQUEST] http request (_ssl.c:841)

I have setup the example automation as provided with the docs for the letsencrypt addon, to check for renewal at midnight, but looking at the timestamps, these errors did not happen as a consequence…

Is this anything you can get useful info from?

rbray89 · September 2, 2018, 4:22pm

It looks like there is a periodic outage of network connection. I assume you have it connected via ethernet? Could you try using a different port on your router/switch? Another thing to look at is to make sure your power supply is adequate. You need a solid 5V 2A supply with this board.

There are a few possibilities here:

A Kernel/Driver bug for the ethernet controller on the board
HW issues (broken port, cable not fully seated in port, not enough power, bad cable)
External problems (router bad, router port bad, internet going down, etc.)

Would you be able to enable SSL via these instructions: https://github.com/home-assistant/hassos/blob/dev/Documentation/configuration.md (via authorized_keys) and then post the output of running dmesg?

aetjansen · September 3, 2018, 2:10pm

I must be doing something wrong (again) or misunderstand the instructions in the link you sent me I formatted the usb drive, created a new key pair (no passphrase) on the laptop (mac osx) I want to use to connect to the odroid running hassio.

The instructions state to copy the text in the box “Public key for pasting into OpenSSH authorized_keys file”, but there is no such thing when I do that on my machine. So instead I copied the contents on the .pub file to the authorized_keys file on the usb drive (no extention on that file).

Any attempt to ssh into the odroid as root at port 22222 results in a Permission denied (public key) error
Sorry to be such a pain, but I don’t know how to get that debug access working after a couple attempts. On the other side, those errors have not shown up in the logs for the past 30 or so hours.

rbray89 · September 3, 2018, 4:17pm

There are a couple of really important bits… Did you format the USB drive with the name “CONFIG”?

For more detailed docs on the keys, see here: https://developers.home-assistant.io/docs/en/hassio_debugging.html

You also have to reboot with the drive plugged in, else it won’t initialize.

The one last thing I can think of, is that in my testing, I found that some devices due to a kernel bug, won’t initialize the USB controller or power IC if plugged in alone. Most other devices seem fine, it just seems to be USB flash drives. If you plug in pretty much any dongle along with the flash drive (mouse, wifi, BT, etc.) the USB should initialize.