hijinx
August 29, 2017, 11:39am
1
My hass.io is on supvervisor version 0.58 and hass version 0.51.2:
Both hass and hass.io versions have changed but these are not being seen or reported by hass.io .
In the logs I am seeing:
17-08-29 11:27:26 INFO (MainThread) [__main__] Start Hassio
17-08-29 11:27:26 INFO (MainThread) [__main__] Run Hassio loop
17-08-29 11:27:26 INFO (MainThread) [hassio.addons.git] Pull addon https://github.com/notoriousbdg/hassio-addons repository
17-08-29 11:27:26 INFO (MainThread) [hassio.addons.git] Pull addon https://github.com/bestlibre/hassio-addons repository
17-08-29 11:27:26 INFO (MainThread) [hassio.addons.git] Pull addon https://github.com/james-fry/hassio-addons repository
17-08-29 11:27:26 INFO (MainThread) [hassio.addons.git] Pull addon https://github.com/hassio-addons/repository repository
17-08-29 11:27:26 INFO (MainThread) [hassio.addons.git] Pull addon https://github.com/home-assistant/hassio-addons repository
17-08-29 11:27:26 INFO (MainThread) [hassio.dns] Start DNS port forwarding for host add-ons
17-08-29 11:27:26 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-29 11:27:31 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-29 11:27:31 INFO (MainThread) [hassio.core] Start hassio api on 172.30.32.2
17-08-29 11:27:31 INFO (MainThread) [hassio.addons] Startup system run 1 addons
17-08-29 11:27:31 INFO (SyncWorker_7) [hassio.dock.interface] Clean bestlibre/amd64-influxdb docker application
17-08-29 11:27:32 ERROR (MainThread) [hassio.addons.git] Can't pull https://github.com/bestlibre/hassio-addons repo: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v origin
stderr: 'fatal: unable to access 'https://github.com/bestlibre/hassio-addons/': Couldn't resolve host 'github.com''.
17-08-29 11:27:32 ERROR (MainThread) [hassio.addons.git] Can't pull https://github.com/notoriousbdg/hassio-addons repo: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v origin
stderr: 'fatal: unable to access 'https://github.com/notoriousbdg/hassio-addons/': Couldn't resolve host 'github.com''.
17-08-29 11:27:32 ERROR (MainThread) [hassio.addons.git] Can't pull https://github.com/james-fry/hassio-addons repo: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v origin
stderr: 'fatal: unable to access 'https://github.com/james-fry/hassio-addons/': Couldn't resolve host 'github.com''.
17-08-29 11:27:32 ERROR (MainThread) [hassio.addons.git] Can't pull https://github.com/hassio-addons/repository repo: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v origin
stderr: 'fatal: unable to access 'https://github.com/hassio-addons/repository/': Couldn't resolve host 'github.com''.
17-08-29 11:27:32 ERROR (MainThread) [hassio.addons.git] Can't pull https://github.com/home-assistant/hassio-addons repo: Cmd('git') failed due to: exit code(1)
cmdline: git pull -v origin
stderr: 'fatal: unable to access 'https://github.com/home-assistant/hassio-addons/': Couldn't resolve host 'github.com''.
Looks like a problem with DNS in the supervisor container since the version.json on github cannot be retrieved since hosts are not getting resolved…
The OS hosting hass.io (ubuntu) can resolve github.com :
james@hassio:~$ ping github.com
PING github.com (192.30.253.113) 56(84) bytes of data.
64 bytes from lb-192-30-253-113-iad.github.com (192.30.253.113): icmp_seq=1 ttl= 47 time=79.4 ms
64 bytes from lb-192-30-253-113-iad.github.com (192.30.253.113): icmp_seq=2 ttl= 47 time=79.5 ms
Any ideas why this can happen @pvizeli ?
hijinx
August 29, 2017, 4:19pm
2
Actually the problem is wider than just updated.hass.io host none of the hass.io docker containers can resolve hostnames.
mqtt
owntracks
darksky weather
unifi presence detection with cloud hosted unifi server
From the host OK resolution works fine:
james@hassio:~$ ping github.com
PING github.com (192.30.253.113) 56(84) bytes of data.
64 bytes from lb-192-30-253-113-iad.github.com (192.30.253.113): icmp_seq=1 ttl=47 time=79.5 ms
64 bytes from lb-192-30-253-113-iad.github.com (192.30.253.113): icmp_seq=2 ttl=47 time=80.0 ms
64 bytes from lb-192-30-253-113-iad.github.com (192.30.253.113): icmp_seq=3 ttl=47 time=79.5 ms
^C
--- github.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2000ms
rtt min/avg/max/mdev = 79.544/79.706/80.013/0.217 ms
When I connected to the hass.io supervisor container resolution fails, but you can see that there is connectivity as ping by IP works fine…
bash-4.3# ping github.com
^C
bash-4.3# ping 192.30.253.113
PING 192.30.253.113 (192.30.253.113): 56 data bytes
64 bytes from 192.30.253.113: seq=0 ttl=46 time=79.944 ms
64 bytes from 192.30.253.113: seq=1 ttl=46 time=79.485 ms
^C
--- 192.30.253.113 ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max = 79.485/79.714/79.944 ms
bash-4.3#
So seems definitely is a DNS issue.
@pvizeli could this be related to the networking changes added in 0.58?
Hass.io 0.58
Fix network options: null/None now disable that port
New network layer for internal processes
The new network layer allow now to communicate inside hassio network with every addon with hostname or docker name of that addon. Also all addon can use homeassistant that resolve to homeassistant instance. You can also use the add-on names inside homeassistant configuration. More feature, based on that will follow soon. All internal containers like supervisor have now a fix IP on that n…
pvizeli
August 29, 2017, 4:40pm
3
No, the hassio container will be run on host dns and it run in both network on same time (hassio-network/docker-network). Only the containers that run on host network will be use the internal dns server from hassio. Other add-ons use the internal docker dns server.
I think you need update the docker, maybe there is a bug? It could be also a local time problem that made that the ssl certs are not valid
pvizeli
August 30, 2017, 12:24pm
4
Do you see more? On you logs, I can’t see a failing ping. I think there is a time issue.
hijinx
August 30, 2017, 1:11pm
5
Thanks for checking further.
james@hassio:~$ date
Wed 30 Aug 13:52:11 BST 2017
james@hassio:~$ docker exec -i -t hassio_supervisor /bin/bash
bash-4.3# date
Wed Aug 30 12:52:23 UTC 2017
bash-4.3#
I didnt change anything in the host OS or docker install.
Out of interest for the hassio_supervisor container I can see following resolv.conf, which only contains loopback address:
bash-4.3# cat /etc/resolv.conf
search local
nameserver 127.0.0.11
options ndots:0
The homeassistant container has this resolv.conf:
bash-4.3# cat /etc/resolv.conf
search local
nameserver 172.30.32.2
172.30.32.2 is the IP address of the hassio supervisor…
I hacked the hassio supervisor container resolve.conf (yes I know, naughty raw.githubusercontent.com and github.com and hence can get new versions.json and the update is detected + works…
So definitely a DNS issue where the hassio supervisor cannot resolve names.hass.io install, which I have updated several times, before stop working in this way?
hijinx
August 30, 2017, 1:19pm
6
Even with the above “success” for the hassio supervisor, the homeassistant container does not resolve names via hassio supervisor as supposed to:
the containers that run on host network will be use the internal dns server from hassio
pvizeli
August 30, 2017, 2:07pm
7
hijinx:
127.0.0.11
That is the internal docker DNS Server. Hassio use this server to resolve DNS requests and at the end also all other hassio host add-ons.
If the internal docker DNS server will no work, it mean that some things in your docker Setup is wrong. To fix that permanently you need care that you docker setup work correct with his own internal DNS server. Maybe a docker update on host? I can confirm that with 1.17.03 It work all like it should.
hijinx
August 30, 2017, 2:11pm
8
OK will look into this.
hijinx
August 30, 2017, 3:52pm
9
I’m going to go with the brutal approach to delete hass.io and reinstall from script.
pvizeli
August 30, 2017, 4:49pm
10
I’m not sure if this will help you. Since it looks like the docker internal dns server is broken. I think a docker reinstall or update can be have a real affect for you problems. It is true that we first use the internal docker dns approach since 0.58
hijinx
August 30, 2017, 4:54pm
11
I have a hunch that 0.58 broke it hass.io .
BTW the commands I used to install docker + hassio, and am following again this time, are documented here:
From bash history this is what I executed:
sudo apt-get install apt-transport-https
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
apt-cache policy docker-ce
sudo apt-get install -y docker-ce
sudo systemctl status docker
sudo usermod -aG docker ${USER}
su - ${USER}
sudo apt -y install bash socat jq curl
s…
hijinx
August 30, 2017, 4:58pm
12
And this time I took a VM snapshot before I installed hassio so no need to reinstall the OS again
hijinx
August 30, 2017, 7:49pm
13
Hi @pvizeli
It’s not good news.hass.io as with my original install that was working fine up until 0.58 update (see post above linking to the exact commands I ran).hass.io will not install fully:http://hassio:8123/ , but it never completes the install.
Last time it completed in less than a minute…
Pretty sure its still the docker DNS issue at heart.
This is how I install docker on Ubuntu x86:
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get update
apt-cache policy docker-ce
sudo apt-get install -y docker-ce
Thanks for your input as I currently have zero HASS
hijinx
August 30, 2017, 8:00pm
14
@delstel @mak are you guys still using hass.io on ubuntu? Are you seeing similar issues post 0.58?
hijinx
August 30, 2017, 8:17pm
15
I have verified that its the same DNS issue.github.com ’:
ERROR (MainThread) [hassio.addons.git] Can't clone https://github.com/home-assistant/hassio-addons repo: Cmd('git') failed due to: exit code(128)
cmdline: git clone --recursive -v https://github.com/home-assistant/hassio-addons /data/addons/core
stderr: 'Cloning into '/data/addons/core'...
fatal: unable to access 'https://github.com/home-assistant/hassio-addons/': Couldn't resolve host 'github.com'
Full logs:
james@hassio:~$ docker logs hassio_supervisor
starting version 3.2.2
17-08-30 16:58:51 INFO (MainThread) [__main__] Initialize Hassio setup
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create Home-Assistant config folder /data/homeassistant
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio ssl folder /data/ssl
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio addon data folder /data/addons/data
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio addon local repository folder /data/addons/local
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio addon git repositories folder /data/addons/git
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio temp folder /data/tmp
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio backup folder /data/backup
17-08-30 16:58:51 INFO (MainThread) [hassio.bootstrap] Create hassio share folder /data/share
17-08-30 16:58:51 INFO (MainThread) [hassio.dock.network] Can't find HassIO network, create new network
17-08-30 16:58:52 INFO (MainThread) [__main__] Run Hassio setup
17-08-30 16:58:52 INFO (SyncWorker_0) [hassio.dock.supervisor] Attach to supervisor homeassistant/amd64-hassio-supervisor with version 0.60
17-08-30 16:58:58 INFO (MainThread) [hassio.host_control] Send 'info' to HostControl.
17-08-30 16:58:59 INFO (MainThread) [hassio.host_control] Receive from HostControl: { "type": "generic", "version": "0.3", "last_version": "0.3", "hostname": "hassio", "features": ["shutdown", "reboot", "update"], "os": "GNU/Linux" }.
17-08-30 16:58:59 INFO (MainThread) [hassio.homeassistant] No HomeAssistant docker homeassistant/qemux86-64-homeassistant found.
17-08-30 16:58:59 INFO (MainThread) [hassio.homeassistant] Setup HomeAssistant landingpage
17-08-30 16:58:59 INFO (SyncWorker_2) [hassio.dock.interface] Pull image homeassistant/qemux86-64-homeassistant tag landingpage.
17-08-30 16:59:01 INFO (SyncWorker_2) [hassio.dock.interface] Tag image homeassistant/qemux86-64-homeassistant with version landingpage as latest
17-08-30 16:59:01 INFO (MainThread) [hassio.addons.git] Clone addon https://github.com/home-assistant/hassio-addons repository
17-08-30 16:59:07 ERROR (MainThread) [hassio.addons.git] Can't clone https://github.com/home-assistant/hassio-addons repo: Cmd('git') failed due to: exit code(128)
cmdline: git clone --recursive -v https://github.com/home-assistant/hassio-addons /data/addons/core
stderr: 'Cloning into '/data/addons/core'...
fatal: unable to access 'https://github.com/home-assistant/hassio-addons/': Couldn't resolve host 'github.com'
'.
17-08-30 16:59:07 ERROR (MainThread) [hassio.addons] Can't load from repository core
17-08-30 16:59:07 INFO (MainThread) [hassio.addons] Load addons: 0 all - 0 new - 0 remove
17-08-30 16:59:07 INFO (MainThread) [hassio.addons] Startup initialize run 0 addons
17-08-30 16:59:07 INFO (MainThread) [hassio.snapshots] Found 0 snapshot files
17-08-30 16:59:07 INFO (MainThread) [__main__] Start Hassio
17-08-30 16:59:07 INFO (MainThread) [__main__] Run Hassio loop
17-08-30 16:59:07 INFO (MainThread) [hassio.dns] Start DNS port forwarding for host add-ons
17-08-30 16:59:07 INFO (MainThread) [hassio.addons] Load addons: 0 all - 0 new - 0 remove
17-08-30 16:59:07 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 16:59:12 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 16:59:12 INFO (MainThread) [hassio.tasks] Found new HassIO version None.
17-08-30 16:59:12 INFO (MainThread) [hassio.dock.supervisor] Update supervisor docker to homeassistant/amd64-hassio-supervisor:None
17-08-30 16:59:12 INFO (SyncWorker_3) [hassio.dock.interface] Pull image homeassistant/amd64-hassio-supervisor tag None.
17-08-30 16:59:13 ERROR (SyncWorker_3) [hassio.dock.interface] Can't install homeassistant/amd64-hassio-supervisor:None -> 404 Client Error: Not Found ("manifest for homeassistant/amd64-hassio-supervisor:None not found").
17-08-30 16:59:13 INFO (MainThread) [hassio.core] Start hassio api on 172.30.32.2
17-08-30 16:59:13 INFO (MainThread) [hassio.addons] Startup system run 0 addons
17-08-30 16:59:13 INFO (MainThread) [hassio.addons] Startup services run 0 addons
17-08-30 16:59:13 INFO (SyncWorker_2) [hassio.dock.homeassistant] Start homeassistant homeassistant/qemux86-64-homeassistant with version landingpage
17-08-30 16:59:13 INFO (MainThread) [hassio.addons] Startup application run 0 addons
17-08-30 16:59:13 INFO (MainThread) [hassio.homeassistant] Setup HomeAssistant
17-08-30 16:59:13 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 17:00:13 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 17:00:18 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 17:00:18 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 17:01:18 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 17:01:23 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 17:01:23 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 17:02:23 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 17:02:28 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 17:02:28 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 17:03:28 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 17:03:33 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 17:03:33 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
delstel
August 30, 2017, 8:22pm
16
Hi - Haven’t used it for a couple of weeks… I’ll check tonight.
mak
August 30, 2017, 8:55pm
17
yep … my hassio is alive and well :
Supervisor
Host OS
Home Assistant
mak
August 30, 2017, 9:04pm
19
i also thing that this is a DNS misuse.
connect to the hassio_supervisor container and start /bin/bash :
sudo docker exec -i -t hassio_supervisor /bin/bash
try :
ping google.com
ping github.com
DNS response :
nslookup google.com
nslookup github.com
and try with google dns as the DNS server (8.8.8.8)
nslookup google.com 8.8.8.8
nslookup github.com 8.8.8.8
Usage: nslookup [HOST] [SERVER]
NOTE: type :
exit
1 Like
hijinx
August 30, 2017, 10:24pm
20
Thanks for checking, @mak
You can see the output for some of these in earlier posts as I was diagnosing…
Anyway… here goes:
james@hassio:~$ docker exec -i -t hassio_supervisor /bin/bash
bash-4.3# ping google.com
ping: bad address 'google.com'
bash-4.3# ping github.com
ping: bad address 'github.com'
bash-4.3# nslookup google.com
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'google.com': Try again
bash-4.3#
bash-4.3# nslookup github.com
nslookup: can't resolve '(null)': Name does not resolve
nslookup: can't resolve 'github.com': Try again
bash-4.3# nslookup google.com 8.8.8.8
Server: 8.8.8.8
Address 1: 8.8.8.8
nslookup: can't resolve 'google.com': Try again
bash-4.3# nslookup github.com 8.8.8.8
Server: 8.8.8.8
Address 1: 8.8.8.8
nslookup: can't resolve 'github.com': Try again
But if I update /etc/resolv.conf (in the container) the ping and nslookup both immediately work:
bash-4.3# cat /etc/resolv.conf
search local
nameserver 8.8.8.8
nameserver 127.0.0.11
options ndots:0
bash-4.3#
bash-4.3#
bash-4.3# nslookup github.com
nslookup: can't resolve '(null)': Name does not resolve
Name: github.com
Address 1: 192.30.253.112 lb-192-30-253-112-iad.github.com
Address 2: 192.30.253.113 lb-192-30-253-113-iad.github.com
bash-4.3#
bash-4.3#
bash-4.3# ping github.com
PING github.com (192.30.253.113): 56 data bytes
64 bytes from 192.30.253.113: seq=0 ttl=46 time=579.916 ms
64 bytes from 192.30.253.113: seq=1 ttl=46 time=80.160 ms
64 bytes from 192.30.253.113: seq=2 ttl=46 time=84.541 ms
64 bytes from 192.30.253.113: seq=3 ttl=46 time=79.929 ms
^C
--- github.com ping statistics ---
4 packets transmitted, 4 packets received, 0% packet loss
round-trip min/avg/max = 79.929/206.136/579.916 ms
Also now HASS kicked into life on the 60 sec retries since github.com now can be resolved:
james@hassio:~$ docker logs hassio_supervisor
...
...
17-08-30 22:17:45 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 22:18:45 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 22:18:50 WARNING (MainThread) [hassio.updater] Can't fetch versions from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json -> [Errno -3] Cannot connect to host raw.githubusercontent.com:443 ssl:True [Try again]
17-08-30 22:18:50 WARNING (MainThread) [hassio.homeassistant] Error on install HomeAssistant. Retry in 60sec
17-08-30 22:19:50 INFO (MainThread) [hassio.updater] Fetch update data from https://raw.githubusercontent.com/home-assistant/hassio/master/version.json
17-08-30 22:19:50 INFO (SyncWorker_2) [hassio.dock.interface] Pull image homeassistant/qemux86-64-homeassistant tag 0.52.1.
17-08-30 22:21:10 INFO (SyncWorker_2) [hassio.dock.interface] Tag image homeassistant/qemux86-64-homeassistant with version 0.52.1 as latest
17-08-30 22:21:10 INFO (MainThread) [hassio.homeassistant] HomeAssistant docker now installed
17-08-30 22:21:10 INFO (SyncWorker_3) [hassio.dock.interface] Cleanup docker images: ['homeassistant/qemux86-64-homeassistant:landingpage']
17-08-30 22:21:15 INFO (SyncWorker_0) [hassio.dock.interface] Stop homeassistant/qemux86-64-homeassistant docker application
17-08-30 22:21:16 INFO (SyncWorker_0) [hassio.dock.interface] Clean homeassistant/qemux86-64-homeassistant docker application
17-08-30 22:21:16 INFO (SyncWorker_0) [hassio.dock.homeassistant] Start homeassistant homeassistant/qemux86-64-homeassistant with version 0.52.1
) to also contain nameserver 8.8.8.8 (google public DNS server) and now hassio supervisor can resolve 
