Docker pull issues: error 404 and TLS handshake timeout

Hi,

I have issues with Docker pull on Hassio (with a fresh new install on RPi 3b+). I’m looking for a help and I want to share my actual workaround.

When I’m trying to install different Hassio addons sometimes installation fails with the following message:

19-03-08 11:42:56 ERROR (SyncWorker_18) [hassio.docker.interface] Can't install esphome/esphome-hassio-armhf:1.11.2 -> 404 Client Error: Not Found ("no such image: esphome/esphome-hassio-armhf:1.11.2: No such image: esphome/esphome-hassio-armhf:1.11.2").

I’ve seen this post and of course I’ve checked the disk space usage. There were a plenty of free space on 32GB card.

I’ve seen this issue on a fresh Hassio 0.87 setup. Usage of hassio CLI (via SSH) helped me and after multiple failing attempts installation succeeded (hassio addons install ADDON_NAME). I’ve installed all the necessary plugins and there were no need to debug this issue anymore…

…until I’ve tried to build my own Hassio addon. I followed this instructions. Hassio passes a build argument of BUILD_FROM to the Dockerfile during a local addon build. I had the following error in the logs:

19-03-08 11:54:08 INFO (SyncWorker_18) [hassio.docker.addon] Start build local/armhf-addon-monitor_sh:1
19-03-08 11:54:20 ERROR (SyncWorker_18) [hassio.docker.addon] Can't build local/armhf-addon-monitor_sh:1: error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/49/491e35c555c44cdb8bfe02f014d4aa805c7f284463d4683bac1938492201531e/data?verify=1552049050-fw4yHYrFe%2BWi%2B%2BLluukWabxsAQw%3D: net/http: TLS handshake timeout

So it looked like a problem with a pull of the images. Addon installation also raised 404 Not found errors. Pulling of the same arm architecture Docker image on my desktop in the same LAN works without any issues.

I followed an official instructions to get a debug access to Hassio (gen SSH public key, place on USB drive called config, SSH to port 22222). When I’ve tried to pull and addon image I received the following error:

# docker pull esphome/esphome-hassio-armhf:1.11.2
1.11.2: Pulling from esphome/esphome-hassio-armhf
48cb104a58ab: Already exists
5010f922d782: Already exists
6b93c416a0aa: Already exists
ffa8bb6d2501: Already exists
bf854b36793b: Already exists
3e7ede0524df: Already exists
31ccb0690cef: Pulling fs layer
e1749995cd9b: Pulling fs layer
f8b85d0c79e2: Pulling fs layer
b03f8f76ce31: Waiting
b7ada96e53b5: Waiting
925f28556344: Waiting
544ec465634c: Waiting
3bc8a6b9948d: Waiting
error pulling image configuration: Get https://production.cloudflare.docker.com/registry-v2/docker/registry/v2/blobs/sha256/b5/b5f38ea84cab2c054a98e33d0c9008b0dccee465d54c79072e12f2450af26969/data?verify=1552049321-cIY%2FtE%2BiHTV7o5bhw%2FmAvoYrMdE%3D: net/http: TLS handshake timeout

That was an original cause of 404 Not found error. I’ve checked DNS settings and it is similar to what I have on the desktop. Ping to production.cloudflare.docker.com also was successful.

I found a very similar report of TLS handshake timeout issue. The solution in that case was to reconfigure Docker daemon and use only 1 concurrent layer download.

So I decided to reconfigure Hassio Docker daemon, but it was painful because:

  • the only place with docker params configuration is /etc/systemd/system/docker.service.d/hassos.conf which is place on read only volume
  • restart of the Docker daemon crashes SSH connection (and whole Hassio as well, restart required)
  • vi is the only editor available there

I reconfigured running Docker daemon this way by applying the following configuration (vi /etc/docker/daemon.json):

{
    "max-concurrent-downloads": 1,
}

Docker supports change of the configuration for some options without daemon reload (see CONFIGURATION RELOAD BEHAVIOR section here). After saving daemon.json you need to send a signal to Docker to reload config. First you need Docker daemon PID. See ‘Main PID’ in the output of systemctl status docker. Then you need to run the following command kill -SIGHUP 310 where 310 is the PID (replace it with yours).

Unfortunately changing max-concurrent-downloads didn’t help. Setting production.cloudflare.docker.com as an insecure registry also didn’t help. The workaround that I’m using at the moment is the following:

  1. I’ve installed Docker registry cache from this repo (you need Docker and Docker compose on your Desktop)
  2. I’m using the following daemon.json config:

.

{
    "registry-mirrors": ["http://192.168.0.108:5000"]
}

Where 192.168.0.108 is my desktop IP with running registry cache. In this case Hassio pulls images from my Desktop and everything works as it should.

Do you have any recommendation on how to fix that TLS handshake error?

Thanks in advance :slight_smile:

I’m getting the following error when trying to install Caddy. I just referenced this post in another post

19-03-09 17:09:37 INFO (SyncWorker_9) [hassio.docker.interface] Pull image korylprince/hassio-caddy-armv7 tag 1.4. 19-03-09 17:09:40 ERROR (SyncWorker_9) [hassio.docker.interface] Can’t install korylprince/hassio-caddy-armv7:1.4 -> 404 Client Error: Not Found (“pull access denied for korylprince/hassio-caddy-armv7, repository does not exist or may require ‘docker login’”)

i’ve installed this addon many times before - no issues. And it seems to do this for a few other addons as well?

I think you have a different issue… But anyway I recommend you to ssh to the host system and run docker logout and docker login commands (with your Docker Hub credentials). P.S. Don’t forget to make a backup.