How to successfully configure HA Docker behind Traefik for use_x_forwarded_for and ip_ban_enabled

I am attempting to migrate from hassbian to using docker on a Raspberry Pi to consolidate services onto one device and improve security by using a reverse proxy setup.

So far everything is working as I have letsencrypt providing the ssl certificate and Traefik is creating the locations based on the docker tags.

The issue I am running into is when ever I attempt to access the new setup and check the logs, the IP shown is always from the Traefik proxy container no mater the configuration I have applied via tags in the docker setup or on the front end config with use_x_forwarded_for or trusted_proxies I can not get the IP shown in the logs for access to come from the outside IP.

The configuration is as follows:

Outside IP ā†’ Modem ā†’ Router ā†’ Pi ā†“
___________________________Traefik ā†’ HASS

Curent configuration.yaml:

http:
  base_url: hass-test.my.duckdns.org
  use_x_forwarded_for: true
  trusted_proxies:
    - 127.21.0.2 #Traefik container IP
    - 192.168.1.0/24 #Router IP
  ip_ban_enabled: true
  login_attempts_threshold: 5

And the docker-compose.yml:

version: "3"

services:
  traefik:
    container_name: traefik
    image: traefik
    # Warning: Enabling API will expose TrƦfik's configuration.
    # It is not recommended in production,
    # unless secured by authentication and authorizations
#    command: --api --logLevel=INFO
    command: --logLevel=INFO
    domainname: ${DOMAINNAME}
    networks:
      - traefik_proxy
      - local
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    environment:
      - DUCKDNS_TOKEN=${DUCKDNS_TOKEN}
      - TZ=${TZ}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/docker/traefik/traefik.toml:/traefik.toml
      - ~/docker/traefik/acme/acme.json:/acme.json
      - ./log:/var/log/traefik
    labels:
      - traefik.enable=true
      - traefik.port=8080
      - traefik.backend=traefik
      - traefik.docker.network=traefik_proxy
    restart: always

  homeassistant:
    container_name: homeassistant
    image: homeassistant/raspberrypi3-homeassistant:0.80.0
    depends_on:
      - traefik
    volumes:
      - ~/docker/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    labels:
      - traefik.enable=true
      - traefik.backend=homeassistant
      - traefik.frontend.rule=Host:hass-test.${DOMAINNAME}
      - traefik.docker.network=traefik_proxy
      - traefik.port=8123
      - traefik.frontend.whiteList.useXForwardedFor=true
      - traefik.frontend.passHostHeader=true
    networks:
      - traefik_proxy
      - local
    ports:
      - 8123:8123
    environment:
      - TZ=${TZ}
    restart: unless-stopped

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  local:
    driver: bridge

docker-compose logs -f shows the following when I point my web browser to the external DNS:

homeassistant    | 2018-10-13 17:10:40 INFO (MainThread) [homeassistant.components.http.view] Serving / to 172.21.0.2 (auth: False)
homeassistant    | 2018-10-13 17:10:40 INFO (MainThread) [homeassistant.components.http.view] Serving /auth/authorize to 172.21.0.2 (auth: False)
homeassistant    | 2018-10-13 17:10:40 INFO (MainThread) [homeassistant.components.http.view] Serving /auth/providers to 172.21.0.2 (auth: False)
homeassistant    | 2018-10-13 17:10:41 INFO (MainThread) [homeassistant.components.http.view] Serving /auth/login_flow to 172.21.0.2 (auth: False)

Anyone have any thoughts as to where I have the configuration wrong?

1 Like

Youā€™re going to kick yourselfā€¦ you typed your first trusted_proxies in wrong.

In HA you typed 127.21.0.2 and it should have been 172.21.0.2

And you only need the IP of the TrƦfik containerā€¦ not the router

1 Like

ha_steve,

Good catch! Unfortunately after changing the IP that was not the issue. I now believe this is a bug as there are three open issues with X-Forwarded-For all showing nearly the same issue:

https://github.com/home-assistant/home-assistant/issues/16693
https://github.com/home-assistant/home-assistant/issues/16298
https://github.com/home-assistant/home-assistant/issues/15350

Thanks for the help as am sure that config issue would have gotten me after the bug is fixed.

For reference configuration.yaml now is as follows:

# Uncomment this if you are using SSL/TLS, running in Docker container, etc.
http:
  base_url: hass-test.my.duckdns.org
  use_x_forwarded_for: true
  trusted_proxies:
    - 17.21.0.2 #Traefik container IP
#IP ban dissabled while X-Forwarded-For is broken to prevent blocking reverse proxy
#  ip_ban_enabled: true
#  login_attempts_threshold: 5

Weird. I am in 0.80 with the same setup and my HA logs the real IP of the connecting device.

Just to add a bit more info now that Iā€™m at my laptopā€¦

Hereā€™s what I have in my config.yaml:

http:
  api_password: !secret http_password
  base_url: !secret url
  use_x_forwarded_for: true
  trusted_proxies: !secret trusted_proxies

trusted_proxies is set to my traefik container IP address

One thing I did to help debug this is that is installed a docker container whoami:

docker run -d --name=whoami -p 80:80 emilevauge/whoami

Then set this behind traefik like your hass (traefik needs to forward to port 80). Then if you use a browser and go to (through traefik) to your whoami instance, you will see all of the headers that come through like this (I sanitized for public posting):

Hostname: *****
IP: ***.***.***.***
IP: ***.***.***.***
GET / HTTP/1.1
Host: whoami.mydomain.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Encoding: br, gzip, deflate
Accept-Language: en-us
X-Forwarded-For: ***.***.***.***
X-Forwarded-Host: whoami.mydomain.com
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: server1
X-Real-Ip: ***.***.***.***
2 Likes

ha_steve,

Thanks for the help. Iā€™ll try setting my config.yaml the same way later and see if that fixes it.

I tried debugging with that container but could not get it to run as it apparently does not support the ARM platform:

standard_init_linux.go:190: exec user process caused "exec format error"

Do you have any ā€œadvancedā€ Traefik labels do you have set on the homeassistant container other than the host and frontend and backend names? I tried adding traefik.frontend.whiteList.useXForwardedFor=true and traefik.frontend.passHostHeader=true to see if those were required but they did not help.

Iā€™ll play around with this later this evening. Hopefully this is not related to be run on an ARM platform.

@ha_steve,

Thanks for the second set of eyes on this. I must have been using 127.xxx.xxx.xxx the entire time and messed it up.

After correcting the IP (and making sure I did not end it in .0.1), everything now works like it should.

Iā€™ll leave the working config for anyone else:
docker-compose.yml

version: "3"

services:
  traefik:
    container_name: traefik
    image: traefik:latest
    # Warning: Enabling API will expose TrƦfik's configuration.
    # It is not recommended in production,
    # unless secured by authentication and authorizations
#    command: --api --logLevel=INFO
    command: --logLevel=INFO
    domainname: ${DOMAINNAME}
    networks:
      - traefik_proxy
      - local
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    environment:
      - DUCKDNS_TOKEN=${DUCKDNS_TOKEN}
      - TZ=${TZ}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ~/docker/traefik/traefik.toml:/traefik.toml
      - ~/docker/traefik/acme/acme.json:/acme.json
      - ./log:/var/log/traefik
    labels:
      - traefik.enable=true
      - traefik.port=8080
      - traefik.backend=traefik
      - traefik.docker.network=traefik_proxy
    restart: always

  homeassistant:
    container_name: homeassistant
    image: homeassistant/raspberrypi3-homeassistant:0.80.2
    depends_on:
      - traefik
    volumes:
      - ~/docker/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    labels:
      - traefik.enable=true
      - traefik.backend=homeassistant
      - traefik.frontend.rule=Host:hass-test.${DOMAINNAME}
      - traefik.docker.network=traefik_proxy
      - traefik.port=8123
    networks:
      - traefik_proxy
      - local
    ports:
      - 8123:8123
    environment:
      - TZ=${TZ}
    restart: unless-stopped

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
  local:
    driver: bridge

http: config

http:
#  api_password: !secret http_password
  base_url: !secret url
  use_x_forwarded_for: true
#  trusted_proxies: !secret trusted_proxies
# or
  trusted_proxies:
    - 172.xxx.0.2 #IP of Traefik container
  ip_ban_enabled: true
  login_attempts_threshold: 5

Thanks again!

Are you doing anything to ensure your container always has the same IP?

The 172 address is assigned dynamically. What happens if you reboot your host and it issues a new IP range?

@poldim

I have configured static IPā€™s in the docker-compose.yml for the containers for deterministic IPā€™s for any of the containers in that stack. The network is external and as such does not change IP range.

Example:

version: "3"

services:
  traefik:
    image: traefik:latest
    container_name: traefik
    ....
    networks:
      default:
        ipv4_address: 172.xxx.0.2
    ports:
      ....

networks:
  default:
    external:
      name: traefik_proxy

@TheNotSoSmartHome I canā€™t find any info on docker-config.yml. Did you mean docker-compose.yml?

Yes.

That is correct. docker-config.yml should be docker-compose.yml

Iā€™m getting a docker error when trying to use the ipv4 syntax. It says:

ERROR: for traefik user specified IP address is supported only when connecting to networks with user configured subnets

Any ideas?

You need to create an external network first. I think that error pops up if youā€™re trying to use the default default network / not defined an external network so itā€™s trying to create it during deployment which has been known to cause that error.

Post your docker-compose.yaml?

I split my docker-compose into multiple files, so each directory below has its own docker-compose.yml file. When I use portainer, this shows up as three different stacks. I suppose itā€™s possible that using three three different stacks like this might be part of the problem but I doubt it, so I havenā€™t tried a ā€œsingleā€ stack yet.

/home/futuretense/stacks/network
/home/futuretense/stacks/homeautomation
/home/futuretense/stacks/media

But since the issue here is related to home assistant, Iā€™m only showing the network
and homeautomation files. I snipped out parts I didnā€™t think were relevant, and left things that were network related.

/home/futuretense/stacks/network/docker-compose.yaml

Summary
#network stack
version: "3.7"
services:

  traefik:
    hostname: traefik
    image: traefik:latest #1.7-alpine
    container_name: traefik
    restart: unless-stopped
    domainname: ${DOMAINNAME}
    networks:
      default:
     #   ipv4_address: 172.27.0.4  ERROR user specified IP address is supported only when connecting to networks with user configured subnets
      traefik_proxy:
    ports:
      - 80:80
      - 443:443
      - 8080:8080
    environment:
      - CLOUDFLARE_EMAIL=${CLOUDFLARE_EMAIL}
      - CLOUDFLARE_API_KEY=${CLOUDFLARE_API_KEY}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ${USERDIR}/stacks/network/traefik:/etc/traefik
    labels:
      - traefik.frontend.rule=Host:traefik.${DOMAINNAME}  
      - traefik.enable=false
      - traefik.frontend.headers.SSLRedirect=true
      - traefik.backend=traefik
      - traefik.port=8080
      - traefik.docker.network=traefik_proxy
      - traefik.frontend.headers.STSSeconds=315360000
      - traefik.frontend.headers.browserXSSFilter=true
      - traefik.frontend.headers.contentTypeNosniff=true
      - traefik.frontend.headers.forceSTSHeader=true
      - traefik.frontend.headers.SSLHost=${DOMAINNAME}
      - traefik.frontend.headers.STSIncludeSubdomains=true
      - traefik.frontend.headers.STSPreload=true
      - traefik.frontend.headers.frameDeny=true

  openvpn:
    ...
    networks:
      pihole_sub:
        ipv4_address: 172.68.0.99

  pihole:
    ...
    networks:
      pihole_sub:
        ipv4_address: 172.68.0.2
      default:
    ...

  portainer:
    ...
    networks:
    ...
  cloudflare-ddns:
    ...

  cloud9:
    ...

networks:
  traefik_proxy:
    external:
      name: traefik_proxy
      
  default:
    driver: bridge
  pihole_sub:
    driver: bridge
    ipam:
      driver: default
      config:

        - subnet: 172.68.0.0/16

/home/futuretense/stacks/homeautomation

Summary
#home automation stack
version: "3.7"
services:

  homeassistant:
    container_name: homeassistant
    restart: unless-stopped
    image: homeassistant/home-assistant:latest
    depends_on:
      - mqtt
    devices:
      - /dev/ttyUSB0:/dev/ttyUSB0
      - /dev/ttyUSB1:/dev/ttyUSB1
      - /dev/ttyACM0:/dev/ttyACM0
    volumes:
      - ${USERDIR}/stacks/homeautomation/homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 8123:8123
    privileged: true
    environment:
      - PUID=${PUID}
      - PGID=${PGID}
      - TZ=${TZ}
    network_mode: host
    labels:
      - com.centurylinklabs.watchtower.enable=true

  esphome:
    ...
    network_mode: host
    ...
    
  mqtt:
    ...

networks:
  default:
    driver: bridge

Run the following command to create an external network:

docker network create --subnet=172.xxx.xxx.0/16 traefik_proxy

Then you can assign IPā€™s to containers. Change xxx to numbers you want.

1 Like

Hey, Iā€™m trying to follow this thread but when I launch trefik docker-compose i get

ERROR: Service ā€œtraefikā€ uses an undefined network "traefik_proxy

image

I have created the docker network as advised before.

docker network create --subnet=172.xxx.xxx.0/16 traefik_proxy

Any help please ;'(

Can you post your docker-compose.yaml?

There it goes. :+1:

version: ā€œ3ā€
services:
traefik:
container_name: traefik
image: traefik:latest
# Warning: Enabling API will expose TrƦfikā€™s configuration.
# It is not recommended in production,
# unless secured by authentication and authorizations
command: --api --logLevel=INFO
command: --logLevel=INFO
domainname: xxxxx.duckdns.org
networks:
- traefik_proxy
- local
ports:
- 80:80
- 443:443
- 8080:8080
environment:
- DUCKDNS_TOKEN=xxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
- TZ=America-Santiago
volumes:
- /var/run/docker.sock:/var/run/docker.sock
- /mnt/sdb1/docker/traefik/traefik.toml:/traefik.toml
- /mnt/sdb1/docker/traefik/acme/acme.json:/acme.json
- /mnt/sdb1/docker/traefik/log:/var/log/traefik
labels:
- traefik.enable=true
- traefik.port=8080
- traefik.backend=traefik
- traefik.docker.network=traefik_proxy
restart: always

Do you have the following in the YAML?

Also, I would remove the local network.

I get another error now :no_mouth::

image

ERROR: The Compose file ā€˜./docker-compose.ymlā€™ is invalid because:
services.traefik.networks.default contains unsupported option: ā€˜externalā€™