My docker swarm setup

Share your Projects!
1

I’ve been trying to get Homeassistant working on swarm for a few months now, but the thing that was preventing me from moving to swarm was the Homeassistant requirement to use host networking on the container.
I had tried many things but I finally got everything to work with the macvlan driver. I configured the homeassistant service with two networks:

  • One macvlan network, giving it an IP address in my iot network
  • One overlay network, giving traefik access to homeassistant

The macvlan network is configured as follows. I had to create a local macvlan network on each member of the swarm, with a subnet swarm is free to choose an IP from:

1st node:
docker network create --config-only --subnet=192.168.2.0/24 --gateway=192.168.2.1 -o parent=eth0 --ip-range 192.168.2.232/29 macvlan_local
2nd node:
docker network create --config-only --subnet=192.168.2.0/24 --gateway=192.168.2.1 -o parent=eth0 --ip-range 192.168.2.240/29 macvlan_local
3rd node:
docker network create --config-only --subnet=192.168.2.0/24 --gateway=192.168.2.1 -o parent=eth0 --ip-range 192.168.2.248/29 macvlan_local

Then I create a swarm scoped network:

docker network create -d macvlan --scope swarm --config-from macvlan_local macvlan_swarm

Then I can deploy the whole stack, here is the docker-compose file:
(the wait-for-it config is a script that prevents homeassistant from starting before mqtt, it’s not necessary but quite useful)

version: '3.7'

configs:
  wait-for-it:
    file: /srv/docker/homeassistant/wait-for-it/wait-for-it.sh

services:
  homeassistant:
    image: homeassistant/armhf-homeassistant:0.86.4
    configs:
      - source: wait-for-it
        target: /wait-for-it.sh
        mode: 755
    networks:
      - webgateway
      - macvlan_swarm
    command: ["/wait-for-it.sh", "rasper:1883", "--", "python", "-m", "homeassistant", "--config", "/config"]
    volumes:
      - /srv/docker/homeassistant/config:/config
      - /etc/localtime:/etc/localtime:ro
    deploy:
      labels:
        - "traefik.backend=hassio"
        - "traefik.frontend.rule=Host:myhomeassistanthostname"
        - "traefik.port=8123"
        - "traefik.enable=true"
        - "traefik.docker.network=webgateway"
        - "traefik.default.protocol=http"

  mosquitto:
    image: eclipse-mosquitto
    volumes:
      - /srv/docker/mosquitto/config:/mosquitto/config
      - /etc/localtime:/etc/localtime:ro
      - /srv/docker/mosquitto/data/mosquitto-data:/mosquitto/data
      - /srv/docker/mosquitto/data/mosquitto-log:/mosquitto/log
    ports:
      - "1883:1883"
      - "9001:9001"

  nodered:
    image: nodered/node-red-docker:rpi-v8
    deploy:
      labels:
        - "traefik.backend=nodered"
        - "traefik.frontend.rule=Host:mynoderedhostname"
        - "traefik.port=1880"
        - "traefik.enable=true"
        - "traefik.docker.network=webgateway"
        - "traefik.default.protocol=http"
    networks:
      - webgateway
    volumes:
      - /srv/docker/nodered:/data
    environment:
      - TZ:Europe/Brussels


volumes:
  mosquitto-data:
  mosquitto-log:

networks:
  webgateway:
    external: true
  macvlan_swarm:
    external: true
  hostnet:
    external: true
    name: host
7 Likes
2

I am also struggling to get homekit and wemos to be discovered. Exposing ports explicitly is not working in Swarm mode.

It did work when I wasn’t using swarm mode and was using net: host.

I followed your steps and I am getting the following error:

network "webgateway" is declared as external, but could not be found. You need to create a swarm-scoped network before the stack is deployed
3

The webgateway network is something that you have to create outside of the docker-compose, it’s the network I use to connect my containers to the traefik reverse proxy.
If you’re not using a reverse proxy, you should remove it from the configuration

4

Hi! This is a very interesting setup!
I am also working on dockerising my home assistant setup using docker-compose.
Do you know if your setup allows for using Bluetooth device inside home-assistant container?

5

This thread is about a swarm

A swarm is multiple docker hosts running the service in a “high availability” mode…unless you are using net=host, the bluetooth is not accessible to the container.

Are you trying to setup a swarm or just a normal single docker host with docker compose?

6

I am setting a single-host solution at this moment, so I have set the “host” network mode, which runs fine.
Despite that I am curious about the swarm concept and I was wondering about solutions for giving docker container access to bluetooth devices without the use of network_mode = host.

7

This is very interesting ! I’m not sure to undertand the macvlan network configuration though … why use 3 nodes ? One for each of your containers (mosquitto, nodered, hass) ?

Also, would you mind sharing how you integrate all of this with traefik ? I’ve been using it for a few month but I can’t get rid of the network_mode:host setting for hass, or all of my xiaomi devices will stop working.

8

because it’s a swarm?

9

@Vlavonvidden docker swarm is a scheduler for docker, something like kubernetes but much, much simpler to set up (and not as complete).

The three nodes allow me to have high availability, meaning that if one the node dies, the container is moved to another node, with very little downtime and no loss of service.

The macvlan config solves the need to use the host network, which prevented me from using swarm on multiple nodes.
It allows me to assign an IP address in the same network as my IOT stuff, which means that all the stuff that depends on broadcasts actually works (in my case, lifx among other things).

Traefik is only a reverse proxy, it will not allow you to bypass the network host issue, that’s what the macvlan configuration is for.

10

Thanks for your answer ! I’m just starting to learn about docker swarms (well to be honest, about almost everything related to docker) so this is most useful.

I thought using swarms required having multiple (physical) hosts, guess I’ll have to read a lot more about this.

Would you recommand any aprticular resource/article to begin with ?For now, I’m just reading official documentation.

11

Swarm requires at least one physical host, I’m using 3 (actually more) physical hosts, so that everything keeps working if one gets down.

12

@kokninski @flamingm0e

Hey guys…

Were you able to get Bluetooth accessible with docker swarm?

13

@TomerFi
Unfortunetally I have not found a working solution.
All the research I’ve done indicates that network_mode:host has to be set to pass the bluetooth device to the container with home-assistant.

14

Thanks for the reply.
If I’ll happen to find a solution I’ll be sure to updated this post.

15

I’m setting up somethign similar to Edzilla’s setup. For sure a macvlan setup is needed for some integrations to work properly (as they rely on multicast over the host network). I also found out that at the moment there’s a limitation for docker swarm setup to access host’s resources like USB devices (I plan to have a Zigbee network, which by the way will be integrated in HA through mqtt): this limitation comes from cap-add functionality, which aims to add capabilities from the host layer. Until this will be fully implemented, we can’t have access to host devices in a docker swarm, as far as I understood.
But it seems there’s a workaround (I haven’t tested it yet): https://github.com/Koenkk/zigbee2mqtt/issues/2049

16

With your swarm, you have 3 nodes, how many are managers? I’m trying to do the same, but my focus is mostly mosquitto, and the services I have to talk to that. What I’m struggling with is if the primary manager goes down, as this is where I’m most interested in what swarm can do…when the main manager goes goes, the service move to another node and it carries on.

But do this requires the service to be accessible via one name or one ip, and this is presented by docker regardless of what manager is the front. Have you been able to solve that, or is it merely being used for load balancing and assuming the manager is always up>

17

All my nodes are managers. 3 is the minimum number of managers to have high availability.
https://docs.docker.com/ee/ucp/admin/configure/join-nodes/

18

How do you handle accessing home assistant, say if manager 1 dies, and it all moves to manager 2? since the access to HA would change url…do you present one IP via swarm or is there something else that can round robin the 3 managers to figure out which one is up?

19

My fortigate firewall works as a load balancer, so if a node goes down it sends the traffic to another one.
I know that makes it a single point of failure, but I’m more confident in the firewall hardware and stability than in the raspberries :wink:

Also, if the firewall goes down, so does my whole network, so I’ll have greater concerns than home assistant being down!
(but now that you’ve made me think about it, I just might grab an additional firewall from work to set up a high availability cluster)

20

What form does the load balancing take, is it something like haproxy, or have you just done it on dnsmasq? I’d like to get that bit simplified so I can run mulitple managers.