HA in Docker Swarm / high availabilty cluster

Configuration
1

Hi all,
I have searched everywhere to find a working setup for Home Assistant in docker swarm. I have tried to combine everything I found into a (somewhat) working setup. I have three nodes to create a high available cluster. HA is connected to a MariaDB and I manage the entire setup with Portainer. The hardest part was the network setup. This worked for me:

  1. create a macvlan_local on every node, with a subnet that is part of the rest of my network ( I have split my subnet into two parts:
    a) ip range .0 - .200 managed by DHCP and
    b) ip range .224/27 managed by docker macvlan
  2. create a macvlan_swarm, based on the local macvlan (--config-from macvlan_local)

docker network ls | grep macvlan

NETWORK ID     NAME                                 DRIVER    SCOPE
xxxxxxx        macvlan_local                        null      local
xxxxxxx        macvlan_swarm                        macvlan   swarm

For the rest;

  • 3 nodes (LibreElec on WETEK hub, Ubuntu 20.04.1 LTS (64bit) on RPi3b and same on a RPi4)
  • Synology NAS with NFS share with docker config
  • docker version 20.10.2
  • deployed/created everything from Portainer

docker-compose

version: '3.7'

services:
  homeassistant:
    restart: always
    image: homeassistant/home-assistant
    volumes:
      - <path to homeassistant config>:/config
      - /etc/localtime:/etc/localtime:ro
    ports:
      - 8123:8123
    depends_on:
      - mariadb_swarm_mariadb
    deploy:
      replicas: 1
    networks:
      - macvlan_swarm
      
networks:
  macvlan_swarm:
    external: true

Things todo are; my Tesla account gets lost sometimes after a failover or node reboot. Same for Nabu Casa. I think both have to do with token created on the 1st ip address and after failover/reboot HA runs at a different ip address/node.

Still; I am happy with the way it works right now. If you are interested in the detailed setup, let me know. If you have any ideas for improving; do let me know as well… :slight_smile:

Screenshot from cluster visualizer in Portainer:

Untitled
Untitled994×727 198 KB

1 Like
2

Nice setup! I have noticed the same issue with my Nest integration when I run my Home Assistant on a Kubernetes cluster. Keep you posted!

1 Like
3

I considered running HA with k3s on the 3 RPi3 I have in my house. Ended up frustrated everytime, because:

  • persistent volumes: there is no easy way to establish a shared persistent volume, unless using a central NFS server (which introduces a single-point of failure)
  • HA doesn’t scale. It‘s a monolith and you can only run a single instance on your cluster.
4

@verguldebarman, how do you manage MQTT in the swarm? Is it standalaone or in a swarm as well? Which software did you use in case of swarm or other HA mode? I’m currently trialing both Swarm and K8 and find that there aren’t many vendor guides on how to run mqtt cluster in Swarm? It’s mostly about K8.

@m0wlheld, what was your CPU utilisation with K8 on RPi and overall responsivness? I’ve got mine on intel J1800 and it averages on 15-20% idle load w/o any Home Assistant on top. Also, all K8 processes seem to occupy about 1G of RAM. Also, on persistent storage. What software did you tried. I’m trialing Gluster with locally mounted volume and it seems fine so far.

5

I used k3s, a minimized version of K8s along with GlusterFS. K3s does not come with a Gluster storage driver and I failed to use local path.

6

Sorry guys, I’ve quit using swarm… Back to single nodes. I was testing/changing too much, causing the environment unstable. Perhaps someday… If someone (else) finds a good stable solution

Ps, didn’t use MQTT in the swarm.

7

Pity - the more add ons exist, the more you add things in HACS, the more something like swarm makes sense. I hope someone (\wave Nabu Casa) takes swarm seriously.

I’m running HA-Supervised, and the container count keeps growing.

root@HA161:~# docker ps | wc -l
24