I use 3 nodes that are “equal”
- in docker all are managers and workers
- storage is mirrored between the 3 of them by using glusterfs
- all devices are either from mqtt or use another type of network detection, so it does not matter where the device is
- mysql as db storage was setup as master/master to run on 2 different nodes, with the 3rd acting as a backup every hour (and upload encrypted backup to remote server)
Any of the 3 nodes can fail, and almost everything can run on one server if needed. I have a big ups on the 3 nodes ( j1900 quad-core nodes with ssd, about 10w / hour power usage), and if the ups goes under 2 hours runtime one node will shutdown to keep as much time as possible. when power goes under 30 minutes all non-security stuff is stopped and when power goes under 10 minutes all security stuff moves to 2 raspberry pi that have a 48 hour battery.
If you want to use docker and gluster you need to do this
- setup docker and the swarm
- disable docker auto start
- stop all docker nodes
- setup glusterfs
- create a cronjob that check if glusterfs is mounted and running - if yes, start docker, if not - stop docker
If you start docker before gluster, bad things will generally happen…
I installed everything on a basic ubuntu18, on celeron cpus (as I said above), with ssd and 16 gb of ram each and 2 intel network cards (one for storage/inter-cluster), one for the rest of the network. They use about 15w in “idle”, 30w full load…
I tried PI’s but when you add up the pi, the power source and everything else needed to make it stable, power on / shutdown automatically it’s not much cheaper than an asrock j1900 board (or similar) but more complicated… Now the PI’s are running as various radios serial to network adapters and bluetooth scanners (and some other small, read-only jobs).