Iāve been picking away at this a little myself and youāre right, it entirely depends on what youāre running. Towards that, Iāve tried to limit the number of communication protocols used. I am working really hard to limit my primary protocol to MQTT because it can be completely independent of the HA controller. Also, even though MQTT doesnāt natively support load balancing or failover, there are some folks that have developed a basic MQTT server using the protocol that does.
From there, it is entirely possible to have two instances of HA running on two different Pis as long as each share an external recorder database (in my case, using mysql) and one of the two has all of the automations turned off.
I have been running two HAs for a few months so I can see what kind of issues arise and Iām happy to say that very few issues have come up as long as automations are off on the secondary instance.
I havenāt tested it yet but HAProxy should work for handling the URL routing.
My bigger issue is with z-wave. Iām thinking that an external gateway would be the most practical solution and have it run through HAProxy as well but that is just speculative at this point.
No, not yet. Honestly, I havenāt gotten that far yet.
Iām glad the link was useful. That project introduced me to HAProxy which was a very useful find.
Currently, the main focus is HA. Yes, the mechanism could be applied to other services but they wouldāve to implement the heartbeat mechanism to send messages to keeper. Personally I think this project should, for now, focus on Home Assistant which is the principal point of failure. That said, this only performs service restarts but Iām open to suggestions such as backups (or others) which is another piece of high availability.
nragon, very nice indeed.
Digging into HAProxy, could we have multiple instances on this oneā¦ What I mean is that āentry-pointsā should always respond. I saw some pictures about only one entry point, the load balancer, but what if that craps out.
Historically, load balancers are by default a single point device/service and subject to being a SPOF but ordinarily, the load balancer is the only software on the device and is therefore very stable.
Generally, you canāt foolproof everything but, if you ensuring failover or redundancy on the most complicated pieces, that will greatly reduce risk.
My goal is to have redundant access points, redundant HA hub, redundant database, redundant MQTT and backup power for everything. Outside of that, having replacements of the rest (like network switches, routers, cables, etc.) and a restore plan that is documented, is about as good as you really need. Ultimately, making sure that your home automation system fails dumb instead of stupid, is your overall best possible scenario.
I wrote up my goals and assumptions as someone who is designing my system for a person who has a disability (my wife) in this article if interested.
I think load balancing is not the main issue here. Availability is. Does any of you get 100%+ on HA? @anon3881519, Keper can totally evolve into an active/passive approach deployed in multiple machines. Currently the downtime is the time needed for HA to start.
So, I was reading up on Docker, and stumbled upon Docker Swarm.
Seems that with its setup of āmanagersā and āworkerā nodes, High Availability can be achieved.
nodes can be both physical or virtual.
So with āDocker on Dockerā we could add Raspberries or other hardware that runs docker and add it to a cluster.
Thereās been a fair bit of work recently on high availability configurations of kubernetes, and also on using rasberry pi clusters for kubernetes. Assuming you have a high availability kubernetes backplane (i.e. multi master), storage and multiple cluster nodes, high availability for home assistant (excluding Z-Wave) should be possible through:
Configuring HA to run in a kubernetes pod or pods
Deploying a single instance of HA (i.e. your configured pod(s))
Allowing kubernetes to detect failure and restart the failed pod(s) on another cluster node
The kubernetes backplane takes care of monitoring, failover and routing. Failover probably wouldnāt be instant, but tuning could make it acceptably fast.
k3s (https://k3s.io) looks particularly promising, but doesnāt have a high availability config yet.
I run home assistant in a kubernetes cluster using k3s. Even in a non ha cluster you can survive node outages if the master keeps alive. If the master is broken your workers will just continue with their work. A master node can also act as a worker node at the same time. This means in a two node setup you will survive the outage of any one node. With a three node setup you might survive two simulations node outages.
I have recently created a project called HAHA - Highly Available Home Assistant, which creates a highly available cluster which runs Home Assistant.
It runs using Docker swarm and includes a MariaDB Galera cluster and Mosquitto MQTT broker. It uses GlusterFS to synchronize Home Assistantās files, so that in case of a failure, all of the state is transferred to other nodes.
Hi, would you like to share your config? Tried to use the helm config but can only get it to work with hostnetwork=true on k3s?
Are you running it with hostnetwork?
I would like to not use hostnetwork and use metallb as loadbalance for a fixed ip-address to connect inwards and traefik as reverse proxy. Anyone got that working and able to share his/her deployment manifest?
Edit: ok, like always, after posting I find my errorsā¦ Seems to work now, although not yet with runAsUser, which I would like. If anyone is interested I can post the deploy I currently use.
Edit2: saving to lovelace ui fails. Needs some workā¦
Actually, such a problem could be avoided altogether. Think of MS SQL Always-On. This is setup as a cluster. Each individual device has an assigned IP, then there is a āsharedā IP for the cluster (only one-node would actually be active at a time). In the case of MS-SQL, for Write functions, this is true, and only one of the instances is in āwriteā mode, while the other is strictly in read mode. For read operations, it would really matter which node is being looked at.
In this way, delays can be minimized. This would also offer redundancy in that there are actually two complete copies of the DB. Biggest thing to work out is syncing data across both instances. This could be applied to services as well.
I have had an instance of HA fail for no apparent reason (running on rPi). Re flashed the SD card, and restored my config, and everything is up and running again. If I had a cluster though, there would have been no outage, and perhaps even a restore could be achieved faster, since a clustered setup would have to sync the configurations.
Just a thought. A native HA āclusterā/āload-balancedā config would be very cool.
This could be processor intense, so depending on your load, might require more than a rPi3, maybe an rPi4 minimum.
Just my two cents.
Come to think of it, clustering in Linux already exists. If someone could "sync the HA DB, and the configurations (rsync), then this should be possible running HA under linux on a couple of small machines.
,