Kubernetes Helm Chart

Share your Projects!
1

In case anyone is interested, I’ve authored a kubernetes helm chart for running Home Assistant via helm on a kubernetes cluster.

Any feedback is welcomed, as I’m going to submit this to the official helm charts repo.

https://github.com/billimek/billimek-charts/tree/master/home-assistant

6 Likes
2

Very interesting, thank you for sharing that

3

The chart was merged into the official charts repo today: https://github.com/helm/charts/tree/master/stable/home-assistant

1 Like
4

Hi could you please share some details how your home cluster looks like?

5

Hi @runningman84, this is the repo I have with all of the configurations and definitions of the cluster I run: https://github.com/billimek/k8s-templates

Specifically it’s a 3-node cluster for HA. But having a 3-node (HA) cluster is not necessary for home assistance. Especially given that Home Assistant can’t currently run in a clusters/HA fashion right now.

The specific setup for my implementation of Home Assistant via the helm chart is: https://github.com/billimek/k8s-templates/tree/master/deployments/home-assistant

6

I’m looking through your repo @billimek and it’s exactly what I’ve been envisioning but stuck researching each bit and piece, thanks for paving the way! I run an all-in-one ESXi host with FreeNAS VM for NFS, and simple docker VM with separate run files. I’ve been toying with Rancher for k8s. Couple questions though…

I see you have a separate repo for you homeassistant config, how are you getting that in from your helm install? Is it a one-time setup from configurator in the chart that then persists?

Also, just curious do you have an OpenVPN server deployment I missed? I think that’s about the only other thing I’d want to add. My all-in-one setup kinda complicates things (I have extra machines and intrigued by HA, but don’t need extra power or want to pay for it). I feel like I can run almost everything in k8s if I can just get around the chicken and egg problems. Some things that are more fundamental infrastructure I’m running as VMs like graylog, gitlab, jump box, eventually FreeIPA, etc. but debating.

7

Hi @jp83! I keep all of the home-assistant config stuff in a separate repo and leverage the Home Assistant Configurator to manage the edits, commit, and push to github; so that the config changes are versioned and separated from the runtime.

Tactically, when deploying home assistant in kubernetes the first time, I ‘migrated’ the entire /config directory from the old runtime to the new one. How you go about doing this will depend on the type of backend storage persistence used in the kubernetes cluster. Generally speaking though, running something like kubectl cp <some local directory> <the target home assistant pod name>:/config will generally work.

Please note that I sometimes had trouble with using the Home Assistant recorder component in an NFS-backed persistent storage space. I think this is due to the fact that it defaults to a sqllite3 database and that sometimes has trouble when saved to an nfs-backed storage.

I haven’t messed with OpenVPN yet but am using an ikev2-based VPN solution that runs in a different VM.

8

@billimek your github repos are a great insperation and have already helped me a lot.

Today I deployed home assistant using your helm chart which works just fine. But there is an issue if you want to update the home assistant version. Because if you do not use storage like nfs you cannot mount the same pv twice. Which means kubernetes waits forever to deploy home assistant because it cannot start the new pod and it does not delete the old pod.

One solution would be to use statefulsets for home assistant. From my point of view this is a better fit.

What do you think? Btw. I am using opnebs for storage.

9

@runningman84 I’ve been thinking about this a lot recently and 100% agree with you. A StatefulSet approach seems more appropriate than a Deployment for this. I’ll try to get the chart updated to adopt this in case no one else submit a PR for the same.

I, too, leverage a non-shared storage solution (rook/ceph) and have the same issues you describe wit upgrading. One way I’ve been getting around it for now has been to edit the deployment and change the update strategy from RollingUpdate to Recreate. Like you say, StatefulSet is the way to go for this.

10

Have you already migrated your install to a stateful set? I am not sure whether to include appdaemon and cloud9 into the same pod…

11

Ok this is my newest idea. Homeassistant and appdaemon should run as independent deployments with no real storage. The configuration is stored in git and a git sync init container downloada the configuration prior startup. You can exec git commands within the container or cloud9 env to persist configuration. This means you can even run two instances during upgrades. A node failover also works just fine.

The question is how good will be such solution in the long run because homeassistant stores some configuration in the filesystem in some internal files. Is it enough to also include them in git?

1 Like
12

This is a cool idea! If there was a good way to manage secrets via env variables, or a ConfigMap (which creates a secrets yaml file at init perhaps); this would solve the issue with not wanting to persist secrets inside a git repo.

There are some other files, though, that I think require some persistent storage that aren’t appropriate for a git repo. For example:

  • The default database is a local sqllite3 file
  • known_devices.yaml
  • Some specific deice files containing keys, e.g. nest.conf, .ring_cache.pickle, .ios.conf
  • .uuid
13

Let’s look into your examples:

  • The sqllite3 database is only for the history, if this is important you can still an external mysql database.
  • The known_devices.yaml can be stored in git too (I also rename devices there quite often to have a better internal idea). Storing it in git would help
  • The .uuid and other file could also be stored in git

In order to sync the config with git I will use this project:


The sync will be done in an init container as described in their example. Other sync will be manually using kubectl exec or using a permanent side car.

I do not have a problem of storing sensitive information in git if it is a private repo.

But we could also store all usernames, passwords in kubernetes secrets and load them with the env_var option:

14

My POC works fine using git as config storage and env secrets for sensitive information.

Right now I run home assistant within the kubernetes network. My old nmap device tracker does not work in this case. I hesitate to run it in host network mode because I would loose ingress support and I could not run multiple instances.

How do you solve this issue?

I might create a nmap2mqtt app which could run anywhere and sends the data to the mqtt broker in the cluster. Or I could run a single purpose home assistant instance in host network just for tracking and somehow pass the status to the master instance.

15

I leverage the MetalLB tool to run services with real dedicated IPs for things that don’t live behind an https ingress. My plan is to do something similar with mqtt (looking at this as a starting point), and handling mqtt as an external thing instead of the built-in one from Home Assistant. But I suppose The Home Assistant chart could be extended to expose the necessary port.

Not sure why nmap would need hostnetwork. Is it possible to configure it to work with the subnet where your devices live instead of the default kubernetes network?

16

I was thinking to do this as well.
But then I did not know why.

Do you have a problem or usecase you solve with home assistant on kubernetes?

Using kubernetes and helm at work a lot and love it. But software which is not able to cluster is not that well suited.
Therefore I did decide to skip the complexity and go for Hass.io on my local Ubuntu server.

Would love to get your thoughts.

17

Hi @ajfriesen, my personal use-case for Home Assistant in kubernetes was primarily for learning and personal development. Previously I was running Home Assistant via docker executed from a shell script, and it worked well.

I guess aside from learning, a benefit is the scheduling capabilities to move the workload to a different node when one is not available. In my situation, there are three physical nodes so I can take one offline to do work, and not impact Home Assistant. But that’s a stretch.

For me, leveraging the tool I’m learning (kubernetes/helm/all the CNCF stuff) for real-world practical applications like Home Assistant, plex, etc is more meaningful than reading a blog post. I’ve since migrated all of my home-server workloads from a single-node docker implementation to a three-node kubernetes cluster. It’s overkill, there are now several new layers of abstraction in place, but the journey has been worthwhile.

2 Likes
18

@ajfriesen
I my day job I also do a lot of cloud and container stuff. Using the same cutting edge technology at home is a great learning experience.

@billimek
The nmap device tracker does not work in a different subnet because it uses arp to gather the mac addresses. This only works in the same subnet. I already use metallb at home it works just fine. I would recommend to use it with kube-router because this will preserve the client ip.

19

I will solve the device tracker and other multicast related problems by running home assistant with host network. Supporting services like mqtt and myself just run as normal pods with k8s network.

20

Ok I have updated the helm chart to support git:
https://github.com/runningman84/charts/commit/38ff400f4820164ea860491ddeb58becacdac06c

@billimek what do you think about it?

You can just use the hass configurator to persist any state or config file in git. Once the host or pod dies a new pod will come up and restore the latest config from git. No need for persistent volumes.