I’m happily a user of HAOS, but I got curious what it would take to run a similarly encapsulated version of Supervisor, except in Docker. In steps Sysbox.
I won’t get into Sysbox much, but essentially it lets you run unprivileged Docker-in-Docker. Naturally, a Supervisor installation runs quite a few containers, so I thought I’d give it a shot.
TL;DR it runs, but it’s not happy about it. I was satisfied to call it an initial success when I was able to add and control a wifi light bulb, but I suspect most things not requiring hardware access would work fine.
If you’d like to try this out yourself, I’ve summarized what I did and the files required in a gist:
I outline some of the issues I ran into and what I did to fix them. I’m fairly happy with these initial results and if anyone would be interested in trying to move this forward, I’d be more than happy to work with you!
To head off questions on why I did this, it was mostly out of curiosity. However, I think it could be a potentially easier and lighter weight alternative to running HAOS in a VM down the road, so it’s worth exploring!
I’m quite interested in sysbox cus I’m trying to get a multi-container addon to work. Got a alpha version of the addon now that just runs a docker-compose command which wirs up about 4 containers on the host. Ideally I’d like dind and get security backup from 1. How compatible is sysbox with the s6 overlay?
Good idea thought btw
Building upon my last message, how did you actually install sysbox. I’m on the version of HA that uses buildroot and everything is done by the supervisor. Suppose a run script could work.
I’m quite interested in sysbox cus I’m trying to get a multi-container addon to work. Got a alpha version of the addon now that just runs a docker-compose command which wirs up about 4 containers on the host. Ideally I’d like dind and get security backup from 1.
What’s the mutli-container add-on you’re trying to get to work? That’s actually an interesting use case I don’t think add-ons currently support? Based on add-on architecture, I can see a few problems with multiple containers for one add-on, but it also sounds like you roughly got it working? Could be a good use case for Sysbox running within HAOS, though.
How compatible is sysbox with the s6 overlay?
I’m actually not very familiar with s6. From what I gathered, it’s roughly some kind of systemd equivalent?
Building upon my last message, how did you actually install sysbox.
I’m on the version of HA that uses buildroot and everything is done by the supervisor. Suppose a run script could work.
So it sounds like you may be running HAOS?
It’s possible that you could install Sysbox inside HAOS, but generally it isn’t advisable to mess with anything at the host level when running HAOS since it’s supposed to be managed for you and touching anything could break the system. Additionally, you might need to build Sysbox yourself in order to install it on HAOS because I don’t think whatever HAOS is based on is one of the supported distros by Sysbox.
What I was doing was looking to use a Sysbox container in place of the HAOS VM. You can think of the Sysbox container like a VM since it handles a lot of similar workloads as a VM. However, since HAOS is an OS, I was using Supervisor since I’d say that’s the bulk of what HAOS is. If I wanted to get it closer to HAOS, I’d need to modify the Sysbox container to work more like HAOS, if that makes sense?
Replying on my phone, tad difficult to quote replies. Sorry.
Basically I’ve been trying to get Birdcage to work as an addon. The issue being that it’s made up of multiple containers. One option would be to cut it up and splice back into a single container. Bit too much of a hassle tbh.
As it was a docker-compose set up I initially went for the docker in docker and used the privilege option. But for some reason I couldn’t get that to work and I was pressed for time. So I used the other method and spun up a container with docker-compose installed and privilege. Runs ace but the system complains about there being non supervised containers, all the addon does is run docker-compose for the host system.
S6 is the init system in a nutshell, much like systemD.
As you mentioned this would be a great way of adding way more addons that use the docker per service paradigm. Without having to go gung ho about system safety. And even turning HAOS into a single docker install.
About installing sysbox, you are right I’m using the HAOS. But from my understanding, the base OS only really spins up docker and that you can install packages in the supervisor container that’ll run fine. Well from what I read on another thread.
Moreover you are right that sysbox doesn’t exist for arch. But technically, nothing does right. And only gods forsaken would use a package manager
Are you just running the containers as they are, or have you actually set them up as add-on containers? I could be wrong, but I think add-on containers have certain specificities to them and maybe base image?
One thing to mention is that you can’t nest Sysbox containers. So if the Supervisor base image was running Sysbox then the add-ons wouldn’t be able to run in Sysbox.
But from my understanding, the base OS only really spins up docker and that you can install packages in the supervisor container that’ll run fine.
You probably can install packages fine, but I’m sure the base OS has been customized in HAOS.
I would be curious what the differences are between HAOS and Supervisor. Like if it’s just vanilla Supervisor running on HAOS or what else they’ve done to it or the base system.
There’s a good chance we could run into limitations with hardware access or at best it would complicate things a bit where we’d need to mount devices from the host.
Moreover you are right that sysbox doesn’t exist for arch.
Been a while since I used Arch, but I wouldn’t be too surprised if it wasn’t in the AUR. Also their docs are pretty good, so I’m sure even installing on a system they don’t have a package for shouldn’t be too bad.
The first container that is run as an addon follows the guidelines, it uses the s6 overlay as pid 1 etc and is built upon the images distributed by HA. The following containers that are spun up with docker-compose are not and use custom images. However they could be built in situ I suppose with the HA base, not sure how to interface that and get the supervisor to not complain. That’s why I like the sysbox poss.
Ah that would by dindind or d^-3
It would be interesting if a HA dev/guru could give some insight. But I can already imagine the answer, much like what you said prior “set and forget” sort of thing.
