Installing HAOS next do Docker on same host

[shocknl]

Hi there,

Currently i am having difficulties installing HomeAssistant in KVM/LibVirt (using cockpit-project machines plugin).
But that is also not entirely true, installing HomeAssistant is a breeze. But having networking available poses quite difficult.

Backstory: currently running HomeAssistant OS on a Rpi 4 (2gb ram version) and booted from an ssd.
This works fine, but i am now running into the limits of the pi and want something better.
Since I already have a home server for other purposes (a lot of docker containers) i was looking into migrating home assistant to it.
But not run it as docker container but install it in a VM for easier maintainability.

What i want to achieve:
Run both docker containers that are exposed through nginx (nginx-proxy-manager), have AdGuard home as dns in a docker container and next to that run home assistant in a virtual machine.
This all should run on Ubuntu Server 24.04.1 LTS

I thought this would be as easy as just install home assitant on a VM, start it and have it work. Here i got my first issue: the libvirt network is not able to start as dnsmasq is not able to start as adguard already is bound to port 53.
To mitigate this, i’ve set dns enable to no.
See full network config here (retrieved by sudo virsh net-dumpxl default where default is the network name):

<network connections='1'>
  <name>default</name>
  <uuid>bd3673c4-c00a-4419-ad35-38e659c2d0eb</uuid>
  <forward mode='open'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:0b:21:e3'/>
  <domain name='default' localOnly='yes'/>
  <dns enable='no'/>
  <ip address='192.168.100.1' netmask='255.255.255.0' localPtr='yes'>
    <dhcp>
      <range start='192.168.100.2' end='192.168.100.254'/>
    </dhcp>
  </ip>
</network>

This allows the network to be activated.
When i boot HomeAssitant i see that it gets 192.168.100.74 as ip address.
See below how network info looks:

I initially had my network configuration with <forward mode='none'/> but this resulted in HAOS not getting a gateway on the ipv4 interface.

No i would think everything is setup correctly at the home assitant VM side of things.
Then I created a new network in docker using: docker network create -d macvlan --subnet=192.168.100.0/24 --gateway=192.168.100.1 -o parent=virbr0 virbr0_macvlan
And have the Nginx container join that network. To be safe also a restart of that container.
See below the attached networks for that container in Portainer:

Lastly I’ve added a proxy host for home assitant like this:

But when I go to test1.home I get presented a 400: Bad Request page.
I did see one time that i could go to the home assitant page, but no idea anymore what i changed then. I believe it was the same config as above, but i cannot reproduce that anymore.
At least i know that i got presented with an error that home assistant is not able to get the DNS server or something. Where you can select if you want to use cloudflare or some other dns server. Those buttons were greyed out.
So i think it couldn’t even reach internet. Which is also a problem.

I know this is kind of a long shot, as this is really a technical problem, but not sure how to proceed with this.

I did find a lot of topics on the web stating that Libvirt and Docker do not play nice in terms of networking. Well that is definitly the problem here.
But since this server is kind of a production environment, i do not want to just try out all the commands i can find.

Does anyone have an idea what i am missing or doing wrong here?

Thanks in advance!

[shocknl]

For anyone that is having similar issues. I’ve ended up ‘fixing’ it by creating a macvlan bridge network to the hosts network adapter like this on ubuntu server:
ip link add macvlan-br0 link enp3s0 type macvlan mode bridge where enp3s0 is your hosts network adapter/port that you want to bridge to.
ip link set macvlan-br0 up
And since this doesn’t survive a reboot i’ve created a shell script with above content and created a new systemd service in oneshot mode upon boot.
This was not an ideal solution as i initially wanted all my vms to run inside a separate network on the host and then expose it through a proxy.
But this way it just gets an ip address from your ‘normal’ dhcp server (which is my router in my case).
Good enough for now, i do believe there should be a way to make it work in the way i wanted initially, but that is for now a waste of my time.

[wmaker (Tommy Long)]

Hi,
This is a setup I’ve not encountered, so try to learn a little bit from it and have a question.
I guess I should start with do I have the following correct:

• You have setup a libvirt network that will forward traffic to a kernel bridge named virbr0 (virbr0 is typically setup by libvirt anyway). This libvirt network will also assign a static IP 192.168.100.1 to virbr0. The libvirt network will also offer DHCP services for other entities connected to virbr0 (handing out addresses on the same subnet
• Somehow HAOS VM is connected to virbr0. I don’t remember for sure but I believe a libvirt managed VM will use virbr0 by default, so perhaps there was nothing in particular for a user to set up to do this. (I don’t think I saw anything in your writeup on this)
• You have setup a docker network “virbr0_macvlan” that itself sets up macvlan interface that is connected to virbr0. One or more container(s) is setup to use this docker network. So I think this means your Docker containers on this network can now talk with your HAOS VM.
• You have setup another macvlan “macvlan-br0” that connects to enp3s0. But what is connected to or used for? What I think is missing for me is how does HAOS VM get connectivity to the outside world?

[shocknl]

I believe i had tried that. But failed to get it working properly.
Maybe i missed a step or still did some misconfiguration at my end.
Thing is, at least what i’ve read on the interwebs, is that when you install docker, it will mess with your network configuration on your host.
I didnt want to spend any more time on this as i just wanted something to work.

When setting up the network vor libvirt, i was under the impression it would also act as a DHCP server, but i never saw an ip address being assigned to the machine.
Even when i didnt connect the network to a physical adapter i didn’t see the machine getting an IP address. But again, maybe there is some misconfiguration yet again.

I did think of another solution, and that is to make a bridge to the docker network so it talks to that, but I believe that requires me to setup a static ip address for all machines i attach in that way. That in turn will likely cause conflicts with the docker network as it does not know that the static IP was assigned already.

To answer your question: My understanding is that it is more or less of a sidecar to the physicial host connection.
I do not know exactly how this works, but in the configuration mentioned above, the vm will act as a “normal” host on the network. As in it will get an IP from the DHCP server.
The enp3s0 adapter is the physical interface on my machine (can maybe cause some confusion as i was used to stuff like eth0 before)
When I look in my Unifi network environment, it will show as a “hop” after the real host machine. Like it shows up in a similar way as a switch does.