Work in progress: configuration for running a Home Assistant in containers with systemd and podman on Fedora IoT

jeey2k · May 7, 2022, 10:03pm

Hi

Using your runtime in podman but os ubuntu.
Figured out i could bypass the problem with changing

--sdnotify=container \

To

--sdnotify=conmon \

mattdm · May 7, 2022, 10:14pm

Do you have the same issue as a few posts up? Work in progress: configuration for running a Home Assistant in containers with systemd and podman on Fedora IoT - #77 by mattdm

dte · May 8, 2022, 9:04pm

Just wanted to mention that now everything works fine, thanks for the initial write-up.
I just added a node-red container (I still haven’t really understood hass way to automate things) and the only thing I’d like to review is the whole networking between containers/local host/network.

mattdm · May 8, 2022, 10:48pm

Yeah, and with Podman 4.0 there’s a whole new networking stack by default, so I have to figure that out.

I’m also looking at a project called quadlet, which should simplify the systemd unit files significantly.

kdknigga · May 11, 2022, 4:56pm

If your networking equipment doesn’t offer such a service, you could look into this:

$ dnf info mdns-repeater
Last metadata expiration check: 0:00:05 ago on Wed 11 May 2022 11:55:55 AM CDT.
Available Packages
Name         : mdns-repeater
Version      : 1.11
Release      : 5.fc35
Architecture : x86_64
Size         : 24 k
Source       : mdns-repeater-1.11-5.fc35.src.rpm
Repository   : fedora
Summary      : Multicast DNS repeater
URL          : https://github.com/kennylevinsen/mdns-repeater
License      : GPLv2+
Description  : mdns-repeater is a Multicast DNS repeater for Linux. Multicast DNS
             : uses the 224.0.0.51 address, which is "administratively scoped" and
             : does not leave the subnet.
             : 
             : This program re-broadcasts mDNS packets from one interface to other
             : interfaces.

This will allow mDNS resolution (and therefore, finding Chromecasts) across VLANs.

kdknigga · May 11, 2022, 6:44pm

@mattdm , the thing that lead me to this thread is trying to get zwavejs2mqtt in podman on Centos 8 Stream working.

And… I’m having no luck. zwavejs can’t open the serial port no matter what I’ve tried.

I’ve got container_use_devices turned on, I’ve even set selinux to permissive, my normal user can access the serial port as verified by stty, I’m using “–group-add keep-groups”, I’ve even set the /dev/ttyUSB0 to mode 666 and tried running the container as root, all with no luck. Inside the container stty fails with EPERM as well.

My next step is to try to turn on auditing for accesses to /dev/ttyUSB0 to see if I can get more information about what’s not happy, but if that doesn’t shed any light I’m running out of ideas.

Any thoughts about where my hangup might be?

I’m just trying on the command line right now, so no units yet. Here’s my command:

podman run --rm -it -p 8091:8091 -p 3000:3000 \
--device=/dev/serial/by-id/usb-Silicon_Labs_CP2102N_USB_to_UART_Bridge_Controller_88e1d0665594eb11943836703d98b6d1-if00-port0:/dev/zwave:rw \
--mount type=volume,src=zwavejs2mqtt,target=/usr/src/app/store \
--group-add keep-groups \
--name zwavejs \
zwavejs/zwavejs2mqtt:latest

mattdm · May 13, 2022, 5:59am

Hmmm, I think you’ve covered the bases. I guess I’d double check all of them before I’d do much more. Maybe you have, but, you know.

Do you see SELinux AVCs in permissive mode when you try the access from inside the container?

kdknigga · May 13, 2022, 1:15pm

I’ve checked things a few times. --privileged finally gets access to the serial port, but the device mapping doesn’t come through as specified on the command line. For some reason /dev/ttyUSB0 comes through instead of /dev/zwave being created. I don’t get why, but this is my first foray into device passthrough.

I do see some AVCs, but I think unrelated. Looks like something in the container is trying to run iptables?

type=AVC msg=audit(05/12/2022 16:00:57.596:4453) : avc:  denied  { ioctl } for  pid=39100 comm=iptables path=/var/lib/containers/storage/overlay/f9ba98f13481e19eb879bba20eff4e4b8967d141224c40ccbc014b1d3db81853/merged dev="overlay" ino=50416736 scontext=unconfined_u:system_r:iptables_t:s0-s0:c0.c1023 tcontext=system_u:object_r:container_file_t:s0:c667,c847 tclass=dir permissive=1

I dare say I may have found a bug. This is podman 4.0.2 on Stream… so not a shocker I guess. I’ll take this over to the podman Github issue tracker.

Thanks for looking, and I hope mdns-repeater works for you if you try it out!

Edit: Welp… looks like I may have been right about the bug, but it also looks like I wasn’t the first to notice.

Bugfixes

Fixed a bug where devices added to containers by the --device option to podman run and podman create would not be accessible within the container.

akp55 · June 13, 2022, 10:12pm

you can’t use the /dev/serial/by-id path because that is a symlink, and podman won’t create the underlying device for this.

use this command with readlink to follow the device and map it into your pod

 podman run -d --name zwavejs -v /home/pi/zwavejs:/usr/src/app/store --device $(readlink -f /dev/serial/by-id/usb-Silicon_Labs_Zooz_ZST10_700_Z-Wave_Stick_0001-if00-port0):/dev/zwave --security-opt label=disable --annotation run.oci.keep_original_groups=1 -p 3000:3000 -p 8091
:8091 --replace zwavejs/zwavejs2mqtt

remember to replace it with your device.

akp55 · June 13, 2022, 10:13pm

you don’t actually need privileged as long as you are in the correct groups.

kdknigga · June 13, 2022, 10:29pm

The problems I had were due to a bug in the (then) latest version of podman, not due to symlinks or group membership. Moving to an older version of podman fixed everything for me.

mattdm · June 13, 2022, 10:38pm

Possibly a podman version thing? Using the link works for me.

mattdm · June 13, 2022, 10:39pm

I’m hoping to have some time to update this in the next couple of weeks. I’m going to do a lessons-learned setup and use quadlet for the systemd files – and put all of that in a git repo to share with everyone.

akp55 · June 14, 2022, 7:30am

maybe newer versions? i’m stiil using podman 3 on my rpi4

mattdm · September 11, 2022, 3:19pm

Didn’t have much time to play with this this summer. Going to update it soon though, either using quadlet or podman play kube.

GreyLinux · November 24, 2022, 12:33pm

Hi @mattdm , I’m currently in the process of switching from an arm server to an Intel NUC running Fedora Server 37 with container running using Podman. Before I make the jump I’m researching everything as best I can to understand a problem should it arise.

I have a few questions about your setup , currently I’m on the fence about whether to use systemd or podman play kube I was thinking of running the containers from my docker-compose file with podman compose and then generating the systemd and kubernetes yaml files with podman and then deciding from there.

I take it you are running all your containers, rootless and unprivileged? do you run esphome at all? I notice esphome’s docker compose runs it as privileged I susoect this is for access to USB on the host. If needs be will your selinux module allow usb access to esphome and home assistant? (I use home assistant to manage my zigbee device rather than a another container).

Do you run any containers in pods or are they all stand alone containers?

Is there a benefit to placing your config in /srv/ for exanple from a selinux perspective? I intended to store everything in /home/ as I don’t believe I will create other users that will need access to the files.

What issues do think there will be with podman 4.0 and networking ?

Finally, I use Caddy for my reverse proxy setup and I was going to just change the privileged port down to 80 so I can run the container rootless , but do you recommend your option above of forwarding 80,443 to 8080,8443 and setup caddy to listen on those ports?

thanks in advance

DezeStijn · November 24, 2022, 9:25pm

Bookmarking this thread too have a look at it.
I want to move from Docker-Compose to Podman but the proper way and not just copying my Docker setup through e.g., Podman-Compose.

Did you consider working with pods?
I.e. HA and all its add-ons in one pod, so that they can communicate via localhost and you only expose what’s needed?
Identifying which ports I’d need to expose is where I’m getting stuck at

Or is it best to stick to individual containers?
Does the use of VLANs make sense in that case? I.e. containers that don’t need internet (like Z2M, hassconf, MariaDB) could go into a NOT VLAN?

Are you running this rootless?
Most stuff uses high ports anyway and Nginx can be exposed on a high port and port-forwarded on 80/443 anyway.
Things would only get tricky if you want to set up e.g. PiHole.

Why would you gor for IoT and not Core?
As you undoubtably are more experienced in Fedora than me, I’m curious to hear your opinion.

EDIT1:
Found a typo in ~/.config/systemd/user/container-nginx.service:
--detatch should be --detach or just -d

~~Additional question: How do you get the IP of the Home Assistant container (10.0.2.2) and how are you certain it’s static?~~
It’s the virtual gateway for the podman network

DezeStijn · November 28, 2022, 3:33pm

I have no experience with SELinux. First time Fedora user as well.
Does the following mean SELinux may not be installed/enabled on my Fedora CoreOS system?

$ checkmodule -M -m -o container-usbtty.mod container-usbtty.te
$ semodule_package -o container-usbtty.pp -m container-usbtty.mod
$ sudo semodule -i  container-usbtty.mod
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/mod failed with code: 1. (No such file or directory).
container-usbtty: libsemanage.semanage_pipe_data: Unable to execute /usr/libexec/selinux/hll/mod : No such file or directory
container-usbtty:  (No such file or directory).
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
semodule:  Failed!

I already needed to install checkmodule, as it wasn’t installed, using

$ rpm-ostree install checkpolicy
$ sudo rpm-ostree apply-live

mattdm · November 29, 2022, 11:32pm

Hey all — just wanted to note that I’ve seen your recent comments. Been busy lately and haven’t had time to update this.

I really like the quadlet concept, but I think maybe play kube is going to be the longer-term and more portable winner. Not sure, though. I might end up doing… both?

I’m running all of the containers under a hass user, so they’re rootless and I’m not using privileged for any of them currently. (This does mean that DHCP discovery doesn’t work. Haven’t tried to mess with the specific privs.) I’m not using esphome, but you shouldn’t need privs or root access for USB access as log as the user is in the dialout group (and see the selinux stuff in the first post).

Once upon a time when I first learned OpenShift stuff a few years ago, I was taught “1 container per pod is a the right choice, unless they are extremely coupled and share the same lifecycle”. Since these services are fairly loosely coupled and can (mostly) be updated and started/stopped/restarted independently, I chose to separate them. I’m open to revisiting that, especially if that particular bit of advice hasn’t really survived into modern Kubernetes practices. I try to keep up, but all of my hands-on stuff is small potatoes.

Not really sure — it’s just different, so I want to be sure I’m not missing anything.

Plusses and minuses — using high ports means it’s easier for a different non-root process (maybe even running as another user) to grab those ports. But using the low ports directly means you need root privs at some point.

DezeStijn · November 30, 2022, 11:14am

Hi @mattdm

I’ve read through this whole thread but I still get stuck at permissions for the Zigbee serial adapter.

I even tried readlink -f as suggested above.

/usr/bin/podman run \
    --cidfile=%t/%n.ctr-id \
    --cgroups=no-conmon \
    --rm \
    --sdnotify=conmon \
    --replace \
    --detach \
    --label "io.containers.autoupdate=registry" \
    --name=zigbee \
    --group-add keep-groups \
    --network=slirp4netns:allow_host_loopback=true \
    --device=$(readlink -f /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_ee6789cf0260ec11ba5a345f25bfaa52-if00-port0):/dev/zigbee:rw \
    --volume=/srv/hass/zigbee:/app/data:Z \
    -p 8081:8081 \
    docker.io/koenkk/zigbee2mqtt:latest

$ cat /srv/hass/zigbee/log/2022-11-30.10-59-24/log.txt
info  2022-11-30 10:59:24: Logging to console and directory: '/app/data/log/2022-11-30.10-59-24' filename: log.txt
info  2022-11-30 10:59:24: Starting Zigbee2MQTT version 1.28.2 (commit #360a777)
info  2022-11-30 10:59:24: Starting zigbee-herdsman (0.14.68)
error 2022-11-30 10:59:24: Error while starting zigbee-herdsman
error 2022-11-30 10:59:24: Failed to start zigbee
error 2022-11-30 10:59:24: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
error 2022-11-30 10:59:24: Exiting...
error 2022-11-30 10:59:24: Error: Error while opening serialport 'Error: Error: Permission denied, cannot open /dev/zigbee'
    at SerialPort.<anonymous> (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/znp/znp.ts:146:28)
    at SerialPort._error (/app/node_modules/@serialport/stream/lib/index.js:198:14)
    at /app/node_modules/@serialport/stream/lib/index.js:242:12

$ groups hass
hass : hass dialout

$ ls -al /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_ee6789cf0260ec11ba5a345f25bfaa52-if00-port0
lrwxrwxrwx. 1 root root 13 Nov 30 07:35 /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_ee6789cf0260ec11ba5a345f25bfaa52-if00-port0 -> ../../ttyUSB0

$ ls -al $(readlink -f /dev/serial/by-id/usb-ITead_Sonoff_Zigbee_3.0_USB_Dongle_Plus_ee6789cf0260ec11ba5a345f25bfaa52-if00-port0)
crw-rw----. 1 root dialout 188, 0 Nov 30 07:35 /dev/ttyUSB0

As mentioned before, I’m running Fedora CoreOS, should that make a difference. I’m a first-time Fedora user as well.

$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="36.20221030.3.0 (CoreOS)"
ID=fedora
VERSION_ID=36
VERSION_CODENAME=""
PLATFORM_ID="platform:f36"
PRETTY_NAME="Fedora CoreOS 36.20221030.3.0"

$ podman version
Client:       Podman Engine
Version:      4.3.0
API Version:  4.3.0
Go Version:   go1.18.7
Built:        Fri Oct 21 08:16:35 2022
OS/Arch:      linux/amd64

# as sudo-user
$ cat ~/container-usbtty.te
module container-usbtty 1.0;

require {
        type container_t;
        type usbtty_device_t;
        class chr_file { getattr ioctl lock open read write };
}

#============= container_t ==============
allow container_t usbtty_device_t:chr_file { getattr ioctl lock open read write };

$ checkmodule -M -m -o container-usbtty.mod container-usbtty.te
$ semodule_package -o container-usbtty.pp -m container-usbtty.mod
$ sudo semodule -i  container-usbtty.mod
libsemanage.semanage_pipe_data: Child process /usr/libexec/selinux/hll/mod failed with code: 1. (No such file or directory).
container-usbtty: libsemanage.semanage_pipe_data: Unable to execute /usr/libexec/selinux/hll/mod : No such file or directory
container-usbtty:  (No such file or directory).
libsemanage.semanage_direct_commit: Failed to compile hll files into cil files.
 (No such file or directory).
semodule:  Failed!