Work in progress: configuration for running a Home Assistant in containers with systemd and podman on Fedora IoT

there is no ~/.config/containers

I nuked ~/.local/share/containers as well and now am pulling… the CNI error went away.

Now I have this:

Error: writing blob: adding layer with blob "sha256:764d2e53e1a607f2d8261522185d5b9021ade3ec1a595664ee90308c00176899": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid: lchown /etc/shadow: invalid argument

Does this require some sysctl setting?

This might be an issue with raspian and podman. What’s in /etc/subuid and /etc/subgid? If it isn’t there, try adding a line like this to each of them:

homeassistant:165536:65536

ok, added that line, but didn’t change anything.

homeassistant@gluster2:~ $ cat /etc/subuid /etc/subgid
pi:100000:65536
homeassistant:165536:65536
pi:100000:65536
homeassistant:165536:65536

but after pull:

Error: writing blob: adding layer with blob "sha256:764d2e53e1a607f2d8261522185d5b9021ade3ec1a595664ee90308c00176899": Error processing tar file(exit status 1): potentially insufficient UIDs or GIDs available in user namespace (requested 0:42 for /etc/shadow): Check /etc/subuid and /etc/subgid: lchown /etc/shadow: invalid argument

do the entries for pi conflict with the homeassistant entries?

gave me the idea to see if podman was running, and yes it was, so I killed it and re-pulled… lets see if that helps.

so now it’s definitely behaving differently… it’s hanging at:

...
Copying blob 0ef421e49690 done  
Copying blob 003b90cad2bf done  
Copying blob af7f205e2c78 done  
Copying blob 45390663e2a0 done  
Copying blob c5752038fd5a done  

and htop shows podman is using about 5-10% of the CPU

there’s also something called “storage-untar” so I’ll just let it run for a while.

and it finished!

Writing manifest to image destination
Storing signatures
4929cb3c289162ae483f8a68761b0c395a389871e6a13ce0985d63c1daadec29
1 Like

Nope — the second number is how many IDs are allocated, so pi goes from 100000 to 165535, and homeassistant starts right after that at 165536 (to 231071).

there’s also something called “storage-untar” so I’ll just let it run for a while.

Yeah it might just be slow depending on your RPi’s hardware.

Ok, so I did

systemd --user run container-homeassistant

which ran…

now I have no idea what’s going on. it doesn’t respond at http://gluster2.lan:8123 which is the place i expect it to be.

hey, crap I restarted it and now it’s there!

however, I can’t load the config page, or actually control any devices :wink:
I maybe need to move my old config out of the way, and try with a blank config.

1 Like

ok, I think we’re making progress… it needs to pull the other containers, and there’s an issue:

homeassistant@gluster2:~ $ podman pull zwavejs/zwavejs2mqtt:latest
Error: short-name "zwavejs/zwavejs2mqtt:latest" did not resolve to an alias and no unqualified-search registries are defined in "/etc/containers/registries.conf"

either I need a long form name, or define a registry?

I think it’d be better to fully qualify the name. what are the fully qualified names for the containers?

1 Like

Hmmm. Maybe the raspbian config being different from mine — it might not default to searching the Docker registry. Try podman pull docker.io/zwavejs/zwavejs2mqtt:latest.

Assuming that helps, I’ll edit the registry name into the files above.

1 Like

That is working!

1 Like

some stuff is starting, but I can’t get mosquitto to start, because of that the zigbee and zwave to mqtt containers don’t start.

     Loaded: loaded (/var/local/homeassistant/.config/systemd/user/container-mosquitto.service; disabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Wed 2022-03-09 11:16:19 PST; 11s ago
    Process: 1840 ExecStartPre=/bin/rm -f /run/user/998/container-mosquitto.service.ctr-id (code=exited, status=0/SUCCESS)
    Process: 1841 ExecStart=/usr/bin/podman run --cidfile=/run/user/998/container-mosquitto.service.ctr-id --cgroups=no-conmon --rm --sdnotify=conmon -->
    Process: 1902 ExecStopPost=/usr/bin/podman rm -f --ignore --cidfile=/run/user/998/container-mosquitto.service.ctr-id (code=exited, status=0/SUCCESS)
   Main PID: 1841 (code=exited, status=126)
        CPU: 846ms

Figured it out! I was running a system instance of mosquitto, stopped that and now it started.

Exit code 126 from podman (according to man podman) means that the there was a problem executing the command in the container. I’m poking at it and am not quite sure how to get mosquitto to give more helpful output…

it was the systemwide mosquitto daemon already running. So I got everything running. but while onboarding my new homeassistant, it keeps dying and restarting the containers.

Mar  9 12:02:54 gluster2 systemd[5888]: container-zigbee.service: Failed with result 'exit-code'.
Mar  9 12:02:54 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.100s CPU time.
Mar  9 12:03:01 gluster2 systemd[5888]: container-homeassistant.service: Scheduled restart job, restart counter is at 4.
Mar  9 12:03:01 gluster2 systemd[5888]: Stopped Home Assistant Container.
Mar  9 12:03:01 gluster2 systemd[5888]: container-homeassistant.service: Consumed 9.995s CPU time.
Mar  9 12:03:24 gluster2 systemd[5888]: container-zigbee.service: Scheduled restart job, restart counter is at 18.
Mar  9 12:03:24 gluster2 systemd[5888]: Stopped Zigbee To MQTT Container.
Mar  9 12:03:24 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.100s CPU time.
Mar  9 12:03:24 gluster2 systemd[5888]: Starting Zigbee To MQTT Container...
Mar  9 12:03:25 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:03:25 gluster2 systemd[5888]: Started Zigbee To MQTT Container.
Mar  9 12:03:25 gluster2 systemd[5888]: Starting Home Assistant Container...
Mar  9 12:03:25 gluster2 podman[19767]: b9807e462293f5bc2bb9226c4f3d1cd2fc471188f6bede8c123fdc546fda1941
Mar  9 12:03:25 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:03:25 gluster2 systemd[5888]: Started Home Assistant Container.
Mar  9 12:03:25 gluster2 podman[19831]: df09f44ed6637dba24f6a332f981550d6c8bb3f73c037561aba8f2fbd84090bd
Mar  9 12:03:50 gluster2 systemd[5888]: libpod-b9807e462293f5bc2bb9226c4f3d1cd2fc471188f6bede8c123fdc546fda1941.scope: Consumed 6.057s CPU time.
Mar  9 12:03:51 gluster2 systemd[5888]: container-zigbee.service: Main process exited, code=exited, status=1/FAILURE
Mar  9 12:03:51 gluster2 systemd[5888]: container-zigbee.service: Failed with result 'exit-code'.
Mar  9 12:03:51 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.058s CPU time.
Mar  9 12:04:19 gluster2 sssd_pam[13542]: Shutting down (status = 0)
Mar  9 12:04:19 gluster2 systemd[1]: sssd-pam.service: Deactivated successfully.
Mar  9 12:04:21 gluster2 systemd[5888]: container-zigbee.service: Scheduled restart job, restart counter is at 19.
Mar  9 12:04:21 gluster2 systemd[5888]: Stopped Zigbee To MQTT Container.
Mar  9 12:04:21 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.058s CPU time.
Mar  9 12:04:21 gluster2 systemd[5888]: Starting Zigbee To MQTT Container...
Mar  9 12:04:22 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:04:22 gluster2 systemd[5888]: Started Zigbee To MQTT Container.
Mar  9 12:04:22 gluster2 podman[20197]: 1ef17df12e531c8257ae2abcce3e77faf8552966f0185416090c0ae5791c7d55
Mar  9 12:04:25 gluster2 systemd[5888]: container-homeassistant.service: Watchdog timeout (limit 1min)!
Mar  9 12:04:25 gluster2 systemd[5888]: container-homeassistant.service: Killing process 19876 (conmon) with signal SIGABRT.
Mar  9 12:04:25 gluster2 systemd[5888]: container-homeassistant.service: Killing process 19874 (fuse-overlayfs) with signal SIGABRT.
Mar  9 12:04:25 gluster2 systemd[5888]: container-homeassistant.service: Main process exited, code=killed, status=6/ABRT
Mar  9 12:04:26 gluster2 systemd[5888]: libpod-df09f44ed6637dba24f6a332f981550d6c8bb3f73c037561aba8f2fbd84090bd.scope: Consumed 16.033s CPU time.
Mar  9 12:04:26 gluster2 podman[20275]: df09f44ed6637dba24f6a332f981550d6c8bb3f73c037561aba8f2fbd84090bd
Mar  9 12:04:26 gluster2 systemd[5888]: container-homeassistant.service: Failed with result 'watchdog'.
Mar  9 12:04:26 gluster2 systemd[5888]: container-homeassistant.service: Consumed 9.813s CPU time.

Not sure what the watchdog is supposed to be doing, but it keeps killing homeassistant

I’m following your instructions regarding conmon:

# note: this is using https://github.com/brianegge/home-assistant-sdnotify.
# If not using that, remove this and change --sdnotify=container to
# --sdnotify=conmon
WatchdogSec=60
ExecStartPre=/bin/rm -f %t/%n.ctr-id
ExecStart=/usr/bin/podman run \
                          --cidfile=%t/%n.ctr-id \
                          --cgroups=no-conmon \
                          --rm \
                          --sdnotify=conmon \
                          --replace \
                          --detach \
                          --label "io.containers.autoupdate=registry" \
                          --name=homeassistant \
                          --volume=/var/local/homeassistant/.homeassistant:/config:Z \
                          --network=host \
                          ghcr.io/home-assistant/home-assistant:stable

because I’m not doing anything with home-assistant-sdnotify but maybe that’s not right. I have no idea what should be there. I haven’t done anything to enable home-assistant-sdnotify but maybe it’s built in?

Ok, I set it back to what you had with --sdnotify=container and homeassistant seems to be staying up… but zwave and zigbee are both constantly rebooting.

actually I take it back… stuff is still going up and down like a yoyo

Not sure what the watchdog is supposed to be doing, but it keeps killing homeassistant

Oh, that is what it’s supposed to be doing. :slight_smile:

Basically, if the home-assistant-sdnotify integration is installed, it sends a signal periodically saying “I’m still alive”. If it doesn’t happen, the container shuts down.

Quick answer: remove the WatchdogSec=60 line. I’ll make that more clear in the comment!

ok, i commented out the watchdog…

Things are still going up and down… but it looks like it’s driven by zigbee:

Mar  9 12:23:14 gluster2 systemd[5888]: Stopped ZWave To MQTT Container.
Mar  9 12:23:14 gluster2 systemd[5888]: container-zwave.service: Consumed 3.572s CPU time.
Mar  9 12:23:14 gluster2 systemd[5888]: Starting ZWave To MQTT Container...
Mar  9 12:23:14 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:23:14 gluster2 systemd[5888]: Started ZWave To MQTT Container.
Mar  9 12:23:14 gluster2 systemd[5888]: Starting Home Assistant Container...
Mar  9 12:23:14 gluster2 podman[28858]: 8b7615f30bb6e90a5cbdbd8728eff092bd41d6673712f56b9c0c832d81f31ad0
Mar  9 12:23:15 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:23:15 gluster2 podman[28924]: d10d30da1be752c71151ad046fa02247c65de30768b8a3cc402b70fcb3931996
Mar  9 12:23:40 gluster2 systemd[5888]: libpod-a83640d542f94a808c98abaf7ecdecd8a90a793bc30b89067f1c4badab591f14.scope: Consumed 6.834s CPU time.
Mar  9 12:23:41 gluster2 systemd[5888]: container-zigbee.service: Main process exited, code=exited, status=1/FAILURE
Mar  9 12:23:41 gluster2 systemd[5888]: container-zigbee.service: Failed with result 'exit-code'.
Mar  9 12:23:41 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.154s CPU time.
Mar  9 12:24:11 gluster2 systemd[5888]: container-zigbee.service: Scheduled restart job, restart counter is at 39.
Mar  9 12:24:11 gluster2 systemd[5888]: Stopped Zigbee To MQTT Container.
Mar  9 12:24:11 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.154s CPU time.
Mar  9 12:24:11 gluster2 systemd[5888]: Starting Zigbee To MQTT Container...
Mar  9 12:24:12 gluster2 systemd[5888]: Started libcrun container.
Mar  9 12:24:12 gluster2 systemd[5888]: Started Zigbee To MQTT Container.
Mar  9 12:24:12 gluster2 podman[29291]: 61d97d5fd307f454e798e4988df30a6cb24f8546dec97da99346fcba4cf70686
Mar  9 12:24:38 gluster2 systemd[5888]: libpod-61d97d5fd307f454e798e4988df30a6cb24f8546dec97da99346fcba4cf70686.scope: Consumed 5.900s CPU time.
Mar  9 12:24:38 gluster2 systemd[5888]: container-zigbee.service: Main process exited, code=exited, status=1/FAILURE
Mar  9 12:24:38 gluster2 systemd[5888]: container-zigbee.service: Failed with result 'exit-code'.
Mar  9 12:24:38 gluster2 systemd[5888]: container-zigbee.service: Consumed 2.064s CPU time.
Mar  9 12:24:44 gluster2 systemd[5888]: container-homeassistant.service: start operation timed out. Terminating.
Mar  9 12:24:45 gluster2 systemd[5888]: libpod-d10d30da1be752c71151ad046fa02247c65de30768b8a3cc402b70fcb3931996.scope: Consumed 18.337s CPU time.
Mar  9 12:24:47 gluster2 systemd[5888]: container-homeassistant.service: Main process exited, code=exited, status=111/n/a
Mar  9 12:24:47 gluster2 systemd[5888]: container-homeassistant.service: Failed with result 'timeout'.
Mar  9 12:24:47 gluster2 systemd[5888]: Failed to start Home Assistant Container.
Mar  9 12:24:47 gluster2 systemd[5888]: container-homeassistant.service: Consumed 10.909s CPU time.

going to do a reboot, just to see if that fixes anything.

Ok, rebooted, and had to kill off the older homeassistant… then start the containers… and zigbee2mqtt doesn’t want to start:

debug 2022-03-09 20:46:47: Can't load state from file /app/data/state.json (doesn't exist)
info  2022-03-09 20:46:47: Logging to console and directory: '/app/data/log/2022-03-09.20-46-44' filename: log.txt
debug 2022-03-09 20:46:47: Removing old log directory '/app/data/log/2022-03-09.20-36-48'
info  2022-03-09 20:46:47: Starting Zigbee2MQTT version 1.24.0 (commit #7a2ddf2)
info  2022-03-09 20:46:47: Starting zigbee-herdsman (0.14.20)
debug 2022-03-09 20:46:47: Using zigbee-herdsman with settings: '{"adapter":{"concurrent":null,"delay":null,"disableLED":false},"backupPath":"/app/data/coordinator_backup.json","databaseBackupPath":"/app/data/database.db.backup","databasePath":"/app/data/database.db","network":{"channelList":[11],"extendedPanID":[221,221,221,221,221,221,221,221],"networkKey":"HIDDEN","panID":6754},"serialPort":{"path":"/dev/zigbee"}}'
error 2022-03-09 20:47:08: Error while starting zigbee-herdsman
error 2022-03-09 20:47:08: Failed to start zigbee
error 2022-03-09 20:47:08: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start.html for possible solutions
error 2022-03-09 20:47:08: Exiting...
error 2022-03-09 20:47:08: Error: Failed to connect to the adapter (Error: SRSP - SYS - ping after 6000ms)
    at ZStackAdapter.start (/app/node_modules/zigbee-herdsman/src/adapter/z-stack/adapter/zStackAdapter.ts:102:27)
    at Controller.start (/app/node_modules/zigbee-herdsman/src/controller/controller.ts:123:29)
    at Zigbee.start (/app/lib/zigbee.ts:58:27)
    at Controller.start (/app/lib/controller.ts:100:27)
    at start (/app/index.js:101:5)

the homeassistant user is in the dialout group:

homeassistant@gluster2:~/srv/zigbee $ groups
homeassistant dialout gpio i2c

and /dev/zigbee is set up:

ls -l /dev/zigbee 
lrwxrwxrwx 1 root root 7 Mar  9 12:39 /dev/zigbee -> ttyACM0
ls -l /dev/ttyACM0
crw-rw---- 1 root dialout 166, 0 Mar  9 12:49 /dev/ttyACM0

THis is left over from before:

cat  /etc/udev/rules.d/10-zigbezwave.rules 
ACTION=="add", ATTRS{idVendor}=="1cf1", ATTRS{idProduct}=="0030", SYMLINK+="zigbee"

ACTION=="add", ATTRS{idVendor}=="0658", ATTRS{idProduct}=="0200", SYMLINK+="zwave"

Should I get rid of that? It might be confusing the container?

I don’t think the rules will hurt. What do you have for --device on the podman command line? For me, it’s

--device=/dev/serial/by-id/usb-1a86_TubesZB_971207DO-if00-port0:/dev/zigbee:rw \

But you should change /dev/serial/by-id/usb-1a86_TubesZB_971207DO-if00-port0 to the path to your Zigbee controller. This is basically giving you the same advantages of the udev rule without having to change the RPI os system config.

Note that you can debug the zigbee service separately – just start that container (which should auto-start the mosquitto one, too) without Home Assistant, and the web interface should be available on http://192.168.1.10:8092 (replacing the IP address with the one for your Pi, of course.)

I’ve got:

 --device=/dev/serial/by-id/usb-dresden_elektronik_ingenieurtechnik_GmbH_ConBee_II_DE2225777-if00:/dev/zigbee:rw \

When I start the zigbee container it goes up and then dies over and over, can’t connect to the device

this /dev/serial/by-id/... is a symbolic link, not the actual device, does it work like that for you?

Yeah, it’s supposed to be a symlink. If you run

podman exec -it zigbee /bin/sh -c 'stat /dev/zigbee'

in the 60 seconds before it falls over, what do you get? Should be something like:

  File: /dev/zigbee
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: 5h/5d	Inode: 744         Links: 1     Device type: bc,1
Access: (0660/crw-rw----)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Access: 2022-03-09 21:21:32.000000000
Modify: 2022-03-09 21:21:32.000000000
Change: 2022-03-09 16:05:56.000000000

podman exec -it zigbee /bin/sh -c "stat /dev/zigbee"
  File: /dev/zigbee
  Size: 0         	Blocks: 0          IO Block: 4096   character special file
Device: 5h/5d	Inode: 262         Links: 1     Device type: a6,0
Access: (0660/crw-rw----)  Uid: (65534/  nobody)   Gid: (65534/  nobody)
Access: 2022-03-09 20:57:35.000000000
Modify: 2022-03-09 21:22:21.000000000
Change: 2022-03-09 20:57:35.000000000

Huh, I expected that to not show Device: 5h/5d. Okay, next:

podman exec -it zigbee /bin/sh -c 'stty -a -F /dev/zigbee'

Should give you a bunch of stuff, the first line of which is

speed 115200 baud;stty: /dev/zigbee
 podman exec -it zigbee /bin/sh -c 'stty -a -F /dev/zigbee'
speed 115200 baud;stty: /dev/zigbee
 line = 0;
intr = ^C; quit = ^\; erase = ^?; kill = ^U; eof = ^D; eol = <undef>; eol2 = <undef>; swtch = <undef>; start = ^Q; stop = ^S; susp = ^Z; rprnt = ^R; werase = ^W;
lnext = ^V; flush = ^O; min = 1; time = 0;
-parenb -parodd -cmspar cs8 hupcl -cstopb cread clocal -crtscts
-ignbrk -brkint ignpar -parmrk -inpck -istrip -inlcr -igncr -icrnl -ixon -ixoff -iuclc -ixany -imaxbel -iutf8
-opost -olcuc -ocrnl -onlcr -onocr -onlret -ofill -ofdel nl0 cr0 tab0 bs0 vt0 ff0
-isig -icanon -iexten -echo -echoe -echok -echonl -noflsh -xcase -tostop -echoprt -echoctl -echoke -flusho -extproc