Security Alert Update 2021.1.4 Update Broke My Zwave Integration

hackerkatt · January 20, 2021, 7:17pm

It may be coincidence, but It seems that the recent Security Update 2021.1.4 has broken my Zwave integration. The Zwave Stick is seen at the OS level but my hub stick, and then of course, my other devices, are unavailable. I had previously set udev rules so that the USB device was static. I’ve gone over my configuration elements but can’t seem to recover functionality. I’ll post my config elements below. Anybody have any ideas of what I may be missing or how to go about getting the integration restored?

Thank you

Installation

Supervisord Docker
Base OS: Ubuntu 20.04.1 LTS
Docker Version: 20.10.2, build 2291f61

Supervisor log repeated entries

21-01-20 17:49:53 INFO (SyncWorker_4) [supervisor.docker.addon] Starting Docker add-on homeassistant/amd64-addon-zwave with version 0.8.0
21-01-20 17:50:23 WARNING (MainThread) [supervisor.misc.tasks] Watchdog found a problem with core_zwave!

core log
I don’t know why I’m seeing this error in the core log. I can’t find anywhere in the OpenZWave add-on configuration options to configure MQTT credentials

2021-01-20 10:26:58 ERROR (MainThread) [openzwavemqtt] MQTT error: [Errno 111] Connection refused. Reconnecting in 4 seconds
2021-01-20 10:27:03 ERROR (MainThread) [openzwavemqtt] MQTT error: [Errno 111] Connection refused. Reconnecting in 8 seconds
2021-01-20 10:27:26 ERROR (MainThread) [openzwavemqtt] MQTT error: Disconnected during message iteration. Reconnecting in 2 seconds
2021-01-20 10:27:28 ERROR (MainThread) [openzwavemqtt] MQTT error: [Errno 111] Connection refused. Reconnecting in 4 seconds

configuration.yaml

zwave:
  debug: true
  usb_path: /dev/zwaveAeontecG5Stick
  network_key: !secret zwave_network_key

OZW_log.txt
empty

lsusb

Bus 001 Device 003: ID 0658:0200 Sigma Designs, Inc. Aeotec Z-Stick Gen5 (ZW090) - UZB

Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a86:e024 QinHeng Electronics 
Bus 001 Device 003: ID 0658:0200 Sigma Designs, Inc. Aeotec Z-Stick Gen5 (ZW090) - UZB
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

dmesg

services-admin@ubuntu:~/DockerServices$ dmesg | grep -i usb
...
[    1.238206] usb 1-1.3: new full-speed USB device number 3 using ehci-pci
[    1.348166] usb 1-1.3: New USB device found, idVendor=0658, idProduct=0200, bcdDevice= 0.00
[    1.348721] usb 1-1.3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
...

/etc/udev/rules.d/99-usb-serial.rules

services-admin@ubuntu:~/DockerServices$ sudo cat /etc/udev/rules.d/99-usb-serial.rules
[sudo] password for services-admin: 
SUBSYSTEM=="tty", ATTRS{idVendor}=="0658", ATTRS{idProduct}=="0200", SYMLINK+="zwaveAeontecG5Stick"

Developer Tools → Services → Service
No zwave service

Clicking “REFRESH NODE”
Selection_002

M0E-lnx · January 20, 2021, 7:19pm

What version were you upgrading from?

silvrr · January 20, 2021, 7:30pm

Do you have zwave in your configuration.yaml file and the open zwave addon running?

If so, they are likely fighting over your USB stick. You need to use the native Zwave component or the open zwave add-on and beta component, not both.

hackerkatt · January 20, 2021, 7:32pm

@M0E-lnx

The original installed version and what I upgraded from was 0.114.4

hackerkatt · January 20, 2021, 7:34pm

@silvrr

Yes, I have a zwave stanza in my configuration.yaml and OpenZWave add-on running. This was how it was configured prior to the update (which zwave hub and devices worked). What is the beta component?

zwave:
  debug: true
  usb_path: /dev/zwaveAeontecG5Stick
  network_key: !secret zwave_network_key

silvrr · January 20, 2021, 7:39pm

The first log you posted says you are running core_zwave which is the zwave add-on. This will try and use your zwave stick.

You also have zwave in your config file so its going to try and use the zwave stick. They are likely fighting with one another.

In your add-ons page, see if you have an add-on running named “OpenZwave”

In your integrations page see if you are running ‘Z-Wave’ or ‘OpenZWave (beta)’

hackerkatt · January 20, 2021, 8:13pm

@silvrr

Yes, I have both. My config did not change, it’s how it had been configured prior to the update. I’m not dismissing your point about competing services, just that it had been running that way prior.

So It sounds like I need to remove one. Which would I remove? Additionally, if I remove one, say the configuration.yaml entry, will it break my zwave devices that have/were already connected to the zwave network? Finally, the OpenZWave (beta), I’ve tried to remove but it keeps getting discovered.

hackerkatt · January 20, 2021, 8:52pm

Found what looks like another add-on that’s been broken by the update. I don’t want to muddy this post specific to my zwave integration, so I’m happy to create a new topic. But mention it here as an indication of further breakage. POOP!

The Check Home Assistant configuration Add-on now crashes and exits. When I start it, jump over to the Configuration -> Server Controls -> Configuration Validation and CHECK CONFIGURATION, it succeeds. Then shortly thereafter the add-on container exits and requires the add-on be started again.

[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] udev.sh: executing... 
starting version 3.2.9
[13:42:31] INFO: Update udev information
[cont-init.d] udev.sh: exited 0.
[cont-init.d] done.
[services.d] starting services
[services.d] done.
[13:42:31] INFO: Setup udev devices
[13:42:36] INFO: Don't worry, this temporary installation is not overwriting your current one.
[13:42:36] INFO: Installing Home Assistant: latest...
[13:42:36] INFO: Please be patient, this might take a few minutes...
ERROR: After October 2020 you may experience errors when installing or updating packages. This is because pip will change the way that it resolves dependency conflicts.
We recommend you use --use-feature=2020-resolver to test your packages with the new resolver before it becomes the default.
requests 2.25.0 requires idna<3,>=2.5, but you'll have idna 3.1 which is incompatible.
[13:43:06] INFO: Installed Home Assistant 2021.1.4
[13:43:06] INFO: Making a copy of your configuration for checking...
[13:43:06] INFO: Checking your configuration against this version...
[13:44:12] ERROR: The configuration check did not pass!
[13:44:12] ERROR: See the output below for more details.
Testing configuration at /tmp/config
Failed config
  General Errors: 
    - Platform error media_player.vlc_remote - No module named 'xmltodict'
Successful config (partial)
[13:44:12] INFO: The full output has been written to /share/check_config.txt
[cont-finish.d] executing container finish scripts...
[cont-finish.d] done.
[s6-finish] waiting for services.
[s6-finish] sending all processes the TERM signal.
[s6-finish] sending all processes the KILL signal and exiting.

Selection_007

media_player:
  - platform: vlc_remote
    host: 10.40.0.177
    port: 8080
    name: "HTPC VLC" # Optional
    password: "<obfuscated>"

silvrr · January 20, 2021, 8:59pm

The restart likely triggered things to be updated. You could have stopped the zwave network and moved to zwave beta and ran without issue until the restart.

Yes.

If you want to run the beta, remove the config entry and restart.

If you want to want to run the non-beta component. Stop the add-on and remove.

That is up to you, do you want to be on the beta or on the non-beta component.

This is due to the add-on running.

hackerkatt · January 20, 2021, 9:15pm

@silvrr

Thanks for your continued consultation!

Frankly, this is a bit confusing…

In case I understand you correctly, I didn’t install the OpenZWave (beta) after I updated the core. It’s was installed all along prior to the core update. So I don’t know that I “could have stopped the zwave network and moved to zwave beta and ran without issue until the restart.”

The add-on does not say anywhere that it is beta. Nor does it state beta following the home-assistant/addons repo link

Visit OpenZWave page for details

So I’m confused as to which. I don’t want to nor never intended to run any beta. So, do I understand correctly that if I don’t want to run the beta, remove the addon even though it does not state beta? And leave my zwave conf in the configuration.yaml file?

Thanks

silvrr · January 20, 2021, 9:40pm

I don’t believe the add-on will say beta, the integration does. If you don’t want to run the beta, un-install the add-on and things should go back to normal.

I haven’t heard of add-ons installing on their own and there hasn’t been anything recently to make me think that the add-on would have been pushed out. You may have installed it previously, never started it or have stopped it but when you restarted, it automatically started due to the ‘start on boot’ setting in the add-on being selected.

Correct. The integration pop-up for open zwave beta should also stop being auto-discovered.

Note that you don’t need to remove the “Z-wave” integration, that is the one you want. In short, you shouldn’t need to do anything on your integrations page, remove the add-on and restart should take care of it.

hackerkatt · January 21, 2021, 12:14am

@silvrr

I haven’t heard of add-ons installing on their own and there hasn’t been anything recently to make me think that the add-on would have been pushed out. You may have installed it previously, never started it or have stopped it but when you restarted, it automatically started due to the ‘start on boot’ setting in the add-on being selected.

Yes, I had installed it. It was installed and running prior to the core update. And everything was running fine… until the core update.

I removed the OpenZWave addon. The OpenZWave (beta) integration disappeared, as expected. I left the configuration.yaml zwave config stanza.

I’m still unable to get the hub and devices to come available. This exercise have consumed far to much time today, much more than it should have. And still not resolved.

finity · January 21, 2021, 6:48pm

Have you completely rebooted the HA machine that the stick is attached to?

Not a HA restart but a full reboot?

hackerkatt · January 21, 2021, 6:55pm

@finity

I have, multiple times. I’ve also ensured that the /usr/sbin/hassio-supervisor script that starts the docker container has a --device option added to ensure that the device gets passed to the container. But the container still doesn’t see the serial device following the core update.

runSupervisor() {
    docker rm --force hassio_supervisor || true

    # shellcheck disable=SC2086
    docker run --name hassio_supervisor \
        --privileged \
        --security-opt apparmor=hassio-supervisor \
        --security-opt seccomp=unconfined \
        --device=/dev/zwaveStick \
        -v /run/docker.sock:/run/docker.sock \
        -v /run/dbus:/run/dbus \
        -v /etc/machine-id:/etc/machine-id:ro \
        -v "${HASSIO_DATA}":/data:rw \
        -e SUPERVISOR_SHARE="${HASSIO_DATA}" \
        -e SUPERVISOR_NAME=hassio_supervisor \
        -e SUPERVISOR_MACHINE="${MACHINE}" \
        "${SUPERVISOR}" 
}

finity · January 21, 2021, 7:00pm

I’m not familiar with that docker run command but this:

--device=/dev/zwaveStick

isn’t the same as the label you gave the stick in the UDEV rules above.

Also just as a double check does the zstick show up in the Supervisor->System->Host->hardware list?

What system are you running HAOS or HA Supervised?

hackerkatt · January 21, 2021, 7:08pm

This is a docker run option to directly expose/pass the host device to the docker container.
https://docs.docker.com/engine/reference/commandline/run/#add-host-device-to-container—device

And you are correct sir, it isn’t the same. Sorry for the confusion. As I continued to troubleshoot the issue, I shortened the name in my UDEV rules to create a static symlinked mapping to the USB device, and in my configuration.yaml file. I also removed the OpenZWave (beta) add-on at the suggestion of @silvrr.

It does not show up in my Hardware list in Supervisor -> System -> Host -> Hardware

I’m running Supervisord and just upgraded to core-2021.1.4 at the behest of the Security Bulletin advisory… which is what broke my system.

udev rule

SUBSYSTEM=="tty", ATTRS{idVendor}=="0658", ATTRS{idProduct}=="0200", SYMLINK+="zwaveStick"

...
zwave:
   debug: true
   usb_path: /dev/zwaveStick
   network_key: !secret zwave_network_key
...

services-admin@ubuntu: ~/DockerServices$ lsusb
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 1a86:e024 QinHeng Electronics 
Bus 001 Device 003: ID 0658:0200 Sigma Designs, Inc. Aeotec Z-Stick Gen5 (ZW090) - UZB
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

freshcoast · January 21, 2021, 9:18pm

Have you tried setting your zwave configuration to using the default persistent path? I didn’t think Supervisor allowed arbitrary udev symlink paths, so I’m surprised it ever worked. It does explicitly allow /dev/serial/by-id symlinks.

zwave:
   debug: true
   usb_path: /dev/serial/by-id/usb-0658_0200-if00
   network_key: !secret zwave_network_key

phunkymonkey · January 22, 2021, 3:15am

Exact same issue here. Running the OVA version of HAOS in VMWare Fusion on a Mac Pro.

As of the last supervisor update, the USB Z-wave controller is no longer getting passed in to the core container on startup. I also did most of the troubleshooting mentioned above, to no avail. My workaround has been to physically unplug/replug the device, which causes it to show up in the Supervisor, then restart the core. That works for a bit, but it will typically fail within a day.

finity · January 22, 2021, 4:03am

What are all the hardware options listed there?

hackerkatt · January 22, 2021, 12:35pm

@freshcoast

No, I had not. I do know that you can explicitly pass host devices to a container where the container doesn’t use the --privileged run option. In this case the hassio-supervisor run script does use the --privileged option in the run command and therefore gives elevated privileges to access the host’s hardware. Since the device was not showing up in my HA Hardware list, I added the option to the startup script, but no worky.

I used the UDEV Rules prior to statically map the USB device and that worked fine. I did so in my case because I was also seeing posts regarding issues with the Wyze Hub, which I also have. So the UDEV Rules route resolved any issues around either of the USB devices and it was working fine.

I looked for /dev/serial/by-id devices and I have a /dev/serial but no by-id. A Google search revealed that others have reported no /dev/serial/by-id. A review of the kernel documentation seems to indicate that this is because the Aeontec isn’t a recognized device having a driver built into the kernel. Another post suggested:

In general, lsusb tells you the hardware is there. This is the low level view.
Showing up in /dev means the hardware was recognized by the kernel and the drivers for it were loaded and the device is available for use.
It is not uncommon to plug in some USB device - say a wireless “dongle” that is not supported by the current kernel - and have it show up in lsusb, but not in /dev.

All I know is that the Aeontec device was working and since the core update not working.