Do Z-Wave devices have to be installed to add ZWave integration? (Logs included - is this a bug?)

Bottom line: The ZWave integration will NOT install on a fresh HA system on a barebones Pi4.

(This is related to this topic but with more information and looking at it with a different point of view.)

Do I have to have a ZWave dongle on my system for the integration to install? I can’t get the integration to install at all on a new Pi4 system with no extra hardware.

More details:
I have a Pi4 that worked okay with ZWaveJS2MSQTT with an Aeotec ZStick 7. I changed that system to Home Assistant and added the ZWave integration with no problem. But, for various reasons, I wanted a fresh install of HA on that system. I write out a new HA image to an SD card, turned off the Pi4, then put in the new HA image and started it up. Whenever I added the ZWave integration, I’d get vague and useless errors. I’ve had problems, in the past, with the device nodes being unreadable in some programs, and it meant I had to write a udev rule to make sure the ZWave node in /dev was accessible to the program. (I think that happened to me a good while back in HA, when I first started it, but I’m not sure. I know it did happen to me last week with openHAB.)

That made me wonder if the dev node file could be an issue, so I pulled the USB ZStick and wrote a new HA image to a new SD card and put that in.

So this is a Pi4 with no extra hardware. It’s a brand new install of HA, updated, and it won’t install the ZWave integration. Note, also, that previously, I could install ZWave with no problem. So something must have happened to this Pi4 hardware that makes it suddenly unable to install this.

Here’s screenshots. I go to Settings->Integrations->Add Integration:


I pick the ZWave integration, the one that’s selected, the 2nd one in the list and, on a fresh system, I get this:

I continue with the box checked (which is what worked in the past and works on my other HA system). Then I get this:

I wait and watch. Then I get a flicker as a dialog box comes up, but disappears in an instant. It’s larger than the status box saying installation of the add-on has started, but I can’t see anything on it because it’s almost instantly replaced by this:

So I go back to the start and try installing again. I get the dialog asking if I’m using the ZWaveJS add-on again. I proceed as before and now, since the ZWaveJS add-on is installed, I get this:

So if I install fresh, I get “Error.” If I install with the ZWaveJS add-on already on my system, I get “Unknown Error Occurred.”

I have tried this with the USB Z-Stick on the system. No difference.

This appears to be the error message (I have log levels set to debug in configuration.yaml) and stack trace:


2022-09-17 16:43:59.050 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
resp = await request_handler(request)
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 82, in ban_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
result = await result
File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 169, in get
return await super().get(request, flow_id)
File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 96, in get
result = await self._flow_mgr.async_configure(flow_id)
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 280, in async_configure
result = await self._async_handle_step(
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 367, in _async_handle_step
result: FlowResult = await getattr(flow, method)(user_input)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 609, in async_step_configure_addon
ports = await async_get_usb_ports(self.hass)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 144, in async_get_usb_ports
return await hass.async_add_executor_job(get_usb_ports)
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 128, in get_usb_ports
usb_device = usb.usb_device_from_port(port)
File "/usr/src/homeassistant/homeassistant/components/usb/utils.py", line 13, in usb_device_from_port
vid=f"{hex(port.vid)[2:]:0>4}".upper(),
TypeError: 'NoneType' object cannot be interpreted as an integer

If I understand you correctly then yes, you have to have the hardware (USB stick, etc) actually connected to the machine for the integration to install. It has to establish the connection to the stick on installation.

it would be the same even if you had the add-on installed previously and then pulled the stick it would probably fail because the add-on/docker container would fault since it couldn’t communicate with the stick and HA integration would error due to the add-on error.

Why are you trying to install the integration without the stick being connected?

As I said in the post, I’ve had trouble with this integration. It installed on a new HA image last week, but won’t install this week, on a new image. Same hardware. And I’m getting the same results (except for, maybe, the stack trace in the logs - I’ll check that) whether the Z-Stick is connected or not.

I’m now thinking it’s possible that I used a previous Pi 64bit image when the integration installed properly.

I just want to clarify I’ve had this issue with multiple attempts of using a fresh HA image on my Pi4 with the Z-Stick 7 and now without the Z-Stick plugged in. I tried it again with the stick plugged in and the log seems to show the same issue. I haven’t compared the stack trace line by line, but the start and end look the same:


2022-09-17 17:22:13.917 ERROR (MainThread) [aiohttp.server] Error handling request
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_protocol.py", line 435, in _handle_request
resp = await request_handler(request)
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_app.py", line 504, in _handle
resp = await handler(request)
File "/usr/local/lib/python3.10/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 60, in security_filter_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/ban.py", line 82, in ban_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware
return await handler(request)
File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 136, in handle
result = await result
File "/usr/src/homeassistant/homeassistant/components/config/config_entries.py", line 169, in get
return await super().get(request, flow_id)
File "/usr/src/homeassistant/homeassistant/helpers/data_entry_flow.py", line 96, in get
result = await self._flow_mgr.async_configure(flow_id)
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 280, in async_configure
result = await self._async_handle_step(
File "/usr/src/homeassistant/homeassistant/data_entry_flow.py", line 367, in _async_handle_step
result: FlowResult = await getattr(flow, method)(user_input)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 609, in async_step_configure_addon
ports = await async_get_usb_ports(self.hass)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 144, in async_get_usb_ports
return await hass.async_add_executor_job(get_usb_ports)
File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/src/homeassistant/homeassistant/components/zwave_js/config_flow.py", line 128, in get_usb_ports
usb_device = usb.usb_device_from_port(port)
File "/usr/src/homeassistant/homeassistant/components/usb/utils.py", line 13, in usb_device_from_port
vid=f"{hex(port.vid)[2:]:0>4}".upper(),
TypeError: 'NoneType' object cannot be interpreted as an integer

I recommend you don’t do the auto supervisor add-on install.

install the zwavejs2mqtt add-on first and configure it.

then after you are sure the add-on is running and it recognizes the stick correctly then install the integration.

the zwavejs2mqtt gives you more troubleshooting ability since it provides a standalone control panel outside of the zwave integration.

maybe it will help you track down the issues.

Is that easy to setup? I’ve been dealing with this, connecting 2 HA instances, and a few other issues in the past few days and that’s taken up a lot of time and left me with very little sleep. If it’s easy, I can try it now, if it takes a lot of reading to know how to configure it, I need to wait.

That sounds good!

I’ve made an image from an older file. I was using 8.5 and it seems, now that I think about it, that the install where adding the integration went well may have been on 8.1. I checked and still have it, so I’m trying 8.1. If that works, I’ll save the image and test 9.0 also. If 9.0 has the same issues as 8.5, then I’ll bring it up as a bug.

But if 8.1 doesn’t work, I’ll try what you suggest. (It’s also a sanity check - is it possible it installed because it was a different HA source file? If it does, then I won’t feel so delusional as I will if it doesn’t work on that one either. If it doesn’t, I see no reason why it’d work before and not now on the same hardware.)


EDIT: Just tried 8.1 and it won’t add under that, either. Haven’t checked logs yet.

So I’ll be trying your suggestion now.

This is driving me crazy - it worked, I had it working with multiple devices not even a week ago. Now, same hardware, just a different install image of HA, and it won’t work. There’s no clear reason for this.

I don’t use add-ons but the standalone docker container version was pretty straightforward

I’m a bit confused. This is an add-on you’re suggesting. So are you just not using it (because it’s an add-on), or was it included on the docker version?

Just trying to get it straight in my head.

On the logs - same error with 8.1 as with 8.5. While the plan now is to use MQTT, like you suggest, I’m wondering if there is a bug or possibly even an issue with my Z-Stick having gone bad (which I consider unlikely). So I’m going to burn a few more 8.5 images and do a test install on the Pi4 where HA is working well and on a Pi3 to have a comparison. I’m tempted, if it installs on the good Pi, to just take that SD card and put it in the one where it’s having trouble to see what happens. All this is not so much about me, but a lot of it’s about trying to find out if there’s a bug.

It’s occurred to me, since it’s a NoneType issue in the stack trace, that without the stick involved, it can’t add a device and no device could cause that kind of error, so it might not work with it as well if the stick has gone bad.

I suggested the add-on since you are running a Supervised type installation. Because of that you have add-ons that provide similar functionality as the same stand-alone docker images available to those who aren’t running a supervised type install.

I run a HA Container install so I use just regular Docker images. The add-ons are based on those images with other stuff added for use with the supervisor.

Your HA also runs in a docker container in the background (along with the supervisor and all of the other several containers that make uip the HA ecosystem)

That means that you can also still run regular Docker images just like I can but it’s much harder to do for you since you don’t have access to the underlying OS.

So I am running HA Container (which runs in a Docker container itself). Then I run a zwavejs2mqtt Docker container and it handles the zwavejs stuff just like the add-on. Then I use the zwavejs integration to connect between HA and the zwavejs2mqtt container. And you do the same only you don’t “see” the Docker containers.

In your case you have two (or maybe three) options to run the zwavejs server - the official supervisor add-on or the unofficial zwavejs2mqtt add-on.

I almost can say for sure that it’s not a bug or lots of others would be seeing the same thing.

It is possible the stick is dead. It’s happened to others as well.

I’m not sure what the troubleshooting steps are except to try the zwavejs2mqtt add-on and see if the control panel it provides gives any further info.

EDIT:

I just noticed this:

Just to clarify you don’t need to use MQTT with the zwavejs2mqtt add-on.

you will configure it to use websockets and ignore MQTT (unless you really want use it).

Not quoting the rest of what you said about that, for brevity, but thank you for the background on it. Much appreciated.

I would think so, but sometimes there’s an odd configuration that triggers something. I think in terms of “Just in case.”

I did go to the ZWaveJS add-on, which the Integration installs automatically. In it, it gave me two options to use for a USB device in /dev for the stick, including the long path one to go to the device directly and ttyACM0 (or something close to that). So it makes me think that this add-on sees the stick. I’m going to try the zwavejs2mqtt add-on. I take it I install that, then, when adding the Z-Wave integration, I tell it NOT to use ZWaveJS and, instead, give it the port info on localhost to connect to the ZwaveJS2MQTT add-on.

I got it - I was in a hurry, getting ready to go out to dinner and a show with my wife - had to write the post quickly so we could leave on time! :wink:

I’ve tried installing the Z-Wave integration on another Pi, with a Z-Stick that was working before I booted up a new instance of HA. I got the same situation - same useless error messages, same stack trace.

I got curious and started poking around with ZWaveJS. The USB device was not set, so I set that. The auth keys weren’t set, but I saved and went elsewhere, then came back and it had generated its own auth keys. I found when I tried to save the configuration again, it asked me to restart the add-on. I did, and got this error:
Screen Shot 2022-09-18 at 2.22.21 AM

Considering I’m getting the same info in the logs on TWO Pis, I’m beginning to think this is a bug on the recent versions (8.5 and 9.0) of HA. It may not be impacting everyone, but it’s been a bear for me to deal with.

I’m going to try ZJ2M on this system. (I’m testing now - when I’m done testing on this system, I’ll put my normal USB stick with HA on it and go back to that instance.)

Experimented more. On the Pi with the issue, the one I started dealing with, I installed ZWaveJS2MQTT as an add-on, then tried to use it from the integration, but the system crashed on me. It’s down in the barn and it’s past 3 AM, but I do think I’ll get down there, real quick, to put a new SD card in and start from scratch.

I did experiment on the other Pi, as mentioned above. This other Pi has been working with HA on it and the ZWave integration working fine. In theory, I should be able to put a new HA image on it and easily install ZWave because it’s been working on that system.

I first kept the Z-Stick unplugged and, as you said, it wouldn’t install. So I plugged the Z-Stick in and it still wouldn’t install. The ZWaveJS add-on installs, but sometimes the error message I get is for a failure to connect. (I know you told me about that - I just kind of needed to see that happen for sure. After that, I deleted the ZWaveJS add-on so I could start all over.) As I mentioned above, since the Z-Stick on that Pi, the 2nd Pi, is functional, the ZWave integration should have installed without issue. Again, the ZWaveJS add-on installed without issue but the integration would not install.

I found if I try to install from scratch, it takes time, then I get an error message - only “Error.” If I have ZWaveJS on and try to install the integration, I get an error right away. But then I went to the ZWaveJS add-on and set it up, specifying the right USB device and providing auth keys for it. Then I saved the configuration, started the add-on, and tried to install the integration. I got a “Failed to connect” message. I’m not clear if it failed to connect to the add-on, or that the add-on didn’t connect to the Z-Stick. That’s making me think it’s a communications issue at some level.

I did install ZWaveJS2MQTT on the test system (not the one that crashed) and it installed. I couldn’t find a way to tell it what device to use, but with that add-on, I still couldn’t get the ZWave integration to install.

When I finished testing on the 2nd Pi, I shut it down and put my old Home Assistant image back on it and rebooted. It came up okay and the ZWave devices were still accessible.

I think it’s important to note that, on the 2nd system:

  • The Z-stick is good and was functioning before and after I ran tests
  • I had the SAME problems and error messages on the 2nd system as on the first and, on both, could not install the Z-Wave integration, even with a proven good Z-Stick.

Something is wrong here, since it does the same thing on both Pis and I’ve proven one has a good Z-Stick.

Ok, the only other thing I can think of seeing that first “already in use” error is that you might be trying to install two add-ons for zwavejs at the same time.

there can be only one zwavejs add-on accessing the zwave stick at a time (whether its the supervisor zwavejs, non-official zwavejs2mqtt or even a standalone zwavejs2mqtt docker image makes no difference). If more than one server add-on is installed they will both try to communicate with the stick and whichever one is first it blocks the other one and you will get stick access errors.

But in this case the error is referring to the add-on so it seems you tried to install another of the same type of add-on. Obviously you can’t do that either.

go to the add-ons section of HA and make sure there is only one instance of any form of zwavejs add-on there.

When I got that “already in use” error, I had only the ZWaveJS add-on installed. I had gone to the config page, picked what device to use, then saved the config and started it, then I went back and decided to up the logging level. When I did that, it said it had to restart the add-on for the changes to take effect. I clicked okay and when it tried to restart it, that’s when I got the error. At the time, that was the only add-on of any type that was installed.

And it still seems weird to me that I’m getting the same errors on TWO different Pis, when I’ve proven the Z-stick on the 2nd one is good and the integration still won’t install.

I think I’m kind if stuck now.

Not being there and seeing exactly what you see with all of the moving parts you’ve got going on it’s pretty hard to help any further.

I guess you could submit a bug report (but I doubt it is) and maybe someone can help you better there.

But you could try one more thing…

install a version 2022.8.x version of home assistant and a version 9.x of the zwavejs add-on.

I did hear that there was a communication issue with version 10 of zwavejs and some devices. I don’t think it prevented the zwavejs add-on from even working or the zwavejs integration from installing tho.

And just to be sure you understand that to run zwavejs on HA v2022.9.x or later you need to have zwavejs v10.x or later running.

That’s why I suggested to use two earlier versions of both.

Right now I have HA v2022.8.7 running and zwavejs server v9.6.2 running using the standalone zwavejs2mqtt image (v6.14.0).

Yeah, I’ve had that feeling for most of a week now. I kind of want to put this in a diagram, but that’s not as easy as I thought. I have two Pis, Pi-H (for House) and Pi-B (for Barn).

  • Pi-B had a working Z-Wave integration I installed a week or two ago.
  • Pi-H has a working Z-Wave integration. I installed it weeks ago and it installed without issue.
  • Installing NEW images of HA on Pi-B, without any hardware changes, gives errors.
  • Doing the same on Pi-H, also without hardware changes, gives the same errors.
  • After testing on Pi-H, I put the working HA instance, on a USB stick, back on Pi-H and it works, proving the Z-Stick is okay.

So what has changed between the time I could install the integration on both systems and now that I can’t? The only thing that has changed is the HA image.

The Pi images don’t have dates on them and, sadly, when I upgraded to a new HA image, I deleted the older ones. I have 8.1, 8.5. and 9.0 Pi images of HA. The tests I did last night are on 9.0. I don’t know how to translate the versions I have with the dated versions. I wish I could get an older version image, but it appears HA keeps rather tight control on version availability. I’d like to get a version before 8.1, since there’s a good chance that’s what I used previously that worked. If I could get earlier versions, I’d be willing to go through them, stepping back until I find one that works.

Also, I’ve been thinking about this. If it is a bug, the upgraded HA that’s working (All upgrades but the most recent) is okay, so it must only be an issue during setup and installing that integration.

Have you tried the 8.1/8.5 versions yet?

I wish I knew how to tell you to install a specific older version of HA OS but I’ve never used it so I’m not sure how you would find the image for sure.

EDIT: looking at the HA OS Github it looks like either 8.1 or 8.5 should be using the august version of HA (2022.8.x) so that one should work with the older version of the zwavejs add-on.

try one of those and see how it goes.

Adding this as general documentation. I’m trying a new install with version 9.0 of the Pi image. I installed it and did nothing with it other than add the Z-Wave integration. It installed the ZWaveJS add-on, but gave me just “Error” as a message for not being able to install the integration. I went to the add-on and selected the serial device and saved the configuration, then started the add-on. Tried to reinstall the integration again. Got a “Could not connect” error. (If I try to install the integration with the add-on stopped, I get a general error message.)

I went back to the integration, made sure it was running, then went to Config and updated logging to “silly.” Restarted it. Here’s the log, at the bottom.

Again, just adding for documentation. The add-ons (ZWaveJS and ZWaveJS2MQTT) install and seem to find the serial device without issue. The issue seems to be, to my uneducated eye, that the integration is not able to communicate with either add-on. While I have not yet checked the main log on this test install yet, in the past, every time, I get the same error for the integration - the one that includes the stack trace.

Yes. I was using 8.5 for almost everything, then went back to 8.1, thinking that may be the one I used earlier. But I am pretty sure I deleted one version at one point, so I may have started with one before that. I have an idea that MIGHT give me earlier versions - just replace the version number in the link to the image. Right now I’m thinking I may need to get earlier than the August version.

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/config.sh
[17:00:10] INFO: Both 'network_key' and 's0_legacy_key' are set and match. All ok.
[17:00:12] INFO: Virtual Machine not detected, enabling soft-reset
cont-init: info: /etc/cont-init.d/config.sh exited 0
cont-init: info: running /etc/cont-init.d/structure.sh
cont-init: info: /etc/cont-init.d/structure.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun zwave_js (no readiness notification)
s6-rc: info: service legacy-services successfully started
[17:00:13] INFO: Successfully send discovery information to Home Assistant.
It is highly recommended to set `predictableActionArguments` to `true` when using `createMachine`. https://xstate.js.org/docs/guides/actions.html
2022-09-18T21:00:15.871Z DRIVER   ███████╗ ██╗    ██╗  █████╗  ██╗   ██╗ ███████╗             ██╗ ███████╗
                                  ╚══███╔╝ ██║    ██║ ██╔══██╗ ██║   ██║ ██╔════╝             ██║ ██╔════╝
                                    ███╔╝  ██║ █╗ ██║ ███████║ ██║   ██║ █████╗   █████╗      ██║ ███████╗
                                   ███╔╝   ██║███╗██║ ██╔══██║ ╚██╗ ██╔╝ ██╔══╝   ╚════╝ ██   ██║ ╚════██║
                                  ███████╗ ╚███╔███╔╝ ██║  ██║  ╚████╔╝  ███████╗        ╚█████╔╝ ███████║
                                  ╚══════╝  ╚══╝╚══╝  ╚═╝  ╚═╝   ╚═══╝   ╚══════╝         ╚════╝  ╚══════╝
2022-09-18T21:00:15.878Z DRIVER   version 10.0.4
2022-09-18T21:00:15.879Z DRIVER   
2022-09-18T21:00:15.881Z DRIVER   starting driver...
2022-09-18T21:00:15.928Z DRIVER   opening serial port /dev/ttyAMA0
2022-09-18T21:00:15.955Z DRIVER   serial port opened
2022-09-18T21:00:15.958Z SERIAL » [NAK]                                                                   (0x15)

I found my old ZWave2JSMQTT setup I had been using on this Pi. (Remember, I don’t want to use that now, and want a full HA install so I can use Insteon and other devices as well.)

The ZWave devices all show up on this install (which is ZWave2JSMQTT running on Debian). So both devices are good and ZWave works on both Pis with their Z-Wave controllers. HA just can’t install it.

I’m not following what you mean here.

What is stopping you from using your previous setup?