Not sure if it makes a difference, but are you running the latest firmware for the switch?
Ah this could be it. I have firmware 1.35. Rookie question here - do you know how to update the firmware in Zwave JS, if it is possible? I cannot find a way to access the XML file manually to make changes.
Sorry…confusing the firmware and configuration files. Looks like I just need to up-pair the device and go through the Z-wave PC controller.
Unfortunately I dont believe you can update firmware with just zwavejs. I couldn’t find it in the documentation or going through the screens.
I use zwavejs2mqtt and if you use zwavejs2mqtt instead of zwavejs it includes a control panel where you can update the firmware that way.
If you don’t want to go to zwavejs2mqtt, you can leave the device paired and just plug the zstick into a windows computer using the silicon labs program to update the firmware. If paired unsecure it will recognize it still without removing the device from the stick. Follow the instructions on the Inovelli website.
Got the firmware updated. I think I have narrowed it down to an issue with security. I want to add that when I add the node via the ZWave integration (WITHOUT using secure inclusion), I get the following error in the logs:
2021-09-08T00:06:12.951Z DRIVER Unexpected error during SerialAPI command: ZWaveError: The network key for the
security class S2_Unauthenticated has not been set up yet!
at SecurityManager2.getKeysForSecurityClass (/usr/src/node_modules/@zwave-
js/core/src/security/Manager2.ts:117:10)
at SecurityManager2.initializeSPAN (/usr/src/node_modules/@zwave-js/core/s
rc/security/Manager2.ts:220:21)
at Security2CCMessageEncapsulation.serialize (/usr/src/node_modules/zwave-
js/src/lib/commandclass/Security2CC.ts:932:34)
at SendDataRequest.serialize (/usr/src/node_modules/zwave-js/src/lib/contr
oller/SendDataMessages.ts:99:37)
at getSerialAPICommandMachineConfig (/usr/src/node_modules/zwave-js/src/li
b/driver/SerialAPICommandMachine.ts:143:18)
at Object.createSerialAPICommandMachine (/usr/src/node_modules/zwave-js/sr
c/lib/driver/SerialAPICommandMachine.ts:385:3)
at executeSerialAPICommand (/usr/src/node_modules/zwave-js/src/lib/driver/
CommandQueueMachine.ts:226:14)
at Interpreter.exec (/usr/src/node_modules/xstate/lib/interpreter.js:844:6
1)
at Interpreter.execute (/usr/src/node_modules/xstate/lib/interpreter.js:20
3:14)
at Interpreter.update (/usr/src/node_modules/xstate/lib/interpreter.js:229
:12)
I have a 32-digit network key in the settings of the Open ZWave add-on configuration. I’m not sure exactly what is causing this error. I cannot seem to find online with this error log. Now feeling stuck again.
Finally got it connected with multiple attempts, and back at square one. It does now show the updated firmware version at 1.45 for LZW31, but still not reporting the state of the light correctly and giving the same error:
2021-09-08T00:22:58.319Z SERIAL » [ACK] (0x06)
2021-09-08T00:22:58.324Z DRIVER « [Node 041] [REQ] [ApplicationCommand]
└─[MultilevelSwitchCCReport]
current value: undefined
And to confirm, when you use zwavejs to include, you have “use secure inclusion” toggled off?
Also, are you on the absolute latest home assistant 2021.9.4? And latest zwavejs? I see a recent PR that might have fixed this issue. https://github.com/home-assistant-libs/zwave-js-server-python/pull/289
Yes - confirming the secure inclusion is toggled off and I am on the latest version of both core and supervisor.
It looks like your device is including as s2_unauthenticated even though you have secure inclusion toggled off, which it shouldn’t be doing. Then it’s not getting the updates to the controller due to the issue about not deserializing commands that I linked above.
I’m also assuming it’s limited to the black series switches. I just excluded a red series switch and included it with no issues, and it has no security and status updates fine.
At this point I would file a github issue and include all relevant logs and info.
1 Like
Interestingly I uninstalled and did a complete reboot of the ZWave JS add-on in the supervisor so that I could take a look at the log outputs. One of the first log errors is:
[services.d] starting services
[services.d] done.
[18:43:24] INFO: Successfully send discovery information to Home Assistant.
ThenetworkKeyoption is deprecated in favor ofsecurityKeysoption. To eliminate this warning, move your networkKey into the securityKeys.S0_Legacy option. Refer to the Z-Wave JS docs for more information
I’m not sure what to make of this but it seems to tie in with some of the issues above.
You probably can specify those keys in yaml somehow in the add on configuration for the supervisor, but I’m not sure how to format it exactly. It would be here.
Zwavejs should have 4 security keys configured per its documentation.
https://zwave-js.github.io/node-zwave-js/#/getting-started/security-s2
Zwavejs2mqtt let’s you configure the keys in the UI in the control panel, but I’m not sure how to specify the keys for the zwavejs add on without the control panel. Hopefully someone else can chime in.
1 Like
Yeah I swapped over to ZWave2MQTT and had no issues with the switch reporting data. So it is a problem that seems to be narrowed down to something having to do with the security keys and ZWaveJS.
I think it’s a problem with the actual zwavejs integration/addon in Home Assistant not keeping up with the recent changes in the zwavejs server.
It’s all a bit confusing, and hopefully I’m explaining this right, but basically you have zwavejs, which is a server and driver that manages the ins and outs of the actual zwave network. Then you have zwavejs2mqtt, which includes the zwavejs server, but also has a control panel that sits on top of it. I would think of zwavejs like DOS, and zwavejs2mqtt like early versions of Windows that sits over DOS.
Now you have the zwavejs integration in Home Assistant, which is also basically a small control panel that can manage the zwavejs server within Home Assistant itself(either installed through the zwavejs official addon, the zwavejs2mqtt community addon, or a zwavejs2mqtt docker install even) and also communicate with it to bring your devices into Home Assistant. That integration doesn’t seem to be managing inclusion and security properly at this point, because zwavejs changed it recently and the integration hasn’t caught up yet.
You can see here from the documentation and little video that including a device in zwavejs2mqtt you are given options for no security, s0 security, s2 unauthenticated, and s2 authenticated (where dsk is needed from the device’s barcode)
https://zwave-js.github.io/zwavejs2mqtt/#/guide/nodes-management
Versus including from the Home Assistant integration, you only get a toggle for security or no security, which doesn’t even seem to communicate the choice to zwavejs properly.
This all has created a lot of issues for people recently coming over on fresh zwave setups. I’m sure it will all be sorted out eventually, but there are a lot of moving pieces here. I think the best bet is to just use zwavejs2mqtt right now and use that control panel to add and remove devices until the integration catches up, and avoid S2 inclusion until this PR is merged - https://github.com/zwave-js/node-zwave-js/pull/3240
I think you are spot on with this and I essentially came to the same conclusion after a few hours of trying to process all this. Thanks Tim - people like you make the community great!
1 Like
It’s not really possible for HA to become out of date with any zwave-js-server changes without you knowing. The zwave_js integration requires a minimum API schema version to be supported by the remote server. If the API schema version HA requires is newer than what the server version supports you’ll get an error and the integration will not load.
The zwave-js-server is also backwards compatible with previous versions of the API schema. Even if you install a newer version of zwave-js-server, HA will still be compatible, as long as the API schema versioning requirements are met.
The official addon also pins specific versions of zwave-js-server and zwave-js, and these are always compatible with the latest version of HA. Usually the addon versions are bumped prior to an HA release when HA requires a new minimum API schema version.
Zwavejs2mqtt hosts the same zwave-js-server code that the official addon does. They may not have the same versions at the same time, yet you can switch between both (assuming the same version requirements are met). If you are using HA to talk to the server hosted by zwavejs2mqtt, it’s really no different than using the official addon. However, there is a difference when using the z2m control panel since it talks to the driver via its API directly.
Regarding inclusion, there was the bug (referenced earlier) in 2021.9.0 to 2021.9.3, and fixed in 2021.9.4, that broke inclusion in some ways, but it was not because of changes to the server. The fix did not require any server changes. HA always sets the inclusion strategy to S0 (when the secure toggle is on) or Insecure. It never enables the S2 inclusion strategies. While the server does support the functionality, HA needs to implement the DSK PIN prompt in the UI.
You don’t need the S2 keys defined to use S0. S2 bootstrapping should never be attempted when including with HA. Assuming you tried this on 2021.9.4 (and not 2021.9.0-3), and since HA cannot include with S2, if this node was somehow included with S2 then there is an unknown problem that would be worth investigating. Even if S2 inclusion was specified, without the S2 keys defined the bootstrapping would fail, and node-zwave-js should then downgrade to Insecure. However, since you’ve moved on to zwavejs2mqtt that may be moot at this point. Without driver debug logs of the inclusion process it isn’t possible to debug any further.
Regarding the official addon, S2 keys are not supported yet but a PR is in progress. Since HA itself doesn’t support S2 yet, the keys will not be useful unless you are switching back and forth between zwavejs2mqtt and the official addon.
Thanks for the additional info and context on the relation between home assistant and zwavejs. When I said the homeassistant integration hasn’t “caught up”, I don’t mean to imply a version or compatibility issue, which would be a far bigger problem.
This really gets to what I mean by “hasn’t caught up”. You have new features in the zwavejs server that can’t be accessed through the Home-assistant UI until there is an update. Actually even before this you couldn’t and can’t do things like set associations and update firmware with the home assistant zwavejs integration, so while I’m not trying to knock all the progress around the integration, it remains incomplete compared to all the abilities zwavejs can control.
So yes things will remain backwards compatible and “work”, but whenever you have two different development projects that are semi dependent and comingled like this, they’re bound to get out of sync at some point.
While I agree the fact the switch included fine with zwavejs2mqtt and didn’t with zwavejs should be further investigated, the fact there was a difference at all, is evidence there’s something missing on the homeassistant side, probably not in the zwavejs add on, but in the config for it or the integration’s UI.
Another thought on this, do you think it’s possible that during inclusion the code is just running through methods that start with the highest security inclusion first, see no key is specified for s2 in the config, then throws the warning that is seen in the OP’s logs? If that’s the case, is it just a warning in the log that doesn’t apply, or does it cause some other issue, like the OP’s lack of status updates? If it does create an issue, even if S2 security is not being used, could the lack of a defined key in the config in the zwavejs addon create a problem? Just some thoughts
I was mostly responding to this comment:
My long-winded reply was just arguing that the integration (in 2021.9.4 at least) is managing inclusion and security just fine. There is no issue with changes to the driver or the server that would require the integration to “catch up” because backwards compatibility is maintained. Only using S0 inclusion is a valid configuration for an application, something that zwave-js still supports. Maybe you meant something else, if so, my mistake.
With that said, the observation about the missing S2 keys looks to be spot on though, and is a bug in the driver, fixed in zwave-js 8.2.2. That’s not an integration problem, it’s a driver problem. The integration is doing everything it’s supposed to be doing.
So it does make sense that switching to zwavejs2mqtt and setting the S2 keys fixed the problem, and if included with S2, that’s an improvement. Once the official addon incorporates zwave-js 8.8.2 (or the zwavejs2mqtt addon), then insecure inclusion will work again for this device in HA. In fact, if you install zwavejs2mqtt 5.5.2 you can use insecure inclusion as well, without needing S2 keys (not something I’d do though).
The driver docs have a pretty good high level description of the process. During the S2 bootstrapping process the driver will ask the node which security classes it wants granted, it doesn’t attempt to include by a priority. A device might only ask for S2 Unauthenticated, or it might ask for multiple. The user can choose which security classes to grant the device, denying some if desired.
That warning is from the server not the driver, and it’s printed once when the configuration is parsed, not related to the inclusion commands. It’s just a warning so server applications (like the addon) will update their settings before node-zwave-js pulls the plug on the old option. Meanwhile, the server handles the backwards compatibility anyways.
It looks like we got to the root cause of the problem - The way the zwavejs server was including any switch, regardless of security or not, required a security key to be defined, and there was no way to define that security key through the zwavejs addon in Home Assistant. I agree that’s a bug on the zwavejs server side, not the Home Assistant side, since the program should ignore that a key is not needed since you’re not doing a secure inclusion, and I’m glad to see they already fixed it in zwavejs with your linked github issue.
But now this fix, which was already corrected over a week ago, has to trickle its way over to Home Assistant and get fixed on this side, which again gets back to my point about “catching up”.
Look I’m not trying to further an argument here, and there’s a degree of semantics involved on a very complex issue, but it if was working “just fine”, the OP would have had no issue with their device and would not have started this thread.
I think we need to take a step back and look at this through the end user’s perspective, maybe someone just starting out without the technical knowledge and expertise you have. What they see is zwavejs is the “official” add on, and zwavejs2mqtt is the “community” add on. Well to me, official sounds better right? So I pick that one, install it, and probably for the vast majority of users it works just fine and they have a seamless experience.
Now you have some users that are running into complex issues, like here z-Wave JS --> z-wave JS mqtt and here How do it get zwavejs2mqtt installed on a hassio pi4 ssd system , and realize that the zwavejs is missing some zwave management features, and zwavejs2mqtt has a control panel, network graph, ability to set custom config parameters, ability to set associations, ability to update firmware, etc that isn’t in the zwavejs “official” addon. You may have a bit of buyers remorse - why can’t the “official” addon do as much as the community add on? Now you have to figure out how to move network keys and files around to switch. You’re probably not impressed.
How much time had to be wasted here by the OP where if they just installed zwavejs2mqtt from the get go they would have been up and running in no time? How much time are others that aren’t posting in the forum wasting? These are also generally newer users, or even if they are longer users of Home Assistant they’re newly using zwave with it - at best they spend a lot of time and learn something, but at worst they get frustrated, abandon Home Assistant, and just go back to what they had before.
I initially I appeared to be a little off on some of the finer details about where the breakdown happened and why, which I appreciate you clarifying in your posts. I’m not a network engineer and will be the first to admit I may get things wrong, which is why I try and preface a lot of things with “I’m thinking” or “it looks like” if I’m not sure. This is good because it sparks conversation from people like you, and recently even a developer from zwavejs itself to join the conversation and add additional context or correct me when I’m wrong.
So to conclude this long winded post, the only point I am trying to make, as I post this today, is that one is much better off installing zwavejs2mqtt over the “official” zwavejs add on for overall functionality, features, and control of their zwave network. I’m sure soon, hopefully that will no longer be the case with the continuing great work of the developers around the project- both in Home Assistant and Zwavejs.
I guess we should consider this a compliment about the state of the zwave_js integration and addons that we can complain about unincorporated fixes that are “already” a whole 7 days old. 
There was also the Labor Day holiday weekend during this time. The devs do take vacation from HA sometimes. 
To clarify, the “just fine” comment was in regards to how the integration and server/addon/driver work together. It was not a comment on OP’s problem. Again, I guess I misunderstood your comment, but my point was that there are no changes required to the integration to properly manage inclusion.
These would be some excellent points if we were debating the merits of the official addon vs. zwavejs2mqtt. None of my previous comments hinted at support or arguments in favor of one addon vs. the other, so I don’t have a clue where this has come from. 
Well, since you’re asking, as I write this comment:
- If OP had done this a few days earlier and installed upstream zwavejs2mqtt 5.5.1 instead, they would have wasted the same amount of time because it suffered from the same driver bug.
- If OP installs the community addon this very moment, then they would have wasted the same amount of time because the community addon uses zwavejs2mqtt 5.5.1.
That is all assuming that the bug I linked was OPs actually problem. To fix it they would have had to install zwavejs2mqtt upstream, not using an addon, which has never been confirmed. If using the addon, well back to the drawing board!
Meanwhile, as we speak the official addon v0.1.39 now supports node-zwave-js v8.2.3 (thanks to this post, BTW). Neither upstream zwavejs2mqtt nor the community addon have been updated with this newer driver version. I wonder how long it will take for the fixes in that release to “trickle” down to them.
Oh, and here’s the opposite situation where S0 inclusion was broken in zwavejs2mqtt, but was working fine with the official addon and HA 2021.8. Should we tell people to avoid zwavejs2mqtt and stick with the official addon because of that one bug?
The point is, driver bugs can affect all the applications (addons, zwavejs2mqtt standalone) equally, and just because one updated a few days earlier in this specific instance doesn’t make for a great argument to choose it over the other. There may be other reasons to do so, as you’ve enumerated.
As often happens, I guess we are at the point where the conversion devolves and written communication is just not coming through very well. I’ll see you in another post! 
Yes definitely time for me to let it go, lol. No hard feelings and I’m sure will chat again soon.
1 Like