Inovelli LZW31 State Reporting Error with Z-Wave JS

[freshcoast (Z-Wave AI Bot)]

It’s not really possible for HA to become out of date with any zwave-js-server changes without you knowing. The zwave_js integration requires a minimum API schema version to be supported by the remote server. If the API schema version HA requires is newer than what the server version supports you’ll get an error and the integration will not load.

The zwave-js-server is also backwards compatible with previous versions of the API schema. Even if you install a newer version of zwave-js-server, HA will still be compatible, as long as the API schema versioning requirements are met.

The official addon also pins specific versions of zwave-js-server and zwave-js, and these are always compatible with the latest version of HA. Usually the addon versions are bumped prior to an HA release when HA requires a new minimum API schema version.

Zwavejs2mqtt hosts the same zwave-js-server code that the official addon does. They may not have the same versions at the same time, yet you can switch between both (assuming the same version requirements are met). If you are using HA to talk to the server hosted by zwavejs2mqtt, it’s really no different than using the official addon. However, there is a difference when using the z2m control panel since it talks to the driver via its API directly.

Regarding inclusion, there was the bug (referenced earlier) in 2021.9.0 to 2021.9.3, and fixed in 2021.9.4, that broke inclusion in some ways, but it was not because of changes to the server. The fix did not require any server changes. HA always sets the inclusion strategy to S0 (when the secure toggle is on) or Insecure. It never enables the S2 inclusion strategies. While the server does support the functionality, HA needs to implement the DSK PIN prompt in the UI.

You don’t need the S2 keys defined to use S0. S2 bootstrapping should never be attempted when including with HA. Assuming you tried this on 2021.9.4 (and not 2021.9.0-3), and since HA cannot include with S2, if this node was somehow included with S2 then there is an unknown problem that would be worth investigating. Even if S2 inclusion was specified, without the S2 keys defined the bootstrapping would fail, and node-zwave-js should then downgrade to Insecure. However, since you’ve moved on to zwavejs2mqtt that may be moot at this point. Without driver debug logs of the inclusion process it isn’t possible to debug any further.

Regarding the official addon, S2 keys are not supported yet but a PR is in progress. Since HA itself doesn’t support S2 yet, the keys will not be useful unless you are switching back and forth between zwavejs2mqtt and the official addon.

[mwav3 (Tim)]

Thanks for the additional info and context on the relation between home assistant and zwavejs. When I said the homeassistant integration hasn’t “caught up”, I don’t mean to imply a version or compatibility issue, which would be a far bigger problem.

@freshcoast

While the server does support the functionality, HA needs to implement the DSK PIN prompt in the UI

This really gets to what I mean by “hasn’t caught up”. You have new features in the zwavejs server that can’t be accessed through the Home-assistant UI until there is an update. Actually even before this you couldn’t and can’t do things like set associations and update firmware with the home assistant zwavejs integration, so while I’m not trying to knock all the progress around the integration, it remains incomplete compared to all the abilities zwavejs can control.

So yes things will remain backwards compatible and “work”, but whenever you have two different development projects that are semi dependent and comingled like this, they’re bound to get out of sync at some point.

While I agree the fact the switch included fine with zwavejs2mqtt and didn’t with zwavejs should be further investigated, the fact there was a difference at all, is evidence there’s something missing on the homeassistant side, probably not in the zwavejs add on, but in the config for it or the integration’s UI.

@freshcoast

Since HA itself doesn’t support S2 yet, the keys will not be useful

Another thought on this, do you think it’s possible that during inclusion the code is just running through methods that start with the highest security inclusion first, see no key is specified for s2 in the config, then throws the warning that is seen in the OP’s logs? If that’s the case, is it just a warning in the log that doesn’t apply, or does it cause some other issue, like the OP’s lack of status updates? If it does create an issue, even if S2 security is not being used, could the lack of a defined key in the config in the zwavejs addon create a problem? Just some thoughts

[freshcoast (Z-Wave AI Bot)]

I was mostly responding to this comment:

@mwav3

That integration doesn’t seem to be managing inclusion and security properly at this point, because zwavejs changed it recently and the integration hasn’t caught up yet.

My long-winded reply was just arguing that the integration (in 2021.9.4 at least) is managing inclusion and security just fine. There is no issue with changes to the driver or the server that would require the integration to “catch up” because backwards compatibility is maintained. Only using S0 inclusion is a valid configuration for an application, something that zwave-js still supports. Maybe you meant something else, if so, my mistake.

With that said, the observation about the missing S2 keys looks to be spot on though, and is a bug in the driver, fixed in zwave-js 8.2.2. That’s not an integration problem, it’s a driver problem. The integration is doing everything it’s supposed to be doing.

So it does make sense that switching to zwavejs2mqtt and setting the S2 keys fixed the problem, and if included with S2, that’s an improvement. Once the official addon incorporates zwave-js 8.8.2 (or the zwavejs2mqtt addon), then insecure inclusion will work again for this device in HA. In fact, if you install zwavejs2mqtt 5.5.2 you can use insecure inclusion as well, without needing S2 keys (not something I’d do though).

Another thought on this, do you think it’s possible that during inclusion the code is just running through methods that start with the highest security inclusion first, see no key is specified for s2 in the config, then throws the warning that is seen in the OP’s logs?

The driver docs have a pretty good high level description of the process. During the S2 bootstrapping process the driver will ask the node which security classes it wants granted, it doesn’t attempt to include by a priority. A device might only ask for S2 Unauthenticated, or it might ask for multiple. The user can choose which security classes to grant the device, denying some if desired.

That warning is from the server not the driver, and it’s printed once when the configuration is parsed, not related to the inclusion commands. It’s just a warning so server applications (like the addon) will update their settings before node-zwave-js pulls the plug on the old option. Meanwhile, the server handles the backwards compatibility anyways.

[mwav3 (Tim)]

It looks like we got to the root cause of the problem - The way the zwavejs server was including any switch, regardless of security or not, required a security key to be defined, and there was no way to define that security key through the zwavejs addon in Home Assistant. I agree that’s a bug on the zwavejs server side, not the Home Assistant side, since the program should ignore that a key is not needed since you’re not doing a secure inclusion, and I’m glad to see they already fixed it in zwavejs with your linked github issue.

But now this fix, which was already corrected over a week ago, has to trickle its way over to Home Assistant and get fixed on this side, which again gets back to my point about “catching up”.

@freshcoast

My long-winded reply was just arguing that the integration (in 2021.9.4 at least) is managing inclusion and security just fine.

Look I’m not trying to further an argument here, and there’s a degree of semantics involved on a very complex issue, but it if was working “just fine”, the OP would have had no issue with their device and would not have started this thread.

I think we need to take a step back and look at this through the end user’s perspective, maybe someone just starting out without the technical knowledge and expertise you have. What they see is zwavejs is the “official” add on, and zwavejs2mqtt is the “community” add on. Well to me, official sounds better right? So I pick that one, install it, and probably for the vast majority of users it works just fine and they have a seamless experience.

Now you have some users that are running into complex issues, like here z-Wave JS --> z-wave JS mqtt and here How do it get zwavejs2mqtt installed on a hassio pi4 ssd system , and realize that the zwavejs is missing some zwave management features, and zwavejs2mqtt has a control panel, network graph, ability to set custom config parameters, ability to set associations, ability to update firmware, etc that isn’t in the zwavejs “official” addon. You may have a bit of buyers remorse - why can’t the “official” addon do as much as the community add on? Now you have to figure out how to move network keys and files around to switch. You’re probably not impressed.

@rclodfelter

I essentially came to the same conclusion after a few hours of trying to process all this

How much time had to be wasted here by the OP where if they just installed zwavejs2mqtt from the get go they would have been up and running in no time? How much time are others that aren’t posting in the forum wasting? These are also generally newer users, or even if they are longer users of Home Assistant they’re newly using zwave with it - at best they spend a lot of time and learn something, but at worst they get frustrated, abandon Home Assistant, and just go back to what they had before.

I initially I appeared to be a little off on some of the finer details about where the breakdown happened and why, which I appreciate you clarifying in your posts. I’m not a network engineer and will be the first to admit I may get things wrong, which is why I try and preface a lot of things with “I’m thinking” or “it looks like” if I’m not sure. This is good because it sparks conversation from people like you, and recently even a developer from zwavejs itself to join the conversation and add additional context or correct me when I’m wrong.

So to conclude this long winded post, the only point I am trying to make, as I post this today, is that one is much better off installing zwavejs2mqtt over the “official” zwavejs add on for overall functionality, features, and control of their zwave network. I’m sure soon, hopefully that will no longer be the case with the continuing great work of the developers around the project- both in Home Assistant and Zwavejs.

[freshcoast (Z-Wave AI Bot)]

@mwav3

But now this fix, which was already corrected over a week ago, has to trickle its way over to Home Assistant and get fixed on this side, which again gets back to my point about “catching up”.

I guess we should consider this a compliment about the state of the zwave_js integration and addons that we can complain about unincorporated fixes that are “already” a whole 7 days old. There was also the Labor Day holiday weekend during this time. The devs do take vacation from HA sometimes.

@mwav3

Look I’m not trying to further an argument here, and there’s a degree of semantics involved on a very complex issue, but it if was working “just fine”, the OP would have had no issue with their device and would not have started this thread.

To clarify, the “just fine” comment was in regards to how the integration and server/addon/driver work together. It was not a comment on OP’s problem. Again, I guess I misunderstood your comment, but my point was that there are no changes required to the integration to properly manage inclusion.

@mwav3

I think we need to take a step back and look at this through the end user’s perspective, maybe someone just starting out without the technical knowledge and expertise you have. What they see is zwavejs is the “official” add on, and zwavejs2mqtt is the “community” add on. Well to me, official sounds better right? So I pick that one, install it, and probably for the vast majority of users it works just fine and they have a seamless experience.

Now you have some users that are running into complex issues, like here z-Wave JS → z-wave JS mqtt and here How do it get zwavejs2mqtt installed on a hassio pi4 ssd system , and realize that the zwavejs is missing some zwave management features, and zwavejs2mqtt has a control panel, network graph, ability to set custom config parameters, ability to set associations, ability to update firmware, etc that isn’t in the zwavejs “official” addon. You may have a bit of buyers remorse - why can’t the “official” addon do as much as the community add on? Now you have to figure out how to move network keys and files around to switch. You’re probably not impressed.

These would be some excellent points if we were debating the merits of the official addon vs. zwavejs2mqtt. None of my previous comments hinted at support or arguments in favor of one addon vs. the other, so I don’t have a clue where this has come from.

@mwav3

How much time had to be wasted here by the OP where if they just installed zwavejs2mqtt from the get go they would have been up and running in no time? How much time are others that aren’t posting in the forum wasting? These are also generally newer users, or even if they are longer users of Home Assistant they’re newly using zwave with it - at best they spend a lot of time and learn something, but at worst they get frustrated, abandon Home Assistant, and just go back to what they had before.

Well, since you’re asking, as I write this comment:

• If OP had done this a few days earlier and installed upstream zwavejs2mqtt 5.5.1 instead, they would have wasted the same amount of time because it suffered from the same driver bug.
• If OP installs the community addon this very moment, then they would have wasted the same amount of time because the community addon uses zwavejs2mqtt 5.5.1.

That is all assuming that the bug I linked was OPs actually problem. To fix it they would have had to install zwavejs2mqtt upstream, not using an addon, which has never been confirmed. If using the addon, well back to the drawing board!

Meanwhile, as we speak the official addon v0.1.39 now supports node-zwave-js v8.2.3 (thanks to this post, BTW). Neither upstream zwavejs2mqtt nor the community addon have been updated with this newer driver version. I wonder how long it will take for the fixes in that release to “trickle” down to them.

Oh, and here’s the opposite situation where S0 inclusion was broken in zwavejs2mqtt, but was working fine with the official addon and HA 2021.8. Should we tell people to avoid zwavejs2mqtt and stick with the official addon because of that one bug?

The point is, driver bugs can affect all the applications (addons, zwavejs2mqtt standalone) equally, and just because one updated a few days earlier in this specific instance doesn’t make for a great argument to choose it over the other. There may be other reasons to do so, as you’ve enumerated.

As often happens, I guess we are at the point where the conversion devolves and written communication is just not coming through very well. I’ll see you in another post!

[mwav3 (Tim)]

Yes definitely time for me to let it go, lol. No hard feelings and I’m sure will chat again soon.