zwaveJS network stops responding after 24 hours or so with "Timeout while waiting for an ACK"

Configuration Z-Wave
1

My Z-Wave network regularly stops working. After a power cycle it will last somewhere between a few hours to a few days. I feel like it was slightly more reliable when I first set it up, but it has been problematic for a long time now.

Power cycling my Raspberry Pi 4 or just unplugging and replugging the Z-Wave dongle solves the problem temporarily until the next crash. The dongle is connected to a USB 2 port on the Pi through an extender cable about 2 or 3 feet long and suspended as far away from any electronics as possible.

I am using a SLUSB001A by Silicon Labs which is a 700 series controller. I have updated the firmware to 7.18.2 and it shows up like so in Home Assistant devices:

ZST10-700
by Silicon Labs
Firmware: 7.18.2

The firmware update did not seem to have any effect.

I have a total of 13 Z-Wave devices. Only 3 of them are battery powered and I believe the rest are acting as repeaters. I have tried healing the network several times to no effect.

Here is my Home Assistant information:

Home Assistant 2023.1.4
Supervisor 2022.12.1
Operating System 9.4
Frontend 20230110.0 - latest

Below is the log for the zwaveJS add-on. Iā€™ve tried searching some of the lines in there and am getting very few results, nothing that is really helping me out.

Any suggestions for what I might try next to get a bit more stability? Any other information I can provide?

Z-Wave error ZWaveError: Timeout while waiting for an ACK from the controller (ZW0200)
    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)
    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)
    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)
    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)
    at ConfigurationCCAPI.set (/usr/src/node_modules/@zwave-js/cc/src/cc/ConfigurationCC.ts:647:24)
    at Proxy.ConfigurationCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/ConfigurationCC.ts:420:29)
    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)
    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)
    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)
    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99) {
  code: 200,
  context: undefined,
  transactionSource: '    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)\n' +
    '    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)\n' +
    '    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)\n' +
    '    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)\n' +
    '    at ConfigurationCCAPI.set (/usr/src/node_modules/@zwave-js/cc/src/cc/ConfigurationCC.ts:647:24)\n' +
    '    at Proxy.ConfigurationCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/ConfigurationCC.ts:420:29)\n' +
    '    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)\n' +
    '    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)\n' +
    '    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)\n' +
    '    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99)'
}
2023-01-16T16:39:25.999Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:39:27.105Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
2023-01-16T16:39:29.211Z CNTRLR   [Node 017] did not respond after 1/3 attempts. Scheduling next try in 500 ms.
2023-01-16T16:39:30.724Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:39:31.830Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
2023-01-16T16:39:33.939Z CNTRLR   [Node 017] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2023-01-16T16:39:35.457Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:39:36.563Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
Z-Wave error ZWaveError: Timeout while waiting for an ACK from the controller (ZW0200)
    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)
    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)
    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)
    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)
    at Proxy.set (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:238:24)
    at Proxy.MultilevelSwitchCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:344:6)
    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)
    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)
    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)
    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99) {
  code: 200,
  context: undefined,
  transactionSource: '    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)\n' +
    '    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)\n' +
    '    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)\n' +
    '    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)\n' +
    '    at Proxy.set (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:238:24)\n' +
    '    at Proxy.MultilevelSwitchCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:344:6)\n' +
    '    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)\n' +
    '    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)\n' +
    '    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)\n' +
    '    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99)'
}
2023-01-16T16:39:52.078Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:39:53.185Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
2023-01-16T16:39:55.291Z CNTRLR   [Node 009] did not respond after 1/3 attempts. Scheduling next try in 500 ms.
2023-01-16T16:39:56.803Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:39:57.909Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
2023-01-16T16:40:00.016Z CNTRLR   [Node 009] did not respond after 2/3 attempts. Scheduling next try in 500 ms.
2023-01-16T16:40:01.527Z CNTRLR   Failed to execute controller command after 1/3 attempts. Scheduling next try i
                                  n 100 ms.
2023-01-16T16:40:02.634Z CNTRLR   Failed to execute controller command after 2/3 attempts. Scheduling next try i
                                  n 1100 ms.
Z-Wave error ZWaveError: Timeout while waiting for an ACK from the controller (ZW0200)
    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)
    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)
    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)
    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)
    at Proxy.set (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:238:24)
    at Proxy.MultilevelSwitchCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:344:6)
    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)
    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)
    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)
    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99) {
  code: 200,
  context: undefined,
  transactionSource: '    at Driver.sendMessage (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:3990:23)\n' +
    '    at Driver.sendCommandInternal (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4181:28)\n' +
    '    at Driver.sendSupervisedCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4226:27)\n' +
    '    at Driver.sendCommand (/usr/src/node_modules/zwave-js/src/lib/driver/Driver.ts:4278:30)\n' +
    '    at Proxy.set (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:238:24)\n' +
    '    at Proxy.MultilevelSwitchCCAPI.<computed> (/usr/src/node_modules/@zwave-js/cc/src/cc/MultilevelSwitchCC.ts:344:6)\n' +
    '    at ZWaveNode.setValue (/usr/src/node_modules/zwave-js/src/lib/node/Node.ts:932:29)\n' +
    '    at NodeMessageHandler.handle (/usr/src/node_modules/@zwave-js/server/dist/lib/node/message_handler.js:23:38)\n' +
    '    at Object.node (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:40:96)\n' +
    '    at Client.receiveMessage (/usr/src/node_modules/@zwave-js/server/dist/lib/server.js:105:99)'
}
2

In case it helps, my Z-Wave devices consist of:

2 Yale locks
6 Inovelli Red dimmer switches
1 T6 Pro thermostat
1 AEON Labs smart outlet
1 HomeSeer floodlight sensor (issues were happening before I added this device)
1 HomeSeer HS-FS100+ light sensor (issues were happening before I added this device)

Currently the locks are the only devices on battery power.

3

Minor update. I could have sworn a soft reboot of the Raspberry Pi didnā€™t fix anything but I tested it again today and it seems to work to fix things now. I also tried just resetting the zwavejs add on and that also fixes the issue temporarily.

At this point Iā€™d settle for a way to detect errors and reset the add on, though Iā€™m sure resetting the add on multiple times per day is likely to cause some signals to be missed and would still like to solve to root issue.

When restarting the add on I noticed my logs were only set to INFO. Iā€™ve switched to DEBUG and will post new logs next time it hangs.

4

Try disabling the watchdog, seems to solve unexplainable issues sometimes.

5

Iā€™ve had similar experience with an RPi. The timeouts and ā€œfailed to executeā€ behavior seemed to indicate that the system was struggling to keep up with the activity. My logs looked a lot like yours.

In my case, I eventually tracked the problem down to a lack of memory in my RPi setup. My RPi was a 1GB version. Checking the memory and processor usage (in System), after power up it was running at 80% or so memory utilization. Over time, maybe 24 hours or so, that would climb to 95% and behavior would start to get all those timeouts and such errors.

I swapped the system over to a new RPi with 8GB memory and everything has worked fine ever since, with all the same devices, integrations, etc.

I suggest looking at your processor and memory usage to see if that might be the problem. HA seems to go berserk when itā€™s starved for resources. My system now is running at about 15% max utilization.

Good luck!
Jack

6

Thanks for the suggestions!

Last night I migrated from zwavejs integration to zwave js ui (still using websockets) for better visibility into what is going on. Iā€™ll try out these options after the next crash so Iā€™m only changing one thing at a time.

7

If youā€™re still having problems, check to see what your memory/processor usage is immediately after you power up the RPi. More details here: HA Strange Behavior -- SOLVED!

8

Try the advices from one of the zwavejs developers when you have Timeout while waiting for an ACK from the controller:
https://zwave-js.github.io/node-zwave-js/#/troubleshooting/connectivity-issues?id=problems-communicating-with-the-stick

Improve the network health to avoid message flood: Z-Wave JS - Z-Wave driver written entirely in JavaScript/TypeScript

Use the network graph in zwavejs-ui to find how many hops the devices use to communicate with the controller. It should be using one hop to the controller: Z-Wave JS UI

9

Did you mean the watchdog for the zwave add-on? If so, that does not seem to be my issue as I just had a lockup with the watchdog disabled (though it did run for a bit longer than usual).

10

Thanks for the suggestion. Every time I look at the memory usage page it is between 30-40% used. I used to have the systemmonitor sensor logging the memory usage but that seems to have mysteriously broken sometime in September. Iā€™ll try to get that fixed and then cross reference that with the crashes going forward.

For the record, Iā€™m using a Raspberry Pi 4 with 4 GB RAM.

11

Thanks for those links! Iā€™ll dig into those and see what I can tweak.

Looking at my graph, I currently have 3 nodes at the 1 hop level, 6 nodes at the 2 hop level, and 3 nodes at the 3 hop level. Two of those 3 hop devices are actually pretty close to my Z-Wave controller, so Iā€™ll try healing the network to improve that. I have 2 devices inside and one device on the outside of a detached garage about 40 feet from the house which will likely need at least one extra hop, but they seem to be at least as reliable as the rest of the network.

12

You should not have nodes which has so many hops to the controller. It should be one hop to the controller.
I see you have a 700 series controller. I tried to use a Aeotec z-stick7 with a extension usb cable as a controller, but experienced some of my nodes had two hops to the controller. The controller logged lots of packet loss and problems communication with the controller. When I changed my controller to my old Aeotec z-stick5 all the logging disappeared. All the nodes have now one hop to the controller.
If you have a old zstick5 available maybe trying to do a restore to this zstick will help.

13

Unfortunately the 700 series stick is the only one that I have.

As for the memory usage theory, now that Iā€™ve been logging memory usage I can say that usage is pretty steady at 37% used. I had a jump to about 47% during the last update while my database was migrating, but otherwise it has been extremely consistent.

14

You can try to ping the dead nodes: Automate ZwaveJS Ping Dead Nodes? - #3 by freshcoast and maybe it helps.

15

Interesting. Iā€™ll give that a try next time it locks up on me.

16

Based on the logs the problem is with the stick / USB interface. I have read on this forum about similar issues caused by low voltage levels on the PI. See if you can check into that.

Other things Iā€™d consider
a) use a different USB port
b) get a powered USB 2.0 hub, plug this into the PI and the long cable into it. If itā€™s a low voltage on the stick this should solve it.
c) get another stick (youā€™ll want to have a backup stick anyways)

17

Thanks for the suggestion. Iā€™ve borrowed a powered USB hub and connected the Z-Wave stick through that. Fingers crossed that helps. If so, I guess Iā€™ll buy a dedicated powered hub for that server. If not, I guess Iā€™ll be buying an alternate Z-Wave stick.

18

Iā€™m now at 4 days of uptime without any Z-Wave issues, so Iā€™m assuming the powered USB hub is the solution and have ordered a dedicated one.

Thinking back, I used to get a few days of runtime before it would lock up. I suspect that the firmware update fixed a software bug that a lot of people (including me) were experiencing, but maybe just so happened to be more demanding on the power, causing it to actually make my system worse.

Either way, Iā€™m happy I can finally properly use my Z-Wave network!

19

Glad that it is working. At some point try to take a look at the voltage going into the PI from the power supply (or get a high quality one). If this is low - which it appears to be since the powered hub is helping - you may end up with other apparently random failures as the power from the utility fluctuates.

20

Iā€™m using the 5.1V 3.5A USB C power supply from CanaKit. I thought that was supposed to be a high quality supply, but maybe it isnā€™t (or maybe I got a dud). If I see more weirdness Iā€™ll see about measuring it or swapping it out with something better.