Huge delays with Aeotec Z-Wave stick / concurrency issues

I have a simple Z-Wave setup with the Aeotec Z-Wave stick, plugged into my relatively powerful workstation running Debian 10. It’s working fine most of the time, but I’m seeing huge delays in receiving (and sometimes sending) events.

Tried both OpenZwave 1.4 and 1.6, with Zwave2MQTT. Sometimes it takes 10 seconds for an event from my Nanomote Quad to arrive at the gateway, while I’m just meters away from it. It’s not happening with the Neo Coolcam plug switches, but it does also happen with the Aeotec Nano Dimmer and Dual Nano swtich.

My theory is that messages seem to get queued somewhere on the USB serial level when a device sends multiple messages at the same time. This is because the Neo Coolcam device just sends one message with a status update, but all other devices immediately respond with multiple:

  • Nanomote Quad always reports its battery percentage change event on every button press event. With OZW 1.4 I see usually only one event and the other is missing. With OZW 1.6 I always see both (delayed 500ms - 10s) or none (rare).
  • Aeotec Nano Dimmer sends multiple updates on every changes, including time (for nightlight). With OZW 1.4 the state is usually just a bit delayed, with OZW 1.6 it’s a bit better but taking at least a second until everything is processed.
  • Same for Dual Nano Switch as well.

My total number of nodes is just 5 currently of which 4 active, 1 battery.

When I use HA’s built-in Z-Wave integration it’s worse and basically unusable. Most of the messages get dropped and HA’s state is completely out of sync. With Zwave2MQTT and OZW 1.6 it’s much better, and ~95% of the messages arrive within a second but I’m still seeing delays of 10-20 seconds occasionally. With automations doing “toggles” this is super super annoying.

Also reproducible purely interacting with the active nodes (sending); OFF or ON to all four switches at the same time using a scene activation seems to have a loss of 5% of the messages and 10% delayed > 1s.

Does anyone have the same problem and found a way to improve the performance of the USB communication or a clue why this happens?

  • Double checked that ModemManager isn’t interfering with the serial device: it’s not installed, no such systemd unit exists on my machine.
  • Tried other USB ports.
  • Tried using secure and insecure variants of inclusions to the network; no effect.
  • Tried taking out the Neo cheap stuff (then all devices are Aeotec), then healing network, no effect.
  • It must be on the network/stick somewhere because I’ve witnessed taking out the battery and then OZW logging NanoMote events…

I don’t know for sure that this is your issue… but…

Z-Wave needs a mesh. Lots of devices. If you only have 5 devices, as long as they are all pretty close to your USB Stick, it’ll be fine. If they are not, then less fine. Additionally, the USB Stick tends to get buried behind a computer. Get a long USB extension cable and move the stick out from behind the PC where it has to compete with a metal case and get it up on the wall with nothing blocking it.

Z-Wave also works with a “neighbor” concept. I don’t know all the specifics but, the way I’ve understood it is when you add a device to the network, it shouts that it wants to be added and (if you have ZWave Plus devices) the devices that can help it route to the controller speak up and they work out a plan and communication happens and the device is added. If something should happen that the device no longer has the neighbor it once had (you add a new piece of furniture that blocks communication, or in the case of the Nanomote, you move it to another room) that device may have a very easy path to communicate with the controller, but the path to the neighbor is unavailable. This causes it’s communication to be delayed as it tries to find a new path. And different devices do this differently.

All of my Zwave devices are stationary. My USB stick is mounted high on a wall, and in a central location. With that, my devices respond well 99% of the time. However, occasionally, I have 2 devices at my front door (25 feet from the USB Stick, and through 2 walls) that don’t seem to always communicate correctly on the first attempt. One of them is a Inovelli wall switch. I have an automation setup to work on “double tap” and sometimes I double tap and nothing happens. So I double tap again and everything is fine. The other device is a Schlage battery powered door lock. I have an automation that happens when the door unlocks and that automation only fires about 75% of the time.

Healing the network can help get the neighbors all figured out again. However, with these two devices, no amount of “healing” seems to fix them. I only have one of the Schlage devices, so I can’t speak for that. But I have another of the same Inovelli devices in another room and it works perfectly all the time. However, it’s only 10 feet away from the USB Stick and only has to travel through one wall.

Thanks for your reply.

Well, everything is at maximum 6 meters from the USB gateway in a small apartment. And I could live with some messages getting dropped, but the most severe problem is the delays. Last night it took 17 seconds for my NanoMote button press to reach the gateway. By that time I already got out of bed, walked to the wall switch and just before I pressed it (while counting in my head) it responded. If I could just make sure every delay more than 2 seconds would be ignored, it would be much better experience.

I’ve checked the neighbouring info in the Z-Wave control panel and ALL devices list node 1 as their primary. So I think they are all in direct contact with the primary gateway.

Check the config options for the device. Some send Hail or Basic messages by default which don’t really tell what has changed and OZW has to “refresh” all the dynamic values on a device (essential poll it) to figure out what changed.

You would look for something that sends a Binary Switch etc type message.

Just checking.
Did you happen to include the devices as secure nodes ?

Edit: because you should avoid that whenever possible

Im having same issues. It takes sometimes 20-30 before the switch react.
Between all wall switches (Neocoolcam) is less then 5meter distance.

I tried to heal network but didn’t help as well.

Also have 6 motion sensors (battery) and they work super FAST.

So I really don’t know how it can be that powered switches have this delay and battery sensors work like a charm…

I see I haven’t posted any updated to this thread, sorry.

To summarize; I’ve switched to ZWave2MQTT with OZW-1.6 and overall experience is MUCH better. Never looked back at 1.4. Occasionally, like 1 out of 50 switches is delayed by ~ 2 seconds, 1 out of 200 a little more or lost.

I switched also to Zwave2MQTT, what I also did , I bought a extension cable and connect stick on that. SO its far away from my RPI… Since then much much faster!!! even no delay at all

I have the same problem. My network contains 28 devices, some in the house some outside in the garden.
I see delays of 10seconds almost on every switch of a light in our living room as well as on switches in the garden The livingroom switch (Fibaro Dimmer) is located 1m away from the Pi (with UZB on an external hub high up in the wiring closet).
I’m running Zwave2MQTT 3.4 with OZW 1.6.

What would be the best way to troubleshoot this?

I’ve observed the opposite. For me the OZW 1.4 integration works better than the OZW 1.6 addon. The addon was unusable.

I changed my setup from a Pine64 running a built-in zwave controller to an intel nuc with aeotech gen5 controller after my pine64’s sdcard crashed. My zwave reliability has degraded significantly since then. The switches often don’t refresh if operated manually, occasionally there is high latency between requests.

Initially I just setup all devices using the hassio ozw 1.6 addon and noticed that after 10 mins or so the latency becomes really bad (15 seconds for each request). It seemed like there was a lot of network traffic for some reason and I assumed it was because maybe a node wasn’t paired correctly or something. Since then I’ve hard reset the controller, tried ozw 1.4 integration, tried to migrate to 1.6 addon again. It works OK with the integration but terribly with the addon.

I read somewhere that i can try to set the notification options to “Hail” but it seemed like i would have to plug the gen5 controller on a windows machine because the software that lets you do it runs on windows only. The ozw didn’t seem to have that option anywhere. I’m also tempted to upgrade the firmware to see if that fixes anything. Hopefully doing any of this won’t reset the controller again :sweat:

If the device supports it you can use the OZW Admin tool to change the notification reports:

You can see traffic passing along in the bottom of the OZW Admin tool. Or you can use a MQTT browser to see which devices/entities are updating on a too short interval.

I would start by investigating the connections between your nodes. You can check those in the Network Graph. See if any connections are “weird” or missing, if so you can heal that node or the entire network (might take some hours)
What version of HA are you running? There were some issues with dimmers before 0.114.0

Oddly, I don’t see that option on my aeotec stick. I’m guessing it’s because the firmware needs to be upgraded :man_shrugging:

My network looks ok. All nodes are “Ready”. Latency wise, it’s working fine at the moment with the integration v1.4. Most lights are updating status with manual operations.

I’m running on 0.114.4.

The Hail option is available on the device config, not the stick. I’m not sure if there’s even any firmware update available for the stick. If you’re running built-in Z-Wave (1.4) and everything’s okay, don’t change a winning team :wink:

I just wanted to update this thread with great news.

All my intermediate updates of Open-Zwave have failed over 2020 and early 2021 (using zwave2mqtt, upgrading embedded OZW 1.6), it’s been unstable and with great delays, until…

Since two weeks I’m using the new ZwaveJS integration (websocket only, without MQTT) and Home Assistant 2021.4 and most of the delays and issues are gone. :partying_face: It’s SO MUCH better, I cannot believe it. :heart_eyes: I think this is how it was supposed to run ever since I have this hardware. I suspect the Qubino mini dimmers I have in my network were not playing nice with OZW, as these nodes were generally involved in ~ 95% of the error messages in OZW.

Now with Zwave-JS, the network init takes a lot longer (~ 20 minutes), but boy it’s running smooth and FAST, like near-instant scene changes I have never experienced before. If you’re still on any OZW-integration with Home Assistant, take this message as an encouragement to take the effort of the migration to use ZwaveJS.

There’s only a minor new issue and that is concerning the Eurotronic Spirit thermostats I have. About ~ 5% of the times the temperature setpoint update does not get ack’d to HA and it appears to be back to the previous state after about 30 seconds.