ZWave partial (!) down

Tyfoon · April 11, 2020, 3:33pm

After running HASS for years and year (started with .22) I suddenly have a Zwave issue I can’t seem to crack. Overnight most of my zwave devices are not responding anymore. Only about 6 of my total +30 devices are working. The are others not.

I did not change anything to either hard or software. Running. 107.7 , HassIO on an Intel NUC. aeotec zwave gen5 stick.

What have tried:

Healing network
Resetting stick (via hass interface)
Restored to several days before I had this issue

Attached partial log:

2020-04-11 17:19:59.124 Detail,   Received: 0x01, 0x07, 0x00, 0x13, 0xa2, 0x01, 0x0d, 0x6f, 0x2a
2020-04-11 17:19:59.124 Detail,   ZW_SEND_DATA Request with callback ID 0xa2 received (expected 0x00)
2020-04-11 17:19:59.124 Warning, WARNING: Unexpected Callback ID received
2020-04-11 17:20:08.379 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:20:08.379 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa3, 0x7a
2020-04-11 17:20:08.379 Detail,
2020-04-11 17:20:08.379 Info, Node046, Sending (Poll) message (Callback ID=0xa3, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa3, 0x7a
2020-04-11 17:20:09.380 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:20:09.380 Detail, Node046, Removing current message
2020-04-11 17:20:09.380 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:21:09.389 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:21:09.390 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa4, 0x7d
2020-04-11 17:21:09.390 Detail,
2020-04-11 17:21:09.390 Info, Node046, Sending (Poll) message (Callback ID=0xa4, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa4, 0x7d
2020-04-11 17:21:10.390 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:21:10.390 Detail, Node046, Removing current message
2020-04-11 17:21:10.390 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:21:34.272 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x00, 0xe9
2020-04-11 17:21:34.272 Error, ERROR: ZW_SEND_DATA could not be delivered to Z-Wave stack
2020-04-11 17:21:34.274 Detail, CAN received...triggering resend
2020-04-11 17:21:34.274 Warning, m_currentMsg was NULL when trying to set MaxSendAttempts
2020-04-11 17:21:34.274 Always,
2020-04-11 17:21:34.274 Always, Dumping queued log messages
2020-04-11 17:21:34.274 Always,
2020-04-11 17:21:34.274 Always,
2020-04-11 17:21:34.274 Always, End of queued log message dump
2020-04-11 17:21:34.274 Always,
2020-04-11 17:21:34.274 Detail, WriteMsg CAN m_currentMsg=00000000
2020-04-11 17:21:34.275 Detail, Node009,   Received: 0x01, 0x10, 0x00, 0x04, 0x00, 0x09, 0x0a, 0x32, 0x02, 0x21, 0x34, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xcd
2020-04-11 17:21:34.275 Detail,
2020-04-11 17:21:34.275 Detail, Node009, Refreshed Value: old value=false, new value=false, type=bool
2020-04-11 17:21:34.275 Detail, Node009, Changes to this value are not verified
2020-04-11 17:21:34.275 Info, Node009, Received Meter report from node 9: Power=0.0W
2020-04-11 17:21:34.275 Detail, Node009, Refreshed Value: old value=0.1, new value=0.0, type=decimal
2020-04-11 17:21:34.275 Detail, Node009, Changes to this value are not verified
2020-04-11 17:21:34.275 Detail, Node009, Notification: ValueChanged
2020-04-11 17:21:34.294 Detail, Node009, Notification: ValueChanged
2020-04-11 17:22:10.400 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:22:10.400 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa5, 0x7c
2020-04-11 17:22:10.400 Detail,
2020-04-11 17:22:10.400 Info, Node046, Sending (Poll) message (Callback ID=0xa5, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa5, 0x7c
2020-04-11 17:22:11.401 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:22:11.401 Detail, Node046, Removing current message
2020-04-11 17:22:11.401 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:22:22.475 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
2020-04-11 17:22:22.475 Detail,   ZW_SEND_DATA delivered to Z-Wave stack
2020-04-11 17:23:11.411 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:23:11.411 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa6, 0x7f
2020-04-11 17:23:11.411 Detail,
2020-04-11 17:23:11.411 Info, Node046, Sending (Poll) message (Callback ID=0xa6, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa6, 0x7f
2020-04-11 17:23:12.411 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:23:12.411 Detail, Node046, Removing current message
2020-04-11 17:23:12.411 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:24:12.421 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:24:12.421 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa7, 0x7e
2020-04-11 17:24:12.421 Detail,
2020-04-11 17:24:12.421 Info, Node046, Sending (Poll) message (Callback ID=0xa7, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa7, 0x7e
2020-04-11 17:24:13.421 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:24:13.421 Detail, Node046, Removing current message
2020-04-11 17:24:13.422 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:24:32.062 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x00, 0xe9
2020-04-11 17:24:32.062 Error, ERROR: ZW_SEND_DATA could not be delivered to Z-Wave stack
2020-04-11 17:24:32.065 Detail, CAN received...triggering resend
2020-04-11 17:24:32.065 Warning, m_currentMsg was NULL when trying to set MaxSendAttempts
2020-04-11 17:24:32.065 Always,
2020-04-11 17:24:32.065 Always, Dumping queued log messages
2020-04-11 17:24:32.065 Always,
2020-04-11 17:24:32.065 Always,
2020-04-11 17:24:32.065 Always, End of queued log message dump
2020-04-11 17:24:32.065 Always,
2020-04-11 17:24:32.065 Detail, WriteMsg CAN m_currentMsg=00000000
2020-04-11 17:24:32.181 Detail,   Received: 0x01, 0x07, 0x00, 0x13, 0xa6, 0x01, 0x00, 0x0c, 0x40
2020-04-11 17:24:32.181 Detail,   ZW_SEND_DATA Request with callback ID 0xa6 received (expected 0x00)
2020-04-11 17:24:32.181 Warning, WARNING: Unexpected Callback ID received
2020-04-11 17:25:13.430 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:25:13.430 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa8, 0x71
2020-04-11 17:25:13.431 Detail,
2020-04-11 17:25:13.431 Info, Node046, Sending (Poll) message (Callback ID=0xa8, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa8, 0x71
2020-04-11 17:25:14.431 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:25:14.431 Detail, Node046, Removing current message
2020-04-11 17:25:14.431 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:25:20.219 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
2020-04-11 17:25:20.219 Detail,   ZW_SEND_DATA delivered to Z-Wave stack
2020-04-11 17:25:54.627 Detail,   Received: 0x01, 0x07, 0x00, 0x13, 0xa8, 0x01, 0x0d, 0x72, 0x3d
2020-04-11 17:25:54.627 Detail,   ZW_SEND_DATA Request with callback ID 0xa8 received (expected 0x00)
2020-04-11 17:25:54.628 Warning, WARNING: Unexpected Callback ID received
2020-04-11 17:26:14.441 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:26:14.441 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa9, 0x70
2020-04-11 17:26:14.441 Detail,
2020-04-11 17:26:14.441 Info, Node046, Sending (Poll) message (Callback ID=0xa9, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xa9, 0x70
2020-04-11 17:26:15.442 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:26:15.442 Detail, Node046, Removing current message
2020-04-11 17:26:15.442 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:27:15.451 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:27:15.451 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xaa, 0x73
2020-04-11 17:27:15.451 Detail,
2020-04-11 17:27:15.451 Info, Node046, Sending (Poll) message (Callback ID=0xaa, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xaa, 0x73
2020-04-11 17:27:16.452 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:27:16.452 Detail, Node046, Removing current message
2020-04-11 17:27:16.452 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:27:29.775 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x00, 0xe9
2020-04-11 17:27:29.775 Error, ERROR: ZW_SEND_DATA could not be delivered to Z-Wave stack
2020-04-11 17:27:29.777 Detail, CAN received...triggering resend
2020-04-11 17:27:29.777 Warning, m_currentMsg was NULL when trying to set MaxSendAttempts
2020-04-11 17:27:29.777 Always,
2020-04-11 17:27:29.778 Always, Dumping queued log messages
2020-04-11 17:27:29.778 Always,
2020-04-11 17:27:29.778 Always,
2020-04-11 17:27:29.778 Always, End of queued log message dump
2020-04-11 17:27:29.778 Always,
2020-04-11 17:27:29.778 Detail, WriteMsg CAN m_currentMsg=00000000
2020-04-11 17:28:16.461 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:28:16.461 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xab, 0x72
2020-04-11 17:28:16.462 Detail,
2020-04-11 17:28:16.462 Info, Node046, Sending (Poll) message (Callback ID=0xab, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xab, 0x72
2020-04-11 17:28:17.462 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:28:17.463 Detail, Node046, Removing current message
2020-04-11 17:28:17.463 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:28:17.928 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x01, 0xe8
2020-04-11 17:28:17.929 Detail,   ZW_SEND_DATA delivered to Z-Wave stack
2020-04-11 17:28:52.307 Detail,   Received: 0x01, 0x07, 0x00, 0x13, 0xab, 0x01, 0x0d, 0x6f, 0x23
2020-04-11 17:28:52.308 Detail,   ZW_SEND_DATA Request with callback ID 0xab received (expected 0x00)
2020-04-11 17:28:52.308 Warning, WARNING: Unexpected Callback ID received
2020-04-11 17:29:17.470 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:29:17.470 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xac, 0x75
2020-04-11 17:29:17.470 Detail,
2020-04-11 17:29:17.471 Info, Node046, Sending (Poll) message (Callback ID=0xac, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xac, 0x75
2020-04-11 17:29:18.471 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:29:18.471 Detail, Node046, Removing current message
2020-04-11 17:29:18.471 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:30:18.477 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:30:18.477 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xad, 0x74
2020-04-11 17:30:18.478 Detail,
2020-04-11 17:30:18.478 Info, Node046, Sending (Poll) message (Callback ID=0xad, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xad, 0x74
2020-04-11 17:30:19.478 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:30:19.478 Detail, Node046, Removing current message
2020-04-11 17:30:19.478 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:30:27.539 Detail,   Received: 0x01, 0x04, 0x01, 0x13, 0x00, 0xe9
2020-04-11 17:30:27.539 Error, ERROR: ZW_SEND_DATA could not be delivered to Z-Wave stack
2020-04-11 17:30:27.541 Detail, CAN received...triggering resend
2020-04-11 17:30:27.541 Warning, m_currentMsg was NULL when trying to set MaxSendAttempts
2020-04-11 17:30:27.541 Always,
2020-04-11 17:30:27.541 Always, Dumping queued log messages
2020-04-11 17:30:27.541 Always,
2020-04-11 17:30:27.541 Always,
2020-04-11 17:30:27.541 Always, End of queued log message dump
2020-04-11 17:30:27.541 Always,
2020-04-11 17:30:27.541 Detail, WriteMsg CAN m_currentMsg=00000000
2020-04-11 17:31:19.480 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:31:19.480 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xae, 0x77
2020-04-11 17:31:19.480 Detail,
2020-04-11 17:31:19.480 Info, Node046, Sending (Poll) message (Callback ID=0xae, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xae, 0x77
2020-04-11 17:31:20.480 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:31:20.480 Detail, Node046, Removing current message
2020-04-11 17:31:20.480 Detail, Node046, Notification: Notification - TimeOut
2020-04-11 17:31:43.042 Info, Node040, Value::Set - COMMAND_CLASS_SWITCH_MULTILEVEL - Dimming Duration - 5 - 1 - 255
2020-04-11 17:31:43.042 Detail, Node040, Refreshed Value: old value=255, new value=255, type=byte
2020-04-11 17:31:43.042 Detail, Node040, Changes to this value are not verified
2020-04-11 17:31:43.042 Detail, Node040, Notification: ValueChanged
2020-04-11 17:31:43.044 Info, Node040, Value::Set - COMMAND_CLASS_SWITCH_MULTILEVEL - Level - 0 - 1 - 255
2020-04-11 17:31:43.044 Info, Node040, SwitchMultilevel::Set - Setting to level 255
2020-04-11 17:31:43.044 Info, Node040,   Duration: Default
2020-04-11 17:31:43.044 Detail, Node040, Queuing (Send) MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Set (Node=40): 0x01, 0x0f, 0x00, 0x13, 0x28, 0x08, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x01, 0xff, 0xff, 0x25, 0xaf, 0x03
2020-04-11 17:31:43.044 Detail, Node040, Queuing (Send) MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Get (Node=40): 0x01, 0x0d, 0x00, 0x13, 0x28, 0x06, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x02, 0x25, 0xb0, 0x13
2020-04-11 17:31:43.062 Detail,
2020-04-11 17:31:43.062 Info, Node040, Sending (Send) message (Callback ID=0xaf, Expected Reply=0x13) - MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Set (Node=40): 0x01, 0x0f, 0x00, 0x13, 0x28, 0x08, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x01, 0xff, 0xff, 0x25, 0xaf, 0x03
2020-04-11 17:31:44.063 Error, Node040, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:31:44.063 Detail, Node040, Removing current message
2020-04-11 17:31:44.063 Detail, Node040, Notification: Notification - TimeOut
2020-04-11 17:31:44.072 Detail,
2020-04-11 17:31:44.073 Info, Node040, Sending (Send) message (Callback ID=0xb0, Expected Reply=0x04) - MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Get (Node=40): 0x01, 0x0d, 0x00, 0x13, 0x28, 0x06, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x02, 0x25, 0xb0, 0x13
2020-04-11 17:31:45.073 Error, Node040, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:31:45.073 Detail, Node040, Removing current message
2020-04-11 17:31:45.073 Detail, Node040, Notification: Notification - TimeOut
2020-04-11 17:31:48.366 Info, Node040, Value::Set - COMMAND_CLASS_SWITCH_MULTILEVEL - Dimming Duration - 5 - 1 - 255
2020-04-11 17:31:48.366 Detail, Node040, Refreshed Value: old value=255, new value=255, type=byte
2020-04-11 17:31:48.366 Detail, Node040, Changes to this value are not verified
2020-04-11 17:31:48.366 Detail, Node040, Notification: ValueChanged
2020-04-11 17:31:48.367 Info, Node040, Value::Set - COMMAND_CLASS_SWITCH_MULTILEVEL - Level - 0 - 1 - 29
2020-04-11 17:31:48.367 Info, Node040, SwitchMultilevel::Set - Setting to level 29
2020-04-11 17:31:48.368 Info, Node040,   Duration: Default
2020-04-11 17:31:48.368 Detail, Node040, Queuing (Send) MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Set (Node=40): 0x01, 0x0f, 0x00, 0x13, 0x28, 0x08, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x01, 0x1d, 0xff, 0x25, 0xb1, 0xff
2020-04-11 17:31:48.368 Detail, Node040, Queuing (Send) MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Get (Node=40): 0x01, 0x0d, 0x00, 0x13, 0x28, 0x06, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x02, 0x25, 0xb2, 0x11
2020-04-11 17:31:48.384 Detail,
2020-04-11 17:31:48.384 Info, Node040, Sending (Send) message (Callback ID=0xb1, Expected Reply=0x13) - MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Set (Node=40): 0x01, 0x0f, 0x00, 0x13, 0x28, 0x08, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x01, 0x1d, 0xff, 0x25, 0xb1, 0xff
2020-04-11 17:31:49.384 Error, Node040, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:31:49.384 Detail, Node040, Removing current message
2020-04-11 17:31:49.385 Detail, Node040, Notification: Notification - TimeOut
2020-04-11 17:31:49.391 Detail,
2020-04-11 17:31:49.391 Info, Node040, Sending (Send) message (Callback ID=0xb2, Expected Reply=0x04) - MultiChannel Encapsulated (instance=1): SwitchMultilevelCmd_Get (Node=40): 0x01, 0x0d, 0x00, 0x13, 0x28, 0x06, 0x60, 0x0d, 0x01, 0x01, 0x26, 0x02, 0x25, 0xb2, 0x11
2020-04-11 17:31:50.391 Error, Node040, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:31:50.392 Detail, Node040, Removing current message
2020-04-11 17:31:50.392 Detail, Node040, Notification: Notification - TimeOut
2020-04-11 17:32:20.490 Detail, Node046, Polling: COMMAND_CLASS_SENSOR_MULTILEVEL index = 1 instance = 1 (poll queue has 0 messages)
2020-04-11 17:32:20.490 Detail, Node046, Queuing (Poll) SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xb3, 0x6a
2020-04-11 17:32:20.490 Detail,
2020-04-11 17:32:20.490 Info, Node046, Sending (Poll) message (Callback ID=0xb3, Expected Reply=0x04) - SensorMultilevelCmd_Get (Node=46): 0x01, 0x09, 0x00, 0x13, 0x2e, 0x02, 0x31, 0x04, 0x25, 0xb3, 0x6a
2020-04-11 17:32:21.491 Error, Node046, ERROR: Dropping command, expected response not received after 1 attempt(s)
2020-04-11 17:32:21.491 Detail, Node046, Removing current message
2020-04-11 17:32:21.491 Detail, Node046, Notification: Notification - TimeOut

tmjpugh · April 11, 2020, 3:58pm

You have backup.

Try restore zwcfg_*.xml.
Backup current first

Tyfoon · April 12, 2020, 3:00pm

Driving me nuts.

Now also tried:

Different location zwave stick (maybe there is interference)
Used backup of zwcfg_*.xml
Restored several old versions
Deleted .db (as I remember very slow Zwave when database got to big)

All no change.

The weird thing is that zwave network is active but only limited number of devices are reachable. After a restart they are also not the same devices but different devices! Now I have 11 devices active, restart before that 6. p.s. most of my devices are mains powered, I know the battery ones can take a while.

p.s. the not responding devices are marked as complete in zwave management (not as unavailable as you might expect)

Any ideas?

tmjpugh · April 12, 2020, 5:53pm

Yeah…you might be best unpair and re-pair those if it is not a lot…

After, you can delete old devices from zwave then you must rename (entity_id and friendly) to match old device so automations,Alexa, home pick them up likely never gone

EDIT
Zwave stick may have dropped the entities.
If you have backuo of that you should restore it
Else
Likely you must start from scratch. Just realized 2/3 of your devices still missing.

For future keep backup of zwave stick + zwave config file. This is matched pair(from my observation) so if add device you need backup both again

Tyfoon · April 12, 2020, 8:54pm

Well, it’s solved…

It was a zwave node gone bad and spamming the zwave network. After disabling that one all was back to normal. I noticed the CAN errors in the log and they are an indication of a flooded zwave network.

omayhemo · April 26, 2020, 3:27pm

How did you determine which node was jamming your mesh?

Tyfoon · April 26, 2020, 3:48pm

The zwave log is a good place to look. In this case the painter also dropped a motion sensor few days before. So the combination of the two was enough to raise suspicion

omayhemo · April 26, 2020, 4:35pm

TL:DR - I don’t know what to look for, exactly.

I do look at the logs, for sure. They’re a mess. Lots of CAN, lots of timeouts, but eventually (20-30 min after reboot) the mesh will get itself back together. Currently, no neighbors are ever reported on any node, this started with (I think) a second reboot before z-wave finished initializing back on .107.1. Because of all these anomalous things, I’m considering rebuilding the whole mesh from start. But, I’m concerned it’ll just happen again and again. So, I’d like to learn to diagnose it, if possible.

Tyfoon · April 26, 2020, 5:50pm

I’m using hass now for +4 years and never had to rebuild the zwave network. Did you also try restoring the zwave xml file?

Do you have some working nodes?
Did you install the zwave graph tool in Hass? Quite usefull.

omayhemo · April 27, 2020, 12:40pm

Ya, I’ve done lot of all of that. Restores, starting from scratch etc. It continues to happen.
I have all working nodes, but two, which are undefined, and never go failed and cannot be removed. The mesh works well after it settles down after a reboot.

I have the graph tool. It doesn’t work because HA isn’t reporting neighbors.

Tyfoon · April 27, 2020, 12:50pm

And if you are in zwave config and select that node, what does it say? If it’s undefined and never changes anymore I would presume the node is indeed dead. That happens to me also now and then. with 40+ zwave devices about once a year. I also have a few dead ones still in the network although they physically do not exist anymore. Actually never tried to remove these.

Below is how my zwave map

omayhemo · April 27, 2020, 12:53pm

No neighbors are listed in either node information card or the graph.
As you can see, I have a lot. There’s no way they would be working without neighbors.

Tyfoon · April 28, 2020, 6:56am

Looks a lot like mine when I had the issue I described. Did you try taking one of the grey (mains power) devices and tries to heal/test them to see what happens?

omayhemo · April 28, 2020, 3:03pm

Holy funky spit. The combination is the key, one by one. I’ve been trying to heal the whole network, which I think is broken, flooding the mesh. Heal Node discovers neighbors, Test Node immediately after prompts the newly discovered neighbors to be reported in the UI.

Tyfoon · April 28, 2020, 3:47pm

Sounds like progress

omayhemo · April 29, 2020, 7:41pm

Doesn’t survive a reboot. I think there’s a problem with the way z-wave initializes in HA. I believe it queues too many queries at once, resulting in a timeout, at which time it decides to wipe out the routes on the device and start from scratch, which doesn’t work because it’s overwhelmed the mesh. Every reboot seems to result in z-wave routes getting nuked and I can only fix it node by node.

johnflorin · May 4, 2020, 2:17pm

I’ve been having the same problems and can’t figure out how to resolve them…is ZWave simply broken in HA for some people? Because this just sounds broken…I’ve tried using the zwcfg file from a snapshot, deleting it, tried restarting HA, the VM, the PC, nada.

I have a feeling that people who say ZWave works well in HA either have only ZWave Plus devices, a go-between like the Vera or a limited number of devices. I have ~40 (mostly non-Plus devices) and the ZWave situation has gotten much, much worse lately…even before it would sometimes act up for no reason, but in the past couple of months every single restart tends to lead to it not working in the manner described in the OP. Sometimes VM snapshots have resolved the issue, other times HA snapshots, but at this moment it just seems done for.

Also, I really disagree with recommendations like “pair it again”, with IoT devices this is often a major issue…for example, I received some Popp switches which, due to the way their button presses are handled in HA, were an absolute nightmare to pair (had to set each button in Scene mode, etc.), so if I have to re-pair them (they’re all gone in the current instance), I’d rather chuck them all in the recycling bin. Also, there are some roller controllers buried behind wall sockets…they’re thankfully not gone this time, but again, I’m not re-pairing them ever again, if I have to get behind the wall sockets I am replacing them with Shellys or some other sane technology, which doesn’t require this kind of voodoo, because it truly is voodoo: when a Fibaro door sensor wants you to provide your own magnet to reset it, it goes beyond any sane realm.

Sorry about the rant, but in searching for a solution to the current mess I’ve seen several threads exactly like this, with no solution in sight at all. There’s also this issue opened on Github - https://github.com/home-assistant/core/issues/33486 , dozens of comments with the only conclusions being “the new ZWave broker is forthcoming”, “replace the cfg file” (which totally doesn’t work for many people) or “restart till it works, mine did after 5 tries”.

This is crazy

omayhemo · May 4, 2020, 2:25pm

From my perspective, z-wave in HA is extremely fragile and suffers a lot of problems. I’ve learned how to be very very careful and avoid some of the pitfalls, but I’d really like to see this get way more attention than it gets now. I share your frustration and I’m hoping the re-write for the v1.6 will fix things.

johnflorin · May 4, 2020, 3:28pm

The zwave cfg thing finally did work for me, but only like this (I’m on an Ubuntu VM which contains HA):

samba into config folder to copy exported and renamed zwcfg.xml from HA snapshot
stop HA from the Server Controls area
SSH into the HA-containing Ubuntu in /urs/share/hassio/homeassistant and delete zwcfg.xml and rename my backed up one
curl command to reinstall HA (maybe it would have restarted when restarting the Linux, but previous experience has shown me that the Stop command is pretty final, not all the Docker containers are recreated on restart)

So…yay? I’ll keep the list of steps for such future messups and hope that the new HA ZWave implementation is worlds better than the current one (from what I’ve read, it does look like it).