As context, I recently scrapped my HA system and started over from scratch. I have a SkyConnect plugged into a Beelink minicomputer, and I run HA as a VM on Proxmox. I’ve spent several hours over the last couple days adding about a hundred Zigbee devices back to my network using Zigbee2MQTT (Z2M).
This evening I was setting up my lighting scenes, tweaking individual bulbs and saving settings. I’d been at it for maybe 15-20 minutes when one of the bulbs stopped responding. Weird. Rebooted, no change.
Checked Z2M and it’s looping an error. Here’s what I think is the relevant portion of the log, starting from the exact moment Z2M stopped recognizing my SkyConnect adapter:
[2025-02-22 03:46:51] info: z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/Master Bedroom Fan Light 3', payload '{"brightness":255,"color":{"x":0.4599,"y":0.4106},"color_mode":"color_temp","color_temp":370,"energy":19.04,"linkquality":168,"power":0.2,"state":"OFF","update":{"installed_version":71,"latest_version":71,"state":"idle"}}'
[2025-02-22 03:46:59] info: z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/Living Room Fan Light 1', payload '{"brightness":127,"color_mode":"color_temp","color_temp":383,"energy":30.94,"linkquality":236,"power":6.5,"state":"ON","update":{"installed_version":-1,"latest_version":-1,"state":"idle"}}'
**[2025-02-22 03:47:05] error: zh:ember:ezsp: ERROR Transaction failure; status=ASH_ERROR_TIMEOUTS. Last Frame: [FRAME: ID=52:"SEND_UNICAST" Seq=46 Len=7].**
**[2025-02-22 03:47:05] error: zh:ember: Adapter fatal error: ASH_ERROR_TIMEOUTS**
[2025-02-22 03:47:05] info: zh:ember:uart:ash: ASH COUNTERS since last clear:
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Total frames: RX=15222, TX=19619
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Cancelled : RX=0, TX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: DATA frames : RX=15214, TX=4400
[2025-02-22 03:47:05] info: zh:ember:uart:ash: DATA bytes : RX=341154, TX=115646
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Retry frames: RX=0, TX=3
[2025-02-22 03:47:05] info: zh:ember:uart:ash: ACK frames : RX=3, TX=15215
[2025-02-22 03:47:05] info: zh:ember:uart:ash: NAK frames : RX=3, TX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: nRdy frames : RX=0, TX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: CRC errors : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Comm errors : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Length < minimum: RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Length > maximum: RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Bad controls : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Bad lengths : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Bad ACK numbers : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Out of buffers : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Retry dupes : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: Out of sequence : RX=0
[2025-02-22 03:47:05] info: zh:ember:uart:ash: ACK timeouts : RX=0
[2025-02-22 03:47:05] error: zh:ember:ezsp: ERROR Transaction failure; status=NOT_CONNECTED. Last Frame: [FRAME: ID=52:"SEND_UNICAST" Seq=46 Len=7].
[2025-02-22 03:47:05] error: zh:ember:ezsp: ERROR Transaction failure; status=NOT_CONNECTED. Last Frame: [FRAME: ID=52:"SEND_UNICAST" Seq=46 Len=7].
[2025-02-22 03:47:08] error: z2m: EventBus error 'OTAUpdate/deviceMessage': CommandResponse 0xb0ce181400084fb0/1 genOta.queryNextImageResponse({"status":152}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":1,"reservedBits":0,"writeUndiv":false}) failed (~x~> [ZCL to=0xb0ce181400084fb0:36049 apsFrame={"profileId":260,"clusterId":25,"sourceEndpoint":1,"destinationEndpoint":1,"options":4352,"groupId":0,"sequence":0}] Failed to send request with status=NETWORK_DOWN.)
[2025-02-22 03:47:08] error: z2m: EventBus error 'OTAUpdate/deviceMessage': CommandResponse 0xb0ce18140000db78/1 genOta.queryNextImageResponse({"status":152}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":true,"direction":1,"reservedBits":0,"writeUndiv":false}) failed (~x~> [ZCL to=0xb0ce18140000db78:7516 apsFrame={"profileId":260,"clusterId":25,"sourceEndpoint":1,"destinationEndpoint":1,"options":4352,"groupId":0,"sequence":0}] Failed to send request with status=NETWORK_DOWN.)
[2025-02-22 03:47:10] error: z2m: Publish 'set' 'brightness' to 'Office Ceiling Light 1' failed: 'Error: ZCL command 0xa4c138449df932ff/1 genLevelCtrl.moveToLevelWithOnOff({"level":107,"transtime":0}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false}) failed (~x~> [ZCL to=0xa4c138449df932ff:13502 apsFrame={"profileId":260,"clusterId":8,"sourceEndpoint":1,"destinationEndpoint":1,"options":4416,"groupId":0,"sequence":0}] Failed to send request with status=NETWORK_DOWN.)'
[2025-02-22 03:47:18] error: z2m: Publish 'set' 'brightness' to 'Office Ceiling Light 2' failed: 'Error: ZCL command 0xa4c138ba9df7942d/1 genLevelCtrl.moveToLevelWithOnOff({"level":134,"transtime":0}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"reservedBits":0,"writeUndiv":false}) failed (~x~> [ZCL to=0xa4c138ba9df7942d:54578 apsFrame={"profileId":260,"clusterId":8,"sourceEndpoint":1,"destinationEndpoint":1,"options":4416,"groupId":0,"sequence":0}] Failed to send request with status=NETWORK_DOWN.)'
[2025-02-22 03:47:21] info: zh:ember:uart:ash: Port closed.
[2025-02-22 03:47:21] info: zh:ember:uart:ash: ======== ASH stopped ========
[2025-02-22 03:47:21] info: zh:ember:ezsp: ======== EZSP stopped ========
[2025-02-22 03:47:21] info: zh:ember: ======== Ember Adapter Stopped ========
[2025-02-22 03:47:21] error: z2m: Adapter disconnected, stopping
[2025-02-22 03:47:21] info: z2m:mqtt: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload '{"state":"offline"}'
[2025-02-22 03:47:21] info: z2m: Disconnecting from MQTT server
[2025-02-22 03:47:21] info: z2m: Stopping zigbee-herdsman...
[2025-02-22 03:47:21] info: z2m: Stopped zigbee-herdsman
[2025-02-22 03:47:21] info: z2m: Stopped Zigbee2MQTT
[03:47:21] INFO: Preparing to start...
[03:47:21] INFO: Socat not enabled
[03:47:21] INFO: Starting Zigbee2MQTT...
Starting Zigbee2MQTT without watchdog.
[2025-02-22 03:47:22] info: z2m: Logging to console, file (filename: log.log)
[2025-02-22 03:47:22] info: z2m: Starting Zigbee2MQTT version 2.1.1 (commit #unknown)
[2025-02-22 03:47:22] info: z2m: Starting zigbee-herdsman (3.2.5)
[2025-02-22 03:47:22] error: z2m: Error while starting zigbee-herdsman
[2025-02-22 03:47:22] error: z2m: Failed to start zigbee-herdsman
[2025-02-22 03:47:22] error: z2m: Check https://www.zigbee2mqtt.io/guide/installation/20_zigbee2mqtt-fails-to-start_crashes-runtime.html for possible solutions
[2025-02-22 03:47:22] error: z2m: Exiting...
[2025-02-22 03:47:22] error: z2m: Error: USB adapter discovery error (No valid USB adapter found). Specify valid 'adapter' and 'port' in your configuration.
at discoverAdapter (/app/node_modules/.pnpm/[email protected]/node_modules/zigbee-herdsman/src/adapter/adapterDiscovery.ts:523:15)
at processTicksAndRejections (node:internal/process/task_queues:105:5)
at Function.create (/app/node_modules/.pnpm/[email protected]/node_modules/zigbee-herdsman/src/adapter/adapter.ts:68:33)
at Controller.start (/app/node_modules/.pnpm/[email protected]/node_modules/zigbee-herdsman/src/controller/controller.ts:132:24)
at Zigbee.start (/app/lib/zigbee.ts:69:27)
at Controller.start (/app/lib/controller.ts:142:13)
at start (/app/index.js:161:5)
[2025-02-22 03:47:22] error: z2m: If this happens after updating to Zigbee2MQTT 2.0.0, see https://github.com/Koenkk/zigbee2mqtt/discussions/24364
[03:47:23] INFO: Preparing to start...
After that, it just loops that Zigbee herdsman error sequence over and over.
Went to my HA server and checked the actual physical connections to make sure they were okay - still no change. I was flipping out because I just tore my whole network down and put it back together, and I was going to jump out a window if I had to re-pair a hundred Zigbee devices again.
I restored to a Proxmox backup of the full virtual machine - thankfully I had one just a few hours old, that I made right after I finished pairing everything because I’m neurotic like that - and I’ll be damned, Z2M is working just fine again. I was super surprised; this seemed an awful lot like a hardware issue, especially with it borking right in the middle of active use.
The immediate problem is solved - I have my backup, not too much work to re-do. But I’m scratching my head about what could have happened here. Anyone have any thoughts?