Zigbee2MQTT + CC2531 constantly stops working

keex · June 30, 2021, 9:47am

I have a problem with my whole Zigbee network becoming unresponsive after running for a while. This happens seemingly at random. The longest I’ve seen it run fine is a couple of days, perhaps a week, after which I need to restart the Pi to make it work again (also, sometimes it gets stuck after a restart and I need to unplug the CC2531 stick physically to make the host boot).

So, for example, today at 5:21:59 I see the last unique data point from Bathroom temp sensor, after which the chart goes flat, i.e. the Zigbee controller crashed. It had also crashed before, so I restarted it at ca. 20:00 and it was working up until 5:21:

The logs around that time don’t really show anything interesting, except that the plug started throwing red into Zigbee2mqtt logs after a few minutes:

Zigbee2MQTT:info  2021-06-30 05:39:49: MQTT publish: topic 'zigbee2mqtt/Plug 1', payload '{"current":0,"energy":3.75,"linkquality":10,"power":0,"state":"OFF","voltage":221}'
Zigbee2MQTT:info  2021-06-30 05:40:27: MQTT publish: topic 'zigbee2mqtt/Plug 1', payload '{"current":0,"energy":3.75,"linkquality":13,"power":0,"state":"OFF","voltage":221}'
Zigbee2MQTT:info  2021-06-30 05:40:49: MQTT publish: topic 'zigbee2mqtt/Plug 1', payload '{"current":0,"energy":3.75,"linkquality":10,"power":0,"state":"OFF","voltage":221}'
Zigbee2MQTT:error 2021-06-30 08:18:30: Publish 'set' 'state' to 'Plug 1' failed: 'Error: Command 0x588e81fffed97e41/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'MAC channel access failure' (225))'
Zigbee2MQTT:info  2021-06-30 08:18:30: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Publish 'set' 'state' to 'Plug 1' failed: 'Error: Command 0x588e81fffed97e41/1 genOnOff.on({}, {\"timeout\":10000,\"disableResponse\":false,\"disableRecovery\":false,\"disableDefaultResponse\":false,\"direction\":0,\"srcEndpoint\":null,\"reservedBits\":0,\"manufacturerCode\":null,\"transactionSequenceNumber\":null,\"writeUndiv\":false}) failed (Data request failed with error: 'MAC channel access failure' (225))'","meta":{"friendly_name":"Plug 1"},"type":"zigbee_publish_error"}'
Zigbee2MQTT:error 2021-06-30 08:18:45: Publish 'set' 'state' to 'Plug 1' failed: 'Error: Command 0x588e81fffed97e41/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'MAC channel access failure' (225))'
Zigbee2MQTT:info  2021-06-30 08:18:45: MQTT publish: topic 'zigbee2mqtt/bridge/log', payload '{"message":"Publish 'set' 'state' to 'Plug 1' failed: 'Error: Command 0x588e81fffed97e41/1 genOnOff.on({}, {\"timeout\":10000,\"disableResponse\":false,\"disableRecovery\":false,\"disableDefaultResponse\":false,\"direction\":0,\"srcEndpoint\":null,\"reservedBits\":0,\"manufacturerCode\":null,\"transactionSequenceNumber\":null,\"writeUndiv\":false}) failed (Data request failed with error: 'MAC channel access failure' (225))'","meta":{"friendly_name":"Plug 1"},"type":"zigbee_publish_error"}'
Zigbee2MQTT:error 2021-06-30 08:18:53: Publish 'set' 'state' to 'Plug 1' failed: 'Error: Command 0x588e81fffed97e41/1 genOnOff.on({}, {"timeout":10000,"disableResponse":false,"disableRecovery":false,"disableDefaultResponse":false,"direction":0,"srcEndpoint":null,"reservedBits":0,"manufacturerCode":null,"transactionSequenceNumber":null,"writeUndiv":false}) failed (Data request failed with error: 'MAC channel access failure' (225))'

Mosquitto logs after a fresh boot (the devices seem to be actually working despite the socket errors. Not sure if it’s relevant):

1625043854: New connection from 127.0.0.1 on port 1883.
1625043854: Socket error on client <unknown>, disconnecting.
1625043884: New connection from 172.30.32.1 on port 1883.
1625043884: New client connected from 172.30.32.1 as 78gyAuLq4QZPzuZ49K3XwB (p2, c1, k60, u'homeassistant').
1625044055: New connection from 172.30.32.2 on port 1883.
1625044055: Socket error on client <unknown>, disconnecting.
1625044091: New connection from 172.30.32.1 on port 1883.
1625044091: New client connected from 172.30.32.1 as mqttjs_e1bdad43 (p2, c1, k60, u'homeassistant').
1625044175: New connection from 172.30.32.2 on port 1883.
1625044175: Socket error on client <unknown>, disconnecting.
1625044295: New connection from 172.30.32.2 on port 1883.
1625044295: Socket error on client <unknown>, disconnecting.
1625044415: New connection from 172.30.32.2 on port 1883.

Supervisor log also has some indication of trouble:

21-06-30 11:48:06 WARNING (MainThread) [supervisor.addons.options] Unknown option 'new_api' for Zigbee2mqtt (7ad98f9c_zigbee2mqtt)
21-06-30 11:48:06 WARNING (MainThread) [supervisor.addons.options] Unknown option 'restartdelay' for Zigbee2mqtt (7ad98f9c_zigbee2mqtt)
21-06-30 11:48:06 WARNING (MainThread) [supervisor.addons.options] Unknown option 'initialdelay' for Zigbee2mqtt (7ad98f9c_zigbee2mqtt)
21-06-30 11:49:09 WARNING (MainThread) [supervisor.addons.options] Option 'anonymous' does not exist in the schema for Mosquitto broker (core_mosquitto)

Core log looks provides some additional hints, but I’m not sure what to make of them. Maybe :

[34771.737100] usb 2-2: new SuperSpeed Gen 1 USB device number 32 using xhci_hcd
[34771.790545] usb 2-2: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice=62.05
[34771.790558] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[34771.790569] usb 2-2: Product: USB3.1 Hub
[34771.790579] usb 2-2: Manufacturer: GenesysLogic
[34771.797290] hub 2-2:1.0: USB hub found
[34771.799131] hub 2-2:1.0: 4 ports detected
[34772.117056] usb 2-2.4: new SuperSpeed Gen 1 USB device number 33 using xhci_hcd
[34772.160534] usb 2-2.4: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice=62.05
[34772.160554] usb 2-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[34772.160570] usb 2-2.4: Product: USB3.1 Hub
[34772.160586] usb 2-2.4: Manufacturer: GenesysLogic
[34772.169833] hub 2-2.4:1.0: USB hub found
[34772.171718] hub 2-2.4:1.0: 4 ports detected
[34780.466506] usb 2-2: USB disconnect, device number 32
[34780.466534] usb 2-2.4: USB disconnect, device number 33
[34781.357235] usb 2-2: new SuperSpeed Gen 1 USB device number 34 using xhci_hcd
[34781.400901] usb 2-2: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice=62.05
[34781.400921] usb 2-2: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[34781.400937] usb 2-2: Product: USB3.1 Hub
[34781.400953] usb 2-2: Manufacturer: GenesysLogic
[34781.411984] hub 2-2:1.0: USB hub found
[34781.414053] hub 2-2:1.0: 4 ports detected
[34781.737202] usb 2-2.4: new SuperSpeed Gen 1 USB device number 35 using xhci_hcd
[34781.790674] usb 2-2.4: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice=62.05
[34781.790697] usb 2-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[34781.790716] usb 2-2.4: Product: USB3.1 Hub
[34781.790735] usb 2-2.4: Manufacturer: GenesysLogic
[34781.800849] hub 2-2.4:1.0: USB hub found
[34781.802929] hub 2-2.4:1.0: 4 ports detected
[34966.799749] usb 2-2: USB disconnect, device number 34
[34966.799777] usb 2-2.4: USB disconnect, device number 35

On startup, the following is found in the logs:

# Supervisor:
21-06-30 12:04:04 INFO (SyncWorker_3) [supervisor.docker.interface] Attaching to zigbee2mqtt/zigbee2mqtt-armv7 with version 1.18.1-1

# Host
[    2.156389] hub 2-2:1.0: USB hub found
[    2.158210] hub 2-2:1.0: 4 ports detected
[    2.285762] usb 1-1.1: new full-speed USB device number 3 using xhci_hcd
[    2.434503] usb 1-1.1: New USB device found, idVendor=0451, idProduct=16a8, bcdDevice= 0.09
[    2.434529] usb 1-1.1: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[    2.434545] usb 1-1.1: Product: TI CC2531 USB CDC
[    2.434558] usb 1-1.1: Manufacturer: Texas Instruments
[    2.434571] usb 1-1.1: SerialNumber: __0X00124B0014DA2C38
...
[    2.865771] usb 1-1.2: new high-speed USB device number 5 using xhci_hcd
[    2.966342] usb 1-1-port2: attempt power cycle
...
[    3.625800] usb 1-1.2: new full-speed USB device number 6 using xhci_hcd
[    3.734456] usb 1-1.2: not running at top speed; connect to a high speed hub
...
# This seems to keep repeating:
[ 1832.710297] usb 2-2.4: new SuperSpeed Gen 1 USB device number 31 using xhci_hcd
[ 1832.753761] usb 2-2.4: New USB device found, idVendor=05e3, idProduct=0626, bcdDevice=62.05
[ 1832.753780] usb 2-2.4: New USB device strings: Mfr=1, Product=2, SerialNumber=0
[ 1832.753797] usb 2-2.4: Product: USB3.1 Hub
[ 1832.753812] usb 2-2.4: Manufacturer: GenesysLogic
[ 1832.763238] hub 2-2.4:1.0: USB hub found
[ 1832.765117] hub 2-2.4:1.0: 4 ports detected

I have the following setup:

Raspberry Pi 4 8GB running Home Assistant OS 2021.6.6
CC2531 / CC2540 board from Aliexpress with an antenna plugged directly into the Pi. I’ve also tried using a powered USB hub, but it seems more stable being plugged in directly. I used the official guide for flashing firmware on it. Coordinator firmware version: '{"meta":{"maintrel":3,"majorrel":2,"minorrel":6,"product":0,"revision":20190608,"transportrev":2},"type":"zStack12"}'
Zigbee2MQTT + Mosquitto Broker 1.6.12
A few Zigbee devices (all ordered from Aliexpress):
- 2x Lonsonho Tuya ZigBee Smart Plug
- 1x XIAOMI Mija Light Sensor
- 2x SONOFF SNZB-03 Motion Sensors
- 4x Aqara Temperature Sensors (3 of which I added yesterday)
Probably not related, but I also have Z-Stick Gen5 for ZWave, which is plugged into the usb Hub and working perfectly.

To me, it seems that the CC2531 board is to blame (hence I categorized it under Hardware rather than Configuration → Zigbee), but I’ve run out of things to try and ways to debug. I’d highly appreciate some nudges on what else to troubleshoot and try. Alternative Zigbee sticks (Conbee and such) are hard to come by here, so I am trying to understand whether I can fix the current setup rather than replacing it with another one.

callifo · June 30, 2021, 11:48am

I had lots of issues too, it seemed the actual adapter was crashing/dying. The only was to fix it was as you found reboot, or just removing power from the stick, fixed it.

Not that its a great fix, but I ended up going to the Zig-ah stick instead.

duceduc · June 30, 2021, 12:01pm

Are you using a SD Card on the Pi? A faulty and or almost ready to crap out sd card can cause a lot of issue. If you have another sd card to test, try that.

Vlad · June 30, 2021, 1:55pm

Check Zigbee adapter.
“Previously the Texas Instruments CC2531 was the most recommended adapter. This adapter is not recommended anymore since it’s not very powerfull, doesn’t have good range and struggles with larger network (20+ devices).”
Supported adapters

francisp · June 30, 2021, 3:33pm

The latest firmware is 20201128
Try flashing it with the latest

Tinkerer · June 30, 2021, 5:06pm

I had a CC2531 for a long time, and I (too) found out the hard way that from time to time they’ll just lock up and stop responding, or only semi-work.

The answer is to upgrade to a CC2652 based stick (not a Conbee). If you can’t do that then there’s sadly not really any fix, other than what you’ve been doing - when it locks up, pull the stick, and then put it back again a few seconds later.

keex · June 30, 2021, 8:39pm

Thanks! I hadn’t noticed that message. Will look for a CC2652-based stick. Thanks!

Edit: Ordered Slaesh’s CC2652RB-based stick Waiting excitedly now!

MostHated · July 17, 2021, 12:01am

I just had my cc2531 start acting up yesterday. I got that because I was having a bunch of issues dropping connections on my HUSBZ1, and I wanted to go Z2MQTT. It was rock solid until last night. It’s only been a few months since I got it. Now I have to completely unplug it, then it works just long enough for HA Zigbee2Mqtt to start up and send out a few messages, then it craps out a few seconds later.

Based on what everyone is saying, though, I went ahead and ordered a CC2652, but its coming from the UK to the US, and the Z2Mqtt site said there were slow shipping times. I was thinking of also testing out the LAUNCHXL-CC26X2R1 as it looks like it might get here sooner. The things I have read from people who use it are good in terms of stability, but I am just worried about range with no external antenna.

Anyone have any experience with swapping over from a 2531 to that, or that vs the 2562 that might have some thoughts/opinions they might be able to share?

nickrout · July 17, 2021, 12:35am

Yes I shifted to the launch-xl. Glad I did, but i’d probably buy one of tube’s ethernet based co-ordinator.

Tinkerer · July 17, 2021, 7:19am

Don’t confuse Slaesh (not UK based, terrible communication, long shipping times) with Electrolama (UK based, great communication, shipping fast when there’s stock).

I’d second the suggestion to buy from Tube if you’re in North America though.

keex · July 17, 2021, 1:52pm

I received my CC2652RB stick within just over a week after placing the order (shipped to Estonia). It’s working wonderfully and has resolved all my issues, and even some Zigbee devices that I thought were faulty started working perfectly.

MostHated · July 17, 2021, 2:06pm

I saw a lot mention tubes devices, but unfortunately, they are all sold out, so I needed something in the mean time. I ordered another cc2531 from Amazon, which should be here today. Hopefully, it will last long enough for me to sort out a suitable replacement. I will keep an eye out for tube to come back in stock with something and eventually grab one when it is based on all the recommendations I keep seeing.

That is my bad, for some reason I though Slaesh device was coming from the UK. That was the one I had ordered, but after seeing so many mixed reviews of shipping, I may as well keep looking. Ti has the CC26X2R1 in stock, so I may go ahead and grab one for the time being.

nickrout · July 17, 2021, 2:31pm

Tube has a twitter feed which announces when he has stock. https://mobile.twitter.com/Tubeszb

Tinkerer · July 17, 2021, 5:49pm

He’d mentioned on Discord that he hoped to get a delivery of parts on this coming Monday (19th of July) and start getting some products out during the week.

miroslavpetrov · November 26, 2021, 4:56am

I have setup 3 zigbee networks using zigbee2mqtt. All three use a cc2531 as a coordinator. Two of these setups are in remote locations and I haven’t updated zigbee2mqtt for probably more than 2 years. These two work like a charm and have no crashing issue.

On the other hand, the setup I have at home is regularly updated as zigbee2mqtt runs as an addon in home assistatant. 2-3 months ago I started experiencing these crashes and the only solution is to remove the CC2531 stick and plug it in again and then restart the zigbee2mqtt addon.

I cannot understand how the **** the old versions work fine and the new ones don’t…

I also would like to mention that the coordinator that is crashing is responsible for less devices than the other two which work fine.

keex · November 26, 2021, 6:23am

Yeah, I also wondered about this. However, I attributed the issue to the size of my growing network, as I suspect most users do as well. I doubt it’s planned obsolescence, but perhaps improvements to the coordinator just outgrow the hardware. If that’s the case, I fully understand and as a user, you have a choice as well – either use the most recent software versions and stick with a working setup with old hardware or upgrade your hardware to get new features. Not to antagonize you, but I think it might be the “natural” course of things. As natural as they come in the tech world

miroslavpetrov · November 26, 2021, 6:37am

Totally agree with you. I also think that cc2531 simply can’t keep up with the new features introduced by zigbee2mqtt and therefore it crashes. I have ordered a new stick from sonoff - https://itead.cc/product/sonoff-zigbee-3-0-usb-dongle-plus/ and I can’t wait to receive it in order to get my sensor network work fine again.