Z2M won't find any device

mombro · October 16, 2023, 7:56pm

Hi folks,

I’m trying to setup my Home Assistant with Zigbee and I’m at a loss for ideas why it won’t work.

History:
I’ve had HAOS running on my Pi3 incl. Zigbee2MQTT (Z2M) with ~8 devices. I then did a backup, installed HAOS on my Pi4 with SSD and did a recovery. Everything working great so far.

Now I’m trying to get Z2M back running as it was always just for fun, but I want to get more into it and remove my Chinese infrastructure Sadly, I cannot pair new devices, and the old one’s don’t work either. I removed Z2M and Mosquitto broker completely, restarted HAOS and started from scratch. Still won’t work after going through tutorials on the net.

I have a Sonoff stick, according to Amazon it’s a “D4F CC2531” (German page: https://www.amazon.de/gp/product/B09M475L3R).

I installed Mosquitto broker; according to tutorials, no configuration needed.

I then installed Zigbee2MQTT and looked up the port of the Zigbee stick. It seems to be “/dev/ttyACM0”:

So I added that in Zigbee2MQTT config:

I then started the Z2M addon and the log says that it started successfully, 0 devices joined, it’s connected and publishing to MQTT:

[21:32:37] INFO: Preparing to start...
[21:32:38] INFO: Socat not enabled
[21:32:40] INFO: Starting Zigbee2MQTT...
Zigbee2MQTT:info  2023-10-16 21:32:46: Logging to console and directory: '/config/zigbee2mqtt/log/2023-10-16.21-32-46' filename: log.txt
Zigbee2MQTT:info  2023-10-16 21:32:46: Starting Zigbee2MQTT version 1.33.1 (commit #unknown)
Zigbee2MQTT:info  2023-10-16 21:32:46: Starting zigbee-herdsman (0.19.0)
Zigbee2MQTT:info  2023-10-16 21:32:47: zigbee-herdsman started (resumed)
Zigbee2MQTT:info  2023-10-16 21:32:47: Coordinator firmware version: '{"meta":{"maintrel":3,"majorrel":2,"minorrel":6,"product":0,"revision":20211115,"transportrev":2},"type":"zStack12"}'
Zigbee2MQTT:info  2023-10-16 21:32:47: Currently 0 devices are joined:
Zigbee2MQTT:info  2023-10-16 21:32:47: Zigbee: disabling joining new devices.
Zigbee2MQTT:info  2023-10-16 21:32:47: Connecting to MQTT server at mqtt://core-mosquitto:1883
Zigbee2MQTT:info  2023-10-16 21:32:47: Connected to MQTT server
Zigbee2MQTT:info  2023-10-16 21:32:47: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online'
Zigbee2MQTT:info  2023-10-16 21:32:47: Started frontend on port 0.0.0.0:8099
Zigbee2MQTT:info  2023-10-16 21:32:47: MQTT publish: topic 'zigbee2mqtt/bridge/config', payload '{"commit":"unknown","coordinator":{"meta":{"maintrel":3,"majorrel":2,"minorrel":6,"product":0,"revision":20211115,"transportrev":2},"type":"zStack12"},"log_level":"info","network":{"channel":11,"extendedPanID":"0x00124b001cdcebda","panID":6754},"permit_join":false,"version":"1.33.1"}'
Zigbee2MQTT:info  2023-10-16 21:32:47: MQTT publish: topic 'zigbee2mqtt/bridge/state', payload 'online'
Zigbee2MQTT:info  2023-10-16 21:32:47: Zigbee2MQTT started!
Zigbee2MQTT:info  2023-10-16 21:34:00: Zigbee: allowing new devices to join.
Zigbee2MQTT:info  2023-10-16 21:34:00: MQTT publish: topic 'zigbee2mqtt/bridge/response/permit_join', payload '{"data":{"time":254,"value":true},"status":"ok","transaction":"yw9de-1"}'
Zigbee2MQTT:info  2023-10-16 21:50:30: Zigbee: allowing new devices to join.
Zigbee2MQTT:info  2023-10-16 21:50:30: MQTT publish: topic 'zigbee2mqtt/bridge/response/permit_join', payload '{"data":{"time":254,"value":true},"status":"ok","transaction":"ihjwi-1"}'

Mosquitto log looks ok to me, but not 100% sure:

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/mosquitto.sh
[21:30:07] INFO: Certificates found: SSL is available
cont-init: info: /etc/cont-init.d/mosquitto.sh exited 0
cont-init: info: running /etc/cont-init.d/nginx.sh
cont-init: info: /etc/cont-init.d/nginx.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun mosquitto (no readiness notification)
services-up: info: copying legacy longrun nginx (no readiness notification)
[21:30:08] INFO: Starting NGINX for authentication handling...
s6-rc: info: service legacy-services successfully started
[21:30:09] INFO: Starting mosquitto MQTT broker...
2023-10-16 21:30:09: Warning: Mosquitto should not be run as root/administrator.
2023-10-16 21:30:09: mosquitto version 2.0.17 starting
2023-10-16 21:30:09: Config loaded from /etc/mosquitto/mosquitto.conf.
2023-10-16 21:30:09: Loading plugin: /usr/share/mosquitto/go-auth.so
2023-10-16 21:30:09:  ├── Username/password checking enabled.
2023-10-16 21:30:09:  ├── TLS-PSK checking enabled.
2023-10-16 21:30:09:  └── Extended authentication not enabled.
2023-10-16 21:30:09: Opening ipv4 listen socket on port 1883.
2023-10-16 21:30:09: Opening ipv6 listen socket on port 1883.
2023-10-16 21:30:09: Opening websockets listen socket on port 1884.
2023-10-16 21:30:09: Opening ipv4 listen socket on port 8883.
2023-10-16 21:30:09: Opening ipv6 listen socket on port 8883.
2023-10-16 21:30:09: Opening websockets listen socket on port 8884.
2023-10-16 21:30:09: mosquitto version 2.0.17 running
2023-10-16 21:30:09: New connection from 127.0.0.1:35510 on port 1883.
2023-10-16 21:30:09: Client <unknown> disconnected due to protocol error.
[21:30:10] INFO: Successfully send discovery information to Home Assistant.
[21:30:11] INFO: Successfully send service information to the Supervisor.
2023-10-16 21:30:39: New connection from 172.30.33.5:37972 on port 1883.
2023-10-16 21:30:39: New client connected from 172.30.33.5:37972 as mqttjs_ffc901e9 (p2, c1, k60, u'addons').
2023-10-16 21:30:41: Client mqttjs_ffc901e9 disconnected.
2023-10-16 21:30:54: New connection from 172.30.33.5:58242 on port 1883.
2023-10-16 21:30:54: New client connected from 172.30.33.5:58242 as mqttjs_3da95901 (p2, c1, k60, u'addons').
2023-10-16 21:31:10: New connection from 172.30.32.1:44992 on port 1883.
2023-10-16 21:31:10: New client connected from 172.30.32.1:44992 as 0Fl57E4MxDscMyJfpWyr8u (p2, c1, k60, u'mqtt-user').
2023-10-16 21:31:45: New connection from 172.30.32.2:53180 on port 1883.
2023-10-16 21:31:45: Client <unknown> closed its connection.
2023-10-16 21:32:35: Client mqttjs_3da95901 disconnected.
2023-10-16 21:32:47: New connection from 172.30.33.5:41208 on port 1883.
2023-10-16 21:32:47: New client connected from 172.30.33.5:41208 as mqttjs_a5902a9a (p2, c1, k60, u'addons').
2023-10-16 21:33:45: New connection from 172.30.32.2:51502 on port 1883.
2023-10-16 21:33:45: Client <unknown> closed its connection.
2023-10-16 21:35:45: New connection from 172.30.32.2:57266 on port 1883.
2023-10-16 21:35:45: Client <unknown> closed its connection.
2023-10-16 21:37:45: New connection from 172.30.32.2:34612 on port 1883.
2023-10-16 21:37:45: Client <unknown> closed its connection.
2023-10-16 21:39:45: New connection from 172.30.32.2:43060 on port 1883.
2023-10-16 21:39:45: Client <unknown> closed its connection.
2023-10-16 21:41:45: New connection from 172.30.32.2:60632 on port 1883.
2023-10-16 21:41:45: Client <unknown> closed its connection.
2023-10-16 21:43:45: New connection from 172.30.32.2:47586 on port 1883.
2023-10-16 21:43:45: Client <unknown> closed its connection.
2023-10-16 21:45:45: New connection from 172.30.32.2:43660 on port 1883.
2023-10-16 21:45:45: Client <unknown> closed its connection.
2023-10-16 21:47:45: New connection from 172.30.32.2:37450 on port 1883.
2023-10-16 21:47:45: Client <unknown> closed its connection.
2023-10-16 21:49:45: New connection from 172.30.32.2:53128 on port 1883.
2023-10-16 21:49:45: Client <unknown> closed its connection.
2023-10-16 21:51:45: New connection from 172.30.32.2:50962 on port 1883.
2023-10-16 21:51:45: Client <unknown> closed its connection.

What’s odd to me, is that the new connection on port 1883 is closed immediately. I don’t know if that’s supposed to be like that.

Ok, so that’s the config. I then went to Z2M, started listening for devices, and tried paring the following:

PIR movement sensor (Sonoff)
water leak sensor (Aqara)
light switch (Ikea)
Button (Aqara)
Door/window sensor (open/close, Sonoff)

None of these were found, and I tried twice with all of them.

Does anyone have a clue what is going on here, and how I can troubleshoot this? Did I miss something? I remember in one video, they said that you should set a username/password for MQTT, but other pages said it’s unnecessary, so I left that out. I’m also wondering why the Z2M config does not require a MQTT configuration.

I’m looking forward to your responses! <3

<UPDATE>

I plugged in my old Pi with the mentioned stick and all the Zigbee devices appear to still be working with that one …

Here’s the Mosquitto log: (I did an nmap scan from 192.168.3.13, so you can ignore that)

s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
cont-init: info: running /etc/cont-init.d/mosquitto.sh
[21:25:04] INFO: Certificates found: SSL is available
cont-init: info: /etc/cont-init.d/mosquitto.sh exited 0
cont-init: info: running /etc/cont-init.d/nginx.sh
cont-init: info: /etc/cont-init.d/nginx.sh exited 0
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service legacy-services: starting
services-up: info: copying legacy longrun mosquitto (no readiness notification)
services-up: info: copying legacy longrun nginx (no readiness notification)
s6-rc: info: service legacy-services successfully started
[21:25:09] INFO: Starting NGINX for authentication handling...
[21:25:09] INFO: Starting mosquitto MQTT broker...
2023-10-17 21:25:09: Warning: Mosquitto should not be run as root/administrator.
2023-10-17 21:25:09: mosquitto version 2.0.17 starting
2023-10-17 21:25:09: Config loaded from /etc/mosquitto/mosquitto.conf.
2023-10-17 21:25:09: Loading plugin: /usr/share/mosquitto/go-auth.so
2023-10-17 21:25:09:  ├── Username/password checking enabled.
2023-10-17 21:25:09:  ├── TLS-PSK checking enabled.
2023-10-17 21:25:09:  └── Extended authentication not enabled.
2023-10-17 21:25:09: Opening ipv4 listen socket on port 1883.
2023-10-17 21:25:09: Opening ipv6 listen socket on port 1883.
2023-10-17 21:25:09: Opening websockets listen socket on port 1884.
2023-10-17 21:25:09: Opening ipv4 listen socket on port 8883.
2023-10-17 21:25:09: Opening ipv6 listen socket on port 8883.
2023-10-17 21:25:09: Opening websockets listen socket on port 8884.
2023-10-17 21:25:09: mosquitto version 2.0.17 running
2023-10-17 21:25:12: New connection from 127.0.0.1:46732 on port 1883.
2023-10-17 21:25:12: Client <unknown> disconnected due to protocol error.
[21:25:17] INFO: Successfully send discovery information to Home Assistant.
[21:25:19] INFO: Successfully send service information to the Supervisor.
2023-10-17 21:25:58: New connection from 192.168.3.13:9778 on port 8883.
2023-10-17 21:25:58: New connection from 192.168.3.13:9774 on port 1883.
error: received null username or password for unpwd check
2023-10-17 21:26:04: Client nmap disconnected, not authorised.
2023-10-17 21:26:04: OpenSSL Error[0]: error:1408F09C:SSL routines:ssl3_get_record:http request
2023-10-17 21:26:04: Client <unknown> disconnected: Protocol error.
2023-10-17 21:26:04: New connection from 192.168.3.13:9782 on port 8883.
2023-10-17 21:26:04: OpenSSL Error[0]: error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low
2023-10-17 21:26:04: Client <unknown> disconnected: Protocol error.
2023-10-17 21:26:04: New connection from 192.168.3.13:9785 on port 8883.
error: received null username or password for unpwd check
2023-10-17 21:26:10: Client nmap disconnected, not authorised.
2023-10-17 21:28:04: New connection from 172.30.32.1:35024 on port 1883.
2023-10-17 21:28:04: New client connected from 172.30.32.1:35024 as 5i99acm9YGZm3F9P09GG1N (p2, c1, k60, u'mqtt-user').
2023-10-17 21:28:19: New connection from 192.168.3.13:9916 on port 1883.
2023-10-17 21:28:19: New connection from 192.168.3.13:9917 on port 8883.
error: received null username or password for unpwd check
2023-10-17 21:28:20: Client nmapnzktqzdlqlsqvohe disconnected, not authorised.
error: received null username or password for unpwd check
2023-10-17 21:28:20: Client nmapuynuhrrdsigazzor disconnected, not authorised.
2023-10-17 21:29:54: New connection from 192.168.3.13:9933 on port 1883.
2023-10-17 21:29:55: New connection from 192.168.3.13:9938 on port 8883.
error: received null username or password for unpwd check
2023-10-17 21:30:00: Client nmap disconnected, not authorised.
2023-10-17 21:30:01: OpenSSL Error[0]: error:1408F09C:SSL routines:ssl3_get_record:http request
2023-10-17 21:30:01: Client <unknown> disconnected: Protocol error.
2023-10-17 21:30:01: Client connection from 192.168.3.13 failed: error:1420918C:SSL routines:tls_early_post_process_client_hello:version too low.
2023-10-17 21:30:01: New connection from 192.168.3.13:9958 on port 8883.
2023-10-17 21:30:04: New connection from 172.30.33.5:53904 on port 1883.
2023-10-17 21:30:04: New client connected from 172.30.33.5:53904 as mqttjs_654efa35 (p2, c1, k60, u'mqtt-user').
error: received null username or password for unpwd check
2023-10-17 21:30:07: Client nmap disconnected, not authorised.
2023-10-17 21:32:16: New connection from 192.168.3.13:10117 on port 8883.
2023-10-17 21:32:16: New connection from 192.168.3.13:10118 on port 1883.
error: received null username or password for unpwd check
2023-10-17 21:32:16: Client nmapkefllrpxlrtcndyd disconnected, not authorised.
error: received null username or password for unpwd check
2023-10-17 21:32:16: Client nmapdvzzacofykwsmegq disconnected, not authorised.

It seems like there are no disconnects like before, so I’m wondering if that might be the issue here.

Any idea how I can purge all Mosquitto and Z2M config?

francisp · October 18, 2023, 4:12am

The SSD is causing EM interference. Use an USB extension cable to keep your stick away from the SSD (and be sure to use an USB2 port)

mombro · October 24, 2023, 6:19pm

SSDs do that?!

I bought a new Sky Connect :-/

Thanks anyway

francisp · October 25, 2023, 3:39am

hqdefault

dproffer · October 25, 2023, 4:39am

It appears you have made a lots of changes. That’s a recipe for frustration.

My first guess is that you have some problem with your MQTT system running under Home Assistant. There may be more issues as well, back to my point about ‘you seem to have made a number of changes’.

All the input ‘noise’ your are getting about USB 3 noise and other stuff may be something to look at. HOWEVER, as you show in your log files, you have got some basic software issues first before going down the rabbit hole of USB noise and zigbee channel selection and all the other shoot from the hip issues that might be out there… IMHO, until you get rid of these errors, don’t waste your time on anything else.

2023-10-17 21:26:04: Client nmap disconnected, not authorised.
2023-10-17 21:26:04: OpenSSL Error[0]: error:1408F09C:SSL routines:ssl3_get_record:http request
2023-10-17 21:26:04: Client <unknown> disconnected: Protocol error.

I’m pretty opinionated. I do not think it is a good idea to run your MQTT broker and Zigbee2MQTT systems with in Home Assistant HAOS. Too complicated and too inter related.

I recommend installing docker on your RPi and keeping your HA, MQTT server, Zigbee2MQTT isolated and separated using Docker. I ran such a configuration on at RPi3 for 1 plus years.

Yes, taking the time to learn about Docker, and it is a bit of a PIA. However, having the skills to run whole systems like MQTT, Zigbee2MQTT and in my case Home Assistant in small isolated docker containers made my home automation system much more stable and manageable. Yes, in theory, this is the pretty much the same thing that is happening with HAOS, however, you have little control over the upgrade and interaction of significant systems such as MQTT and Zigbee2MQTT when done within HAOS.

With HA, MQTT and Zigbee2MQTT separated you can start, stop, upgrade each with minimum effect on the other components.

Do a bit of googling, youtubing about docker.
install docker on your RPi. Do the Hello World example to make sure it is working. Add the web docker manager Portainer, this will make your management of Docker container much easier.
Uninstall MQTT from within HA.
Spin up a docker version of MQTT server.
test that you can access this MQTT server using a tool like MQTT Explorer on your workstation.
configure HA to used this MQTT instance.
Install Zigbee2MQTT in a docker container. yes, figuring out the f’ing USB setup is a PIA. But take your time and get Zigbee2MQTT running in docker. Check it logs regularly.
See your Zigbee2MQTT publishing devices to your docker MQTT server, by viewing with MQTT Explorer.
Now your Home Assistant instance should see the Zigbee2MQTT devices via your docker MQTT.

You can now stop, start, upgrade, downgrade each of your system components, HA, MQTT, Zigbee2MQTT independently. Hope you find a good path, I just see to many folks getting blocked and frustrated by the ‘all in one’ of HAOS. Yes, it seems a good idea, but there is too many moving part from too many independent open source software systems. Home Assistant, needs to just do its core functions, do them well and solid. Minimize the ‘external’ devices and such running within HA. MQTT running outside HA is a solid way for other system running in docker like zigbee, bluetooth, zwave and so many more to smoothly interact with HA.

Good hunting!

francisp · October 25, 2023, 4:59am

Good idea, because the CC2531 is seriously underpowered.

dproffer · October 25, 2023, 5:08am

I ran Zigbee2MQTT on RPi with CC2531 as coordinator and 20+ devices for over a year with no issues. Saying it is ‘seriously underpowered’ IMHO is not true.

francisp · October 25, 2023, 5:11am

I used a CC2531 in 2019 too, because that was the only adapter supported by Zigbee2MQTT. And you don’t notice it, until you changed to the new generation when they became available. A whole world of difference.

dproffer · October 25, 2023, 5:31am

I upgraded the CC2531 to a CC2652P2 and saw no difference in function of the Zigbee network. Interested in what you found to make ‘a whole world of difference.’ I am not saying there are not benefits to the newer hardware and Zigbee 3 coordinators functions. Though from what I have read, I think a number of folks have had a very painful path using the ‘Sky Connect’ device as a coordinator, hopefully that is solved, however back to my point about ‘change’, sometimes, especially with home automation devices that are working, better to NOT change sometimes. However, from my experience with a network of the size of 20 to 30 devices and the mostly older not the newer vendor devices from the east. And I’m not sure even with these newer devices I saw much, I still have CC253x router devices running Zigbee 2 routing for newer Zigbee 3 devices just fine. Yes, I have run across newer devices that have introduced ‘issues’, however I am not sure the issues would have been solved by newer coordinator hardware and firmware.

Again, my fundamental points : minimize simultaneous changes, upgrades to device firmware is sometimes not a good thing, limit Home Assistant to its core functions and isolate systems using docker or similar technology.

mombro · October 25, 2023, 8:13pm

Wow, my friend, you took a lot of time for that response! Thank you very much!

Yes, I made several changes at the same time, and it is horribly frustrating. I’m getting frustrated quickly anyway That’s why I stuck to Pi3 for so long because I didn’t want to change anything, but the time had come.

I’m working professionally in IT and I’ve tried in several circumstances during my WORK TIME to find out how docker works and failed over and over again. I think it is a really, really poor software and badly documented, too. I know all the upsides, I’ve seen people orchestrate stuff, I know that’s awesome, but it was made for veeeeery special people Unless someone pays me a three day workshop, I’ll never in my life start a docker container…

I’m just glad the sky connect and ZHA work better now. I think it was the proper decision for the moment

dproffer · October 25, 2023, 8:22pm

Glad you got a solution and are back in operation.

yes, Docker is odd and takes a bit of time to get your brain around. However, for my home automation stuff, I am very glad I took the time. I’ve yet to find a better tool to create small isolated ‘apps’ with relatively light weight overhead. I started using it on a RPi 3 running Raspberry linux.

Good hunting!