[SOLVED] MQTT with Multiple Zigbee2MQTT instances

For several frustrating weeks I have been trying to create a reliable Zigbee network.

I have around 200 Zigbee devices (most Philips Hue bulbs) on a single floor. 95% are main powered and 5 IKEA repeaters spread around the house. Nevertheless, commands are often not arriving.
Currently, all devices are connected to SkyConnect (on a well-shielded USB cable far away from all sources of interference, USB 2.0, in the center of the house, channel away from WiFi 2.4 channels). I have tried ZHA and Zigbee2MQTT - currently Z2M.

Other users with many devices reported that they have solved these problems by splitting the network between 2 or more coordinators. So this is my last attempt before I have to go back to my 4 Hue Bridges.

But I’m having trouble getting the different Zigbee2MQTT instances to work.

My plan is to have 4 LAN coordinators (CC2652) distributed in the house with Zigbee2MQTT and one LAN coordinator (EFR32) with ZHA for the outdoor devices.

In the first step, I installed the add-on a second time (with / at the end of the URL). So in the end I could install two Z2M Standard Add-ons and two Z2M Edge Add-ons.

Then I initially configured first the EFR32 coordinator:

data_path: /config/zigbee2mqtt_5
socat:
  enabled: false
  master: pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777
  slave: tcp-listen:8490,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5
  options: "-d -d"
  log: false
mqtt: {}
serial:
  port: tcp://192.168.2.195:6638
  adapter: ezsp

It starts up without any problem. I then changed the Zigbee channel to 15 in the Z2M UI. 22 devices registered without any problems. All devices from both add-ons appear in the MQTT integration and can be controlled in HA - so far everything was fine.

Now the further expansion should take place and I have set up the first CC2652 LAN Coordinator in the first Z2M Edge add-on:

data_path: /config/zigbee2mqtt_3
socat:
  enabled: false
  master: pty,raw,echo=0,link=/tmp/ttyZ2M,mode=777
  slave: tcp-listen:8493,keepalive,nodelay,reuseaddr,keepidle=1,keepintvl=1,keepcnt=5
  options: "-d -d"
  log: false
mqtt:
  base_topic: zigbee2mqtt_3
  server: mqtt://localhost:1883
serial:
  port: tcp://192.168.2.193:6638

It has not started in the Info tab, and the log does not show more than the first 3 lines “Starting …”. The UI is also not available.

The main problem is that all 200 devices of the first add-on, to which all devices in the house are currently connected, have disappeared from the MQTT integration. They are still visible in the Z2M UI and can still be operated via it. I think I remember the Z2M configuration.yaml no longer has any devices.

Since we can no longer sit here in the dark, I had to import a full HA backup (takes 2 hours).

Unfortunately, I don’t understand Z2M and MQTT broker deeply enough. I suspect that this is where the various add-ons interfere. But I have changed the data_path.

Any help is very welcome. I don’t want to give up the self host approach and use the Hue Bridges again.

You have to make sure the different Z2M instances have different base tropic. Done in the MQTT tap under settings.

Standard are “zigbee2mqtt” and I use “zigbee2mqtt1” for my second Z2M instance.

From my experiences, will not call myself an expert, just with a number of ‘scars’…:sunglasses:

A couple things jump out based on your journal of your experiences:

  1. Start with a hard reality statement for you. If as you say, ‘I don’t understand Z2M and MQTT broker deeply enough.’ is where you are at with a setup of this size and you are not willing or able to put in the sweat equity to learn this stuff, then just stick with the Hue hubs. You have invested some serious coin and time in your lighting setup. All the BS around Signify wanting to ‘steal’ all your personal life via their Hue Hub is just that… BS. And I’m pretty sure the conversations with your ‘significant other’ while in the ‘dark’ is not where you want to be in 2024 :wink:. There are millions of solid and performant zigbee lighting setups around the world based on Hue Hubs (and many to your scale), no reason you should not be one of them.

  2. Using the SkyConnect device in general and specifically with Zigbee2MQTT, IMHO is something that based on my reading (not direct experience) may well have caused you some issues. Maybe not the main ones, but enough to distract you. Stop with the SkyConnect!

  3. To your plan to mix both TI CC2652 and EFR32 coordinators, this does not seem to be a good plan. Why add the variable of having to understand two different devices for, arguably, the most critical part of each Zigbee network? The coordinator. Stick with one, and IMHO, especially for Zigbee2MQTT, the TI 26xx chip for your coordinators.

  4. Either via Docker and or on separate (for example Raspberry Pi’s) hosts to run your MQTT broker and each of your Zigbee2MQTT instances OUTSIDE and separate from your Home Assistant server. Why on earth should your lighting system go off line just because either you are upgrading your Home Assistant server or rebooting it to battle a bug within Home Assistant software? Hue Hub rarely does firmware upgrade and it just works for YEARS between reboots. Zigbee2MQTT does do a number of upgrades, however 90 plus % of the time these upgrades are to support new devices, rarely to fix bugs (not as rare as Hue Hub, but far less than Home Assistant related issues). And, once you get a solid Zigbee2MQTT setup running, my advice is DON’T do any upgrade to the zigbee path of your home automation setup unless absolutely necessary.

Just my opins to keep you out of the :dog::house_with_garden: with your significant other and success with your home automation adventure!

Thanks. I will do this.

Do you know what base_topic is for? I have tried to learn about it but I could not find something helpfull.

And do I have to restart the Z2M and/or the MQTT add-on after submit in the UI to make this effective?

Yesterday evening I set up an HA test environment to continue testing today.
I was currently able to set up all four LAN coordinators in this previously empty environment without any problems and all Z2M add-ons started properly straight away.

Now I will unlearn a few bulbs from the production system and learn them in the different Z2M instances. Let’s see how it works.

I will report back.

Thanks David for taking time to read and write.

That also my experience and the reason why I move to the CC2652.

To your plan to mix both TI CC2652 and EFR32 coordinators, this does not seem to be a good plan. Why add the variable of having to understand two different devices for, arguably, the most critical part of each Zigbee network? The coordinator. Stick with one, and IMHO, especially for Zigbee2MQTT, the TI 26xx chip for your coordinators.

As I wrote above. My plan ist to have four CC2652 with the maximum possible number of Z2M add-ons to be installed - for the indoor devices.

In one of the intermediate stages of my learning curve, I ordered an EFR32. I now have it. And I have to use ZHA for a fifth coordinator anyway. And I think you can definitely use an EFR32 for that.

Either via Docker and or on separate (for example Raspberry Pi’s) hosts to run your MQTT broker and each of your Zigbee2MQTT instances OUTSIDE and separate from your Home Assistant server …

This is a new insight for me, but a good hint. I will look into it. Especially because I have everything running in Proxmox on a NUC i5 anyway. That would mean 6 new containers (5x Zigbee2MQTT for each coordinator and 1x MQTT broker). I just found this at a quick search:

Do you have any other good resources for this?

By the way: We built the house with a smart home first approach in 2020. Back then, I was still on ioBroker and connected everything with the respective proprietary bridges and cloud services. I’ve only been moving to HA for a few weeks and have only been thinking intensively about self-hosted and cloud-free strategies since then.

1 Like

This is how I manage ~250 zigbee devices. I have 3 RPi3’s that run instances of zigbee2mqtt that connect back to my main server (I haven’t yet moved my MQTT server to its own platform yet). One of them also hosts my Zwave network.

I have a large amount of Hue bulbs, Samsung smartplugs plus other sensors. I found some devices don’t play well together, e.g. Older Aqara sensors and Samsung smartplugs. Multiple zigbee networks for a large amount of devices helped me manage latency and compatibility issues.

I also agree with sticking to 2652 coordinators for zigbee2mqtt. My tests with sonoff dongle-e were painful.

~Brian

1 Like

Thanks Brian.

Yes, I think I will go the way with running the Zigbee2MQTT and maybe MQTT separately. Especially in view of the fact that it is not known how long it will be possible to operate several identical Zigbee instances in HA in parallel. This is currently only a hack and not officially supported. And light is a too important service.

I will try it with the LAN coordinators instead of Pi’s, because I have bought them in the USA and can’t return.

So I’m off on the way. I’ve just realized that I first have to upgrade Proxmox from 7 to 8, as the MQTT container requires a more up-to-date Debian.

Well, the weekend is too short again…

As i understand, base_tropic needs to be unique for each “controller” of MQTT inputs.
Not sure if you need to restart Z2M.

While I find zigbee2mqtt very robust, keeping your MQTT broker as a process under your Home Assistant server may cause some issues with your Zigbee2MQTT instances if you shutdown Home Assistant (which will bring down the MQTT broker).

Again, I emphasize, keep the components of your lighting system as isolated as possible. This includes your MQTT broker.

Without Home Assistant running, depending on how you have your Zigbee devices configured, you will lose some percentage of your lighting functionality while Home Assistant is down, however your Zigbee mesh networks will stay up and not have to rebuild the way they do when you cycle your Zigbee2MQTT software and/or power cycle your coordinator device. And IMHO, this these two operations often cause the majority of ‘issues’ that I read that people having. It’s not that they are ‘bad’ operations, it is just that people do not understand what is happening under the covers while this is occuring (this is partially caused by zigbee not doing a good job of surfacing what is happening, however mostly a PEBCAK issue where folks get impatient and then start ‘fixing things’.

The write up you cite has some useful info. However, as I state emphatically below, you need to be able to do some basic Proxmox VM setup, and Docker setup under your Linux VM at Linux command line by yourself. Just running someone’s script is not a good path IMHO. Yes, you will have some calluses on your fingers and blurry eyes from all the Googling and reading you will be doing.

IMHO, while complex, using Proxmox is a good base for a home automation setup. I use it now and am very happy.

More memory is probably the first path with proxmox, however with some thought applied you should be able to run HA, MQTT broker and 5 Zigbee2MQTT instances on a 8GB i5 proxmox setup. I won’t tell you how much memory I have in my 13th gen Intel i5 proxmox setup, it’s pretty embarrassing… okay I’ll fes up 128 gigabytes :face_with_open_eyes_and_hand_over_mouth::innocent: .

My MQTT broker is using 5 megabytes of memory and my Zigbee2MQTT instances use about 80 megabytes each. YES, that is MEGABYTES NOT GIGABYTES!!! Chump change. Learn the “docker stats” command. See picture below.

In another post, I see you stating you are upgrading your proxmox from 7 to 8 just to get your MQTT broker running. Dude!! you gotta get out of this type of mindset!!!
Proxmox is the foundation of your setup, MINIMIZE screwing with it. Version 7 had been out and running for years, version 8 and 8.1 are babes in the woods. Stop Stop Stop this. MQTT brokers have been running under proxmox version 7 and lower for years. Just because it is shiny and new, most definitely does not mean you need to use it. Back to my original point 1 hard love statement, if you are not able to put in the sweat equity to figure out how to make something like a MQTT broker run in your current environment by typing linux command line commands, then stop and go back to your Hue Hubs. Just taking someone script from the internet and running it is a path to constant disaster in your home automation setup. You have to learn a little about docker and some basic linux stuff, if you go down this route.

I recommend using Docker containers under a proxmox VM for all three types of your ‘servers’ :
Home Assistant
MQTT Broker
Zigbee2MQTT

I find VM’s easier to work with and understand under proxmox vs. LXC’s. And IMHO the performance difference is small compared with the more documented world of VM’s.

I recommend creating two virtual machines on your proxmox setup, on for production docker and one for testing docker. Use a long term support version of Ubuntu or Debian. People are going to be running Ubuntu 20 and Debian 11 server for the next 20 years, trust me. You do not need the latest shiny toy such as Debian 12 and Proxmox 8.

Below are the simple commands to spin up a mqtt broker and a zigbee2mqtt coordinator in docker containers under Ubuntu 20 VM on proxmox 7. IMHO, you should be able to spin up these specific versions of software on your Proxmox server and put it behind a locked door and not have to touch it for years.

Good hunting!

# mosquitto mqtt

mkdir /home/user/mosquitto-2
nano /home/user/mosquitto-2/mosquitto.conf
#----- mosquitto.conf
#
# https://mosquitto.org/man/mosquitto-conf-5.html
listener 1883
allow_anonymous true
log_timestamp_format %Y-%m-%dT%H:%M:%S
connection_messages true
log_type subscribe
log_type unsubscribe
log_type websockets
log_type error
log_type warning
log_type notice
log_type information
persistence true
persistence_location /mosquitto/config/
log_dest file /mosquitto/config/m.log
# -----

docker run -i -t -d --name="mosquitto-2" \
    -p 1883:1883 -p 9001:9001 \
    -v /home/user/mosquitto-2:/mosquitto/config \
    -v /home/user/mosquitto-2:/mosquitto/data \
    -v /home/user/mosquitto-2:/mosquitto/log \
    -v /etc/localtime:/etc/localtime:ro \
    --user $(id -u):$(id -g) \
    eclipse-mosquitto:2.0.18

# zigbee2mqtt

# make sure you understand where your zigbee coordinator dongle is exposed in the VM
# put it's path in command below and in :
# /user/home/zigbee2mqtt/zigbee2mqtt-data/configuration.yaml :
nano /user/home/zigbee2mqtt/zigbee2mqtt-data/configuration.yaml
# ----- configuration.yaml
permit_join: false
mqtt:
  base_topic: zigbee2mqtt
  server: mqtt://192.168.xxx.yyy
serial:
  port: /dev/serial/by-id/usb-1a86_TubesZB_971207DO-if00-port0
# -----

# PRODUCTION check DIRECTORY map!!!! PRODUCTION:  /home/user/zigbee2mqtt
docker run -i -t -d \
   --name="production-zigbee2mqtt-1.34.0" \
   -v /home/user/zigbee2mqtt/zigbee2mqtt-data:/app/data \
   -v /run/udev:/run/udev:ro \
   --device=/dev/serial/by-id/usb-1a86_TubesZB_971207DO-if00-port0 \
   --user 1000:1000 \
   --group-add dialout \
   --net=host \
   -e TZ=America/Los_Angeles \
   koenkk/zigbee2mqtt:1.34.0

Thanks David! I do as recommended. This is my architecture now:

After one week all was up and running.

And the best: Now with five separate Zigbee-Networks (each with a different channel 11,12,13,14 and 15 and each with around 30-35 devices) everything works rock solid. No lost commands since one week now.

The last thing I have to do is moving Home Assistant from a supervised OS version on a VM to the already running Docker Container. For that I have to make some research first.

And then a backup strategy (VM - Container - Home Assistant)

For those who find this thread: The YT videos und articles from Home Automation Guy

around supervised OS versus self hosted Docker Containers gave me a good overview and tips.

Conclusion:

After my observations and a painful learning curve - for large Zigbee Netzworks the only way is to split up in multiple networks and place LAN coordinators in the corresponding areas of the house. And while this is not possible with HA OS (only one instance/add-on is allowed) you better start with Home Assistant and add-ons in Docker containers.

2 Likes

That is a good looking setup, impressive!

Glad you are getting a stable environment, and the ‘significant other’ can just use it :wink:

Hoping you have much success and fun with your home automation adventures in 2024 and beyond!

1 Like

For those who find this post - here is a short update:

Since I split the Zigbee network into 5 Zigbee LAN coordinators, everything continues to run super stable and completely smoothly.

I have recently changed one thing in my architecture compared to the diagram above.

I switched back from the container version of Home Assistant to Home Assistant OS. Mainly because of the easier update process.

Oh yes, and local video security with Frigate has now also been added:

2 Likes

Hi Frank,

I have a large property and I already 4 Hue Bridges to physically cover the separate buildings. While lights could be run by z2M, I need HueBridges for FriendsofHue Switches which I use a lot. Now I want to use radiator thermostates (TRV) and they don’t pair with Hue, so I need to implement your setup.

I already run HA as VM on Proxmox. Can you please share why your MQTT, z2m and ESPHome Instances are run in docker under VM and not in separate LXC containers?

And, more importantly, can you share how you configured your Home Assistant to use these instances? I am struggling… Do you use the z2m Proxy to access the z2m instances from within the HA user interface or do you open them directly on your browser?

I know it’s a lot of questions but I will be happy to document the process while setting it up so others with similar issues can follow!

Hi

First of all: I changed my setup a bit more in the direction of “managed by HA OS”. Instead of the containers for the MQTT broker and ESPHome, I now also have these running as add-ons in HA. I know, add-ons are nothing more than Docker containers, but they are managed by HA and you can easily configure and update them via the UI.

Why I set up a Docker environment instead of LXC: Because I needed a roadmap and instructions and Homeautomation Guy has done this wonderfully in his videos and blog posts, so I could follow it well. Including the configuration method Docker-Compose, update scripts and backup with Duplicati. I also have the feeling (but this is really just a feeling) that there are many more people here in the community who are familiar with Docker than with Proxmox LXC. So if there is something well documented for Z2M for LXC, why not.

So now I have the Mosquitto MQTT Broker runnung as add-on in HA. It has the standard setup. You can configure it in the HA UI:

logins: []
require_certificate: false
certfile: fullchain.pem
keyfile: privkey.pem
customize:
  active: false
  folder: mosquitto

It uses the credentials which are set up in the HA users section by default. I only changed the PW.

Then in the configuration.yaml of each Zigbee2MQTT instance I configured the MQTT section regarding to this.

Instance 1:

homeassistant: true
frontend: true
mqtt:
  base_topic: zigbee2mqtt_1
  server: mqtt://192.168.2.30
  user: mqtt
  password: xxx
  reject_unauthorized: true
serial:
  port: tcp://192.168.2.191:6638
advanced:
  network_key:
...

Instance 2

homeassistant: true
frontend: true
mqtt:
  base_topic: zigbee2mqtt_2
  server: mqtt://192.168.2.30
  user: mqtt
  password: xxx
  reject_unauthorized: true
serial:
  port: tcp://192.168.2.192:6638
advanced:
  network_key:
...

… and so on.

Make sure, that the base_topic is different in each instance

The mqtt server address is the IP of your HA machine. Make sure you enter the prefix mqtt://

user and password are the ones which are configured in HA in the Settings → Persons → User tab.

The serial port is the IP address and port of each respective Zigbee LAN coordinator.

Editing the configuration.yaml files of the Z2M instances I manage vie SSH connection from a local Visual Studio Code App. There is also a guide from Homeautomation guy available.

The UI of each Z2M instance I first added to the HA side bar. This is very easy since the HA update 2024.3 or 4 i guess. You only have to enter a new Website with the IP and port of the respective Z2M instance:

Since my HA sidebar is already so full and I only really need the Z2M UI to pair new devices, I have now moved it to my Arc browser as a tab.

Hope this helps.

1 Like

Thanks a lot! I will try to replicate on my system, running MQTT and two Z2M instances each as individual LXC on a Proxmos (which allows me to snapshot/backup/restore individually).

OK, please note that my comments above about the MQTT configuration assume that the MQTT broker runs as an HA add-on and not as a separate container. Why do you want to install MQTT as an LXC container and not as an add-on?