Connecting HA with multiple zigbee2mqtt instances

edit: It turned out that this is a problem caused by multiple zigbee2mqtt instances. I edited the title accordingly.
Old post:
Hey, I installed home assistant for the first time, and I’m a bit confused. I use zigbee2mqtt (on another machine), mosquitto and enabled the MQTT binding in HA. I enabled the HA support in the zigbee2mqtt config, and every device immediately showed up in HA with all data I wanted to see (which was quite impressive so far!).

But now I have two problems, and I’m not sure if I’m doing something wrong or if this is the intended way:

  1. If I restart, all mqtt entities are gone, while all non mqtt devices are still present. I have to restart zigbee2mqtt to see all entities again.
  2. While I can see all sensor data, I cannot interact with the light (HG06104A). If I toggle the switch on the dashboard, nothing happens, and it just will go back to off.

The z2mqtt device documentation site mentions a manual configuration; when I use this, everything works, and I can toggle the switch, and it also stays after a restart. Is this the way to go, or should all this work “out of the box” with the auto-discovery?

My confusion comes because the site states “Although Home Assistant integration through MQTT discovery is preferred, manual integration is possible with the following configuration”. So it sounds like it should “just work” with the auto-discovery. I’m fine with configuring it through the configuration.yaml, but right now, I have the feeling that something is broken in my setup. Could someone bring some insight, if everything works as intended or where I could start looking for errors?

About my setup:

Home Assistant:
Docker Latest (Home Assistant 2021.6.0)
Just started it with the basic configuration and added mqtt as well as an ikea gateway.

Zigbee2mqtt: 1.19.0, running on a different raspberry pi, added homeassistant: true and friendly names to the configuration
C2531 firmware: 20201127

Mosquttio: Docker latest (2.0.10)

Devices:
Some Lidl HG06104A, Xiaomi WSDCGQ11LM and Xiaomi MCCGQ11LM devices

That is not how it should work. I run Zigbee2MQTT on a different pi too (and mqtt on yet another pi), and have never had this problem. Does it happen when you restart HA, or when you restart the machine HA runs on ?

It happens even if I just restart HA through the web interface and accordingly if I restart the docker container. Do you also use the container? And do you remember if you configured something besides enabling HA in zigbee2mqtt and enabling the mqtt integration on HA?
Maybe it is something docker related. I have an unused pi where I will install the HA OS and see if it works there.

I don’t use containers for zigbee2mqtt or mqtt, they are installed on bare metal. It is possibly your mqtt container that does not have a persistence file. That needs to be configured to exist outside the container.

Ah, with restarting, I just meant HA. Zigbee2mqtt is running on a pi, and mqtt wasn’t restarted so far.

I just tested the same setup, but with HA running on the HA OS on a pi. And everything works, I can use the light, and everything is still available after a restart. So it looks like something was wrong with my docker container.
I just pulled a newer version (from yesterday) and reset everything. And now it works! I don’t know if some configurations in my container were broken or if there was a bug which was fixed in the release.
Anyways, thanks for your help!

It turns out I was wrong. Unfortunately, the error came back, but now I know roughly the source.
I’m running two zigbee2mqtt instances, one on a docker container and one on a pi. I disabled one for the whole process of testing this. Therefore I didn’t mention it in the beginning. Now that it worked, I enabled it again, and things instantly stopped working.

Both connect to the same mosquitto instance, which is used by HA.
I know that I wouldn’t need two zigbee2mqtt instances in theory since I could just use multiple routers. But right now, my Aqara sensors seem to ignore the routers and often fail to communicate with the coordinator. Therefore I split the devices and ended with this setup. Has someone a similar setup or an idea of how one could get multiple zigbee2mqtt instances work with HA? As soon as the second zigbee2mqtt instance joins, this error rises for all connected devices:

2021-06-05 22:46:44 ERROR (MainThread) [homeassistant.util.logging] Exception in availability_message_received when handling msg on 'zigbee2mqtt/bridge/state': 'online'
Traceback (most recent call last):                                                                                                                                                                                                         
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 31, in wrapper                                                                                                                                           
    _log_message(msg)                                                                                                                                                                                                                      
  File "/usr/src/homeassistant/homeassistant/components/mqtt/debug_info.py", line 21, in _log_message                                                                                                                                      
    messages = debug_info["entities"][entity_id]["subscriptions"][                                                                                                                                                                         
KeyError: 'switch.plug_1'  

edit: Found the answer on github