Before I do that Iāll move lighting back to Hue.
I need the lighting to be reliable or risk the wrath of my better half.
It will reduce the loading on Home Assistant zigbee2MQTT and get rid of the pesky disagreeable lights
Regarding the issue with devices showing as unavailable. I had this same issue with several of my zigbee plugs after I updated to z2m version 1.36.1. In my case though it was about every two minutes these devices would show as unavailable. I ended up reverting back to version 1.36.0 and the issue went away. Per the z2m github page it does appear others are having similar issues. If you can try rolling back to a previous version to see if the problem goes away.
I wouldnāt say there is a recommendation but rather personal preference maybe. I use Proxmox with HA running in a VM with z2m and other addons running in LXCās. I chose this method because I didnāt want everything running in HA in case HA ever had to be taken offline or crashed. If you want to go this route and use Proxmox there are some helper scripts that make it easy to do
Proxmox VE Helper-Scripts | Scripts for Streamlining Your Homelab with Proxmox VE (tteck.github.io)
1 Like
Before Home Assistant I was running MQTT, zigbee2MQTT and Nod Red in containers, keeping the updates up to date and managing them required a higher level of maintenance than was optimal. Especially when my days of command line passed 30-40 years ago 
I am open to all options.
For the moment, everything has been stable since Sunday, normally this hits every 7 - 10 days, I am journaling the issue now so can keep a precise record.
I have disabled 25 integrations, which while a bit of pain is not hitting critical systems but a some point would like the functionality back but not important when lighting, switches, plugs, sockets etc stop working.
This weekend I will migrate Hue lighting back off Home Assistant zigbee2MQTT back to a newly wiped Hue Hub.
Then hand over most of the autonomous lighting (90% of the house is driven by motion/presence sensors) the remaining will be through the official Hue integration.
That will reduce the zigged devices down from 148 to 100 leaving more room to add the remaining Router type devices. That will reduce the loading on zigbee2MQTT and Home Assistant.
Then stand back and see if the problem is still occurring, the stay take a couple of weeks.
But at least the lighting will remain functional if it continuing.
Actually the issue is helping out a little - I have been bouncing back and forth between the monolith solution - ie everything sits in Home Assistant or I use Home Assistant as a support for Node Red operating as Command and Control with autonomous subsystems, ie Tado runs heating and hot water, Hue lighting, Eufy Cameras, with Home Assistant running the Govee/Nanoleaf/Shelly lighting.
I will need to look at automous sub units to support Alarm system.
As part of this if I am still experiencing issues with zigbee2MQTT /Hass integration I will do a rebuild of zigbee2MQTT and MQTT as recommended by @aceindy
If still having issues will look to running Matt and zigbee2MQTT in another VM and follow @billyjoebob999 helpful suggestion.
Also need to raise a ticket on GitHub for the zigbee2MQTT losing every device - but need to collect more data first to see if I can make the issue repeatable on demand.
Also, need to raise a ticket with Home Assistant team onHass losing every zigged device following a restart.
Lots of steps here - thanks to all those who have helped.
ditch the hub, why do you need it?
I have seen posts that state the hue/ikea lights bulbs use a protocol that disrupts other zigbee devices on the same controller.
Also, if zigbee goes down my better half is not amused when lights donāt work
I had hoped I could run it all in Home Assistant but I have also been told that I have too many devices on zigbee2MQTT so this a surgical prune of 48 devices and makes them autonomous to boot with an override still available in Home Assistant
?? 48 ??
i think the amount of repeaters is more important 
For sure the controller is the bottleneck, not Z2M
I am taking a new (to me) and making it smart as I get closer to retirement.
So I know I am stressing the boundaries
48 of the 148 devices on zigbee2MQTT are Hue light bulbs/ motion sensors.
Currently 85/62 Router/Endpoint
By moving over the 48 Hue devices it will be
52/47 until I start adding mainly Router devices
If I remove the Hue bulbs I can keep it all under 150 zigbee2MQTT devices on coordinator/z2m
I have brought zwave up online for heat/fire/smoke/door locks sockets along with some motion/door/window sensors
Iāll eventually end up with 30+ double wall sockets, the four wet areas have 10 leak sensors.
30+ Door window sensors
20+ plug in sockets
15 outdoor sensors, shed doors, gates etc, motion
7 presence sensors
5 air quality sensors
10 switches
8 ikea relays
5 temp/humidity sensors - I get most of the temp and humidity from tado.
4 bed occupancy sensors
15 Blind controllers
If I run the lights on an semi autonomous service the same as I do for heating/hot water then I can build in a higher level of resilience
Most coordinators have a limit of 50 routing devices max. Some can go beyond that limit, but most of the ācommonā ones can barely handle that (usually crapping out between 25-30 routers attached).
If you need more than 50 router devices, youāll need to think about splitting your mesh as the ZStack firmware has a pretty hard limit of 50 routing devices, 100 normal routes and 200 source routes. Youāre going to have a hard time getting around that. Moving the Hue bulbs back to their hub is essentially doing that very same thing.
Useful info thanks.
Moving lights onto Hue hub(s) seems to be a useful way forward.
Flirting with the idea of running up a zha as well, break the house up, downstairs and upstairs + outside.
1 Like
Or you can do a second instance of Z2M. Thatās actually the route I would go as using both ZHA and Z2M at the same time can lead to things like a device having different quirks between the two platforms and such. Plus, imho, Z2M adds devices and quirks (drivers) a lot faster than ZHA does. But, thatās just my opinion.
2 Likes
^^This. Itās super easy to spin up another instance of Z2MQTT Edge. Just remember to change the topic name.
2 Likes
As an update
zigbee2MQTT ālostā all of its devices again at 22:17 Saturday night, 5 days after it did it before.
So have enabled most of the 25 integrations I disabled as they do not appear to have any impact on this.
Not a problem - keeping a journal and log grabs so I can try to identify the issue.
I am migrating the Hue bulbs and motion sensors off zigbee2MQTT back to a Hue hub, 1 to stabilise core services, 2 reduce loading in zigbee2MQTT, 3 reduce the Hue/ikea bulb disruptive effect on zigbee.
It will get solved, I just need to do it slowly so I can track the issue down.
Some of these are known to generate excessive traffic.
I have turned on zigbee2MQTT debug logging to see if I can capture the issue occurring.
1 Like
At the risk of tempting fateā¦
Reduced loading on zigbee2MQTT to 52/49 by moving the Hue lights and motion sensors onto Hue Hubs
So far I have not had another repeat of zigbee2MQTT ālosingā all its devices, which used to occur every 5-7 days.
Currently at day 13 without incident, so fingers crossed
The lights are all doing great.
And my better half is not frustrated by lighting that fails.
So all is good in the world.
Will return to the basic issue of zigbee2MQTT failing to load every time Home Assistant restarts.
The automation is a good workaround for now but would like to get to the bottom of it one day.
Well that did it.
Overnight zigbee2MQTT lost all devices.
After restarting zigbee2MQTT all but two devices reconnected.
Good news it went two weeks this time before dumping, and the lights continued working now they are on Hue.
I may look at bringing a Hubitat hub back up and have that run zigbee devices and share over Mqtt back to Home Assistant to get some stability.
Removing a third of your devices caused your system to be stable for twice the time.
To me, this is starting to sound like something (HA/MQTT/Z2MQTT) is running out of memory or space.
Long shot, but did you leave debug logging enabled? If you did, that would explain why a restart fixes things temporarily - the old logs get flushed (written to a separate backup file) and a new log file is created. The debug messages could cause your logs to balloon to the extent that it would cause a crash.
This is all speculation, but it wouldnāt hurt to have a look.
Whats the brand of those presence sensors?
Also, did you upgrade to last zigbee2mqtt? They changed the log process.
The presence sensors are mostly Sonoff, now I have moved lighting back to Hue I only have one presence sensor working now.
Running zigbee2MQTT 1.37.0-1
Ok finally resolved this issue by completely removing zigbee2MQTT while I was switching to a POE / Ethernet connected zigbee SMLight coordinator.
It was a pain having to re-pair 80 odd devices, bot as bad as it could have been if I had not already moved all the 80+ lights and motion sensors onto Hue Hubs.
All the issues I was having with zigbee2MQTT,
zigbee2MQTT failed to re-start after a Home Assistant reboot
Every 5-10 days zigbee2MQTT would initially make all devices unresponsive followed by declaring them offline.
2 Likes