There can actually be problems with using that many devices if either have a older Zigbee Coordinator (that is using old hardware) or if you are using Multi-PAN and Multi-protocol on a newer Zigbee Coordinator (meaning that using the same radio adapter for both Zigbee and Thread at the same time).
@Bob71 Suggest try flashing the latest Zigbee NCP (EmberZNet) firmware beta instead of RCP firmware if have not already → silabs-firmware/EmberZNet/beta at main · NabuCasa/silabs-firmware · GitHub
Start by flashing a EmberZNet Zigbee NCP firmware (not a RCP firmware) on the SkyConnect and then make sure that you also follow all these best practice setup tips → Zigbee networks: how to guide for avoiding interference + optimizing using Zigbee Router devices (repeaters/extenders) to get best possible range and coverage
That includes connecting the Zigbee Coordinator to a USB 2.0 port or via a powered USB 2.0 hub via a long USB extension cable to get it away from all potential sources of EMF/EMI/RMI interference, (do not connect to a USB 3.0 port or use a USB 3.0 cable to a USB 3.0 port as that is well known to cause serious EMF/EMI/RMI interference to Zigbee communication).
If you still have problems after that then enable debug logging, replicate the issue, collect diagnosics, and report a new issue on GitHub to get help analyzing the debug logs → https://www.home-assistant.io/integrations/zha#reporting-issues