If I was in this situation, I would suspect that my Zigbee network is the culprit.
When the scene is executed, numerous individual ‘turn off’ commands are transmitted, one per light, and the network fails to deliver all of them successfully to each light.
In other words, executing a scene that involves many lights is like a stress test for your Zigbee network.
There are extensive guides for improving Zigbee communications posted elsewhere in the forum.