For those who came for this solution, I disabled both Wi-Fi and Bluetooth as instructed on my Yellow. But then, after a reboot, my ZHA failed due to missing /dev/tty/AMA1. Maybe disabling BT on CM4 somehow also disables the Gecko Module on the Yellow @xpal. So maybe the best approach for HA Yellow is only disabling the Wifi, which I found decreased the interference to ZHA quite a lot.
A quick note to confirm that the solution presented by @oxyd and enhanced by @el_migu_el works perfectly also for Yellow with CM5. I followed @el_migu_el 's instructions (I added the dtoverlay statements under [cm5] section rather than [all] - just a mater of taste as they’d work either way).
Everything seems to work fine, wlan0 and hci0 are gone, Zigbee etc. works okay, no issue.
Thank you!