I have it almost similar, instead of using cloudflares certs, i use nginx proxy manager to do those.
What i like is that you can even strict more with cloudflares firewall, by only allowing your own IP to access the site. Meaning, if you are not coming from home to access HA than you get denied. This of course has mobile issues once you are on cellular.
So to counter that, i have created a custom VPN profile for the iphones to turn on VPN once they are off home network to establish a internal connection.
This way, my IP is not accessible unless from inside, my devices automatically phone home, and all services are encrypted.
Additionally, my pihole keeps working on the devices regardless of where i am, as it will also work on VPN.
Ive not had any issues with speeds, but im not streaming anything.
This way i have been able to turn off ports for wired alrm, nvr cameras system, HA, synology and many more.
Hope these additional tips help