Google Assistant Cloud behind HAProxy?

So I have Home Assistant setup behind HAProxy which is on OPNsense. I used letsencrypt and my already used duckdns.

Local fulfillment works for the Google Homes but now secure devices won’t work. I know they are supposed to be through the cloud still. I even reset up the Google Home integration from scratch and it will not connect via the cloud period. The only way I can now get Google home to work is by local fulfillment but then I lose access to my secure devices.