Google Assistant Setup Issue - "Could not update the setting. Please check your connection"

OK.
I found one more thing to investigate.

I use IPSET to whitelist countries allowed to access my network. Google have data centres all over the world.
Need to rethink my security before changing this.