Home Assistant VPN and Google Assistant

Hi guys,
I recently installed a VPN client on my router which connects to a outside VPN server. Therefore all my home traffic is masked. Unfortunately when I enabled that setting Google Assistant stopped working although Home Assistant can still be accessed from WAN on the “old” IP and port. I’m guessing that because the server doesn’t get a respond from the same IP it can not authorize.

Do you have any idea what should I change to make Google Assistant work again?

Did you happen to solve this?

No I just enabled VPN on computers I use.