Google Assistant: How to control HA form my phone's Google Assistant

Nope, not necessary. I did not experience any delay longer than 5 seconds. Check this topic: How to Integrate Google Assistant and Home Assistant API using only IFTTT