More a question than configuration for Google Assistant speech to text

Configuration
1

Hello,

My current setup:

Google mini => speech => Google home => turn on light or do other stuff

Would like setup:

Google mini => speech => HA core => turn on light or do other stuff

What I don’t want is to install the HA app on all my android, I have the hardware (Google mini).

Is this something I can setup? I have installed and configured Google Assistant SDK, Google Assistant, Google home in HA.
But they all fail to achieve the thing I want to do…

Regards,
Ward

2

Would like setup could also be:

Google mini => speech => Google Home => HA core => turn on light or do other stuff

I want to eliminate the need to have my HUE or other 3rd party linked to Google. Home.