Can I give different Google Smart speaker accounts different permissions to control devices in HA?

I have several Google smart speakers and android phones in my home, owned by different family members, each with their own Google account.

I’d like to give some of them permission to control smart devices like lights or media players in communal areas, but not in each other’s rooms. Is there an easy\simple way to accomplish this in HA?

No. Well maybe. But not in a way that would be easy if it’s even possible.

Doing this natively in Google Home has been a community ask for a while now.