I work at Virginia Tech, and one of our undergraduate research capstones was showcased today that is around augmented reality and gesture control for home automation systems. While not mentioned by name, there is a point in the video where you see an interface that is clearly Home Assistant, so I thought I’d share.
And if there’s anyone at Nabu Casa that would like to find out more about this, feel free to DM me and I can try and make some connections.
I can assure you that most people coming to this forum are gesturing, but possibly not in a controlled fashion…
Voice control was recently introduced. I cannot see why position sensing in 3D cannot be extrapolated to interpretation also.
Bring it on!
As a side note: This will probably go better in certain Italian and Latin American locations where gesticulation forms part of the natural language. You may have challenges with teenagers monosyllable verbage, especially if they have been weaned off un-social media like the Aussie kids with the government ban enforced a few weeks ago.
Just “for inspiration”: the AR aspect of it would for me personally be overkill. But what I’d love would be gesture control by itself. This is something where the shortcomings of voice control (needs a wake word, needs sound) could be overcome with for certain situations.
Example: increasing the volume of a TV by voice control is a horrible user experience (“Hey Nabu” … “volume up by 10%” … volume changes), but having a “the person on the couch is pointing to the ceiling with their index finger => increase volume” logic would be rather intuitive. And you can control things without bugging other people in the same household.
The increase in control of your environment control for people with disabilities is where I would see this as most beneficial. Empowerment. Not everybody has arms to wave about. Some have a mouthpiece, tongue and very little else as a human-machine interface. Conversely, people with uncontrollable tics, such as suffering from Tourettes may not be able to consistently wave their limbs about, and need some filtered pattern recognition to sort it out.
Until you yawn/stretch/point at the weird insect on the ceiling 
The built in laser cannon fly zapper takes care of that.
Real question: has somebody built one of these? A radar tuned to insect size and movement, at the ready with a short burst of intense heat to zap them out of existence, or even just blind them? Built to avoid anything larger such as pets and humans. Whatever could go wrong?
Let’s not derail the thread, but as a one-off-answer: Mosquito laser - Wikipedia
Agreed, but that doesnt mean there cant be different approaches for different people and living circumstances. And i understand your case where voice is actually a vast improvement.