Training the voice assistant to actually be useful

Configuration Voice Assistant
1

So I’ve been working on training the voice assistant- I have a satellite 1 from future proof homes - to be actually useful. Like add things to a grocery list, set timers, turn lights on, turn lights off turn scenes on turn scenes off, etc.. It’s amazing to me just how stupid the voice assistant really is today,

I have a Mac mini running a local Qwen LLM and my voice path is Home Assistant first, then the Qwen LLM and then Google so whatever fails ultimately goes to Google.

I recently tried to use the Tater Totterson voice ecosystem, but ultimately gave up after realizing that there is no sendspin in his native firmware support so MUSIC assistant is essentially useless.

So now I’m back on the core future proof homes native firmware on the satellite 1 ( with limited microphone support) - basically what VPE is today.

I should also mention that I am using Grok Bot to help me figure all this out. I had the bot write an application to track all voice commands in real time. Say a command and it appears. If it works or fails, that’s indicated and then I can tell it what I intended to do and have Grok Bot try and fix it, which it’s mostly good at doing. It also indicates the full paths of each command so I could figure out what is failing and where it is failing. That’s probably been the most aha moment of all. The Home Assistant debugging for voice assistants leaves a lot to be desired. Writing this little application was very enlightening as to what path was being used and where I needed to make changes.

I must say it’s all still very frustrating.

image
image1772×868 129 KB

image
image1101×825 80.8 KB

But we are making progress. I can now stop, continue, pause, and ask for who’s playing on the media player, add things to my grocery list & cross them off. Start and stop named timers. Basically all the things that Alexa previously did - that is the ultimate goal sans the name Alexa, which I have to use - Hey Jarvis - but that’s fine for now.

It’s getting better, but it’s still not fully there. Even with the help of AI to try and tune this damn thing. Grok Bot is helpful but not completely understanding how this ecosystem works. We’re still a long way off from replacing Alexa.

Yes, sometimes swearing at the Grok Bot has added benefits.

image
image394×647 42.4 KB

Honestly IMHO, the Home Assistant team really needs to spend more time getting the voice assistant to be self configuring and work out of the box. We should not have to train all this shit just to make it work. The average user would have absolutely no idea how to do what I j have done over the past week, especially if they don’t subscribe to an AI service. It’s way too complicated and needs to be simplified and boiled down to just work.

Come on guys!!! Step your game up and figure this stuff out. It’s why we’ve been paying the money for the last several years to subscribe to Nabu Casa. I would suggest that voice assistants are the number one priority that Home Assistant should be focusing on right now. Everyone has either Alexa or Google Home and we all want to replace them with local voice assistants that just plain work. We shouldn’t have to train them to do all the basic functions of the commercial devices.

2

This project should help you understand the important points of good context engineering (What you’re trying to do.)

Sorry it’s long it’s one of the most complicated parts of the field rn.

3

Nathan, you are a brilliant man. I’ve read many of your posts over the years and admire your depth of knowledge. However, you just seriously reinforced my exact point of the paragraph above.

The post you referred to is way over the heads of 99% of all Home Assistant users. We should not have to figure this out. It should be just inherent in the system. Period.

4

It won’t happen. It took me two full years for that research. AI and LLMs are in infancy and expect at LEAST two more years before turnkey in HA. For general consumers Consider the core assist / conversation loop a few steps past research and into early implementation phase. People like me are just now figuring out the shapes required to make stuff work

People expect ai to be this all knowing easy button it is not. I should know it is literally my day job.

To work! (How I suspect you want it to work) an LLM needs context. It needs a well grounded prompt space and well defined tools. Ha has basic tools and a very basic prompt space that’s good for little more than see those entities go do… At the moment.
They could spend time on context build, context collection tools prompt substrate caching etc and… They still won’t have an easy button.

The short version is context is extremely hard and the is no easy button.

The closest thing I’ve got for you is

Which is the current consolidation of research of the megathread and will STILL require reading

I’ve got more faith in people I guess I’ve talked to hundreds who genuinely get the concepts. I distilled the Friday’s Party thread. If it’s hard to read you really don’t want the academic paper version. People need to consider what it takes to make this stuff work. :wink:

5

I’m being pedantic, but for the sake of clarity…

A voice assistant is not the same as AI. I have all the features you mention in your original post using just Speech to Phrase, custom sentences and intent scripts.

6

There is a previous project thread for timer/reminder intents that gives some reasons why an LLM-based solution may not be the best option.

If you peruse this forum, you will find numerous threads with users stating that every second of dev time spent on voice is wasted productivity that should have gone into some other aspect of the overall HA project. Personally, I’ll be happy when I can fully phase out (or, even better, reuse) our Alexa devices, but there are plenty of other things that I could see deserving more attention.

7

I agree with you in some ways. I’m a bit baffled how all the wind seems to have left the “Year of Voice” sails. It’s unclear that the OHF has a strategy (try finding a clear roadmap!). Maybe they are waiting for the Tater’s and Nathan’s of the world to wander through the wilderness finding a good path that OHF can then claim and follow as though they invented it. Maybe they are trying to figure out how to sell their own cloud solution (on which they make money, look at all the advertising of it) instead of trying to stay local. Maybe they realize they are painting themselves into a corner where $35 hardware (which you can now buy for about $110) that has become so core to the Home Assistant religion can no longer run the LLM everyone expects when they replace a google mini or alexa, and are unsure how to tell people the truth.

I think it’s a bit of all of the above.

But I do believe at this point it should have been obvious going into doing your own LLM that it is not a well designed path the OHF has laid out yet, and is something of a moving target, not only in the HA world but the LLM worlds as well. So you should not be surprised it is a lot of work. Just look at the “Voice Preview Edition” and ask where the post-preview version actually is?

It’s like nuclear fusion being 10 years off – and it has been 10 years off for the last 40 years, except they decided on one year for Voice.

I think the other thing that is under-appreciated is that what is a local AI assistant’s role and expectations vary hugely among people. With Alexa and Google we were told what to expect, and if we didn’t like it, no one cared what we thought. If you are building your own, you are almost certainly NOT trying to replicate what someone else has, but have your own expectations that you want to build into the processes, prompts, scripts, etc.

So I while I agree with much of the sentiment, I don’t see why you are suprised.

8

Saying, I was surprised as being a little tongue-in-cheek. I pretty much. agree with everything you guys have to say above. It is a ton of work and no, this is not an easy problem at all.

I went down “there waiting for the Tater’sof the world” rabbit hole and that all has a whole host of it’s own issues as well to sort out. It’s a very heavy lift.

I’m almost happy for now with what I’ve been able to cobble together. I feel like it’s almost ready to replace the Alexa’s in our house. A few more things to iron out.

9

Interesting timing, yesterday I ripped almost all my entities from google home. But I’ll probably need a few weeks of going without google to really see what the gaps are.