Voice Assistant, STT recognition, what else can I do?

Configuration Voice Assistant
21

Fair enough, and I may move over and try that. This afternoon I moved whisper to an docker that could use the GPU with a much larger model, and that seems to mangle fewer words in the STT portion. Using the Qwen3:4B with a lower temperature and that, with NOT “prefer handling locally” gives me 3.2 seconds including saying “Turn off the hall light” and it detecting the end of speech. That’s reasonable, and the larger whisper model seems to not mangle so many words, though I spent a lot of time debugging (a restart actually fixed most of it – something does not reset right somewhere as you make changes without restart).

Those two models take up 11 of 16gb memory so space is pretty good.

I honestly still do not (despite reading most of it) understand your approach. The documentation is fairly complete but there’s a lot of context and terminology I think I lack. Please… I’m not saying that your approach is bad or anything, I was being literal, I just didn’t understand it. My brain doesn’t fit what most people do in the world of AI – just cut and paste and go. I have to understand WHY I am doing something before I head down that path. Maybe I need a few more readings.

But I definitely think the big GPU has given me options. What’s unfortunate is the fragmented mess you get – HA running in HAOS, Ollama running native in windows, Whisper running in a docker inside ubuntu inside wsl. I assume at some point the winner in the “best of” contest for voice gets migrated more into HA so it can be an addon and magic and run in those dockers.

22

Doubtful. The GPU requirements are beyond what you should expect from HA. I’d expect it to continue to connect externally to inference services.

23

And to be fair if my HAOS was running inside of VM in linux or something like that, I would be better off than windows. With windows and HyperV as my main home for stuff, it’s limiting, but I need a big windows box for my security cameras, the NVR I use runs only on windows and isn’t happy (from what others have found) in any VM.

So… we work with what we have.

24

Just another thought here about STT. Did you try Home Assistant Cloud also?
I am not a native English speaker and I definitly have an accent. And for me the nabu casa stt works much better.

25

Sounds as if the weak point is the Atom M5 Echo. I get very good results with ESP32 S3 Box 3 and Speech to Phrase. With the exception of simple turn on/off commands, nearly everything is custom sentence/intent script - used a lot of ideas from here:

One of the advantages is that if you later go down an expensive local STT route, the custom sentences will still work.

26

Only for a moment, if I am going to invest in local voice assist, I don’t want to keep a cloud connection at all, it defeats a lot of the purpose. Nabu Casa could be unreachable, or while they are good people now, who knows what the future may bring.

I have an original Atom Echo, and two Atom Echo with the S3 chip (I think model C-126), and one Voice Preview Edition.

I have done all my testing to date with a quiet room and standing near the device.

The VPE has a better mic, definitely, but I do not see that under ideal conditions it had better recognition (it’s the same pipeline, so that really says under ideal, quiet conditions it basically heard the same sounds when I spoke).

The new M5 devices suck, frankly – the mic is fine, but the lack of any feedback makes testing hard, there’s no colored light to show it heard the wake word, no processing light so if your request aborted in pipeline space you wait forever to wander back to the computer to see what happened.

The original Echo is where I did most of my testing (I put the VPE in a bigger room and planned to try it with guests also eventually). If you are within a few feet and quiet, I do not find it any more difficult to capture the audio, and the lights are a great help.

For both echos I cannot hear the response well, so I redirect it to a nearby google speaker. This makes a huge difference and actually makes them usable, though maybe because my pipeline is not perfect yet, the lack of visual feedback is a serious issue with the newer M5’s.

My hope was that there would be a “Voice Non-Preview Edition” soon, and I would buy a bunch. As there hasn’t even been rumors of such (has there?) I may just order a few more.

But… long story short… I don’t think it’s the satellite, it’s my speach and the native STT whisper application don’t seem to get along well. So far (only a few tests as it was late once working) the GPU powered large model whisper seems better.

27

The HA team is exceptionally bad at keeping secrets - so as soon as another Voice device is on the way I’m sure we’ll all know quickly. Also given how much work they have in the platform I strongly expect it’ll just be HW improvements like mics, etc.

Because that Im already looking at collecting a hoarde of old Amz Echo Dots and doing what Thyraz did:
Yet another 3d printed case to combine the Voice Preview Edition with the Echo Dot 3 Speaker - Share your Projects! - Home Assistant Community

28

I’ve got a ton of mini’s, two “home” and two max google speakers. I may look into whether I can cannibalize them as well, eventually. But will remember this as an option as well.

The M5 Echo devices are actually easy to hook into a different media player (like a mini), so my little worthless-speaker M5 devices have their speakers turned down, and I play the responses to the minis. Introduces maybe a second of delay, but nice and loud and good quality. Probably exactly what’s happening with the dots but software not hardware redirected.

The VPE looks a bit more challenging to hack the same way, the code for voice output is a tangled mess compared to the Echo code, plus the built in was good enough for testing.

I’m a long way from getting rid of google though. I need to investigate a keep/lists alternative (I know HA has some) but a huge item I use far too much for my mental sanity is “Hey Google, find my phone”. :rofl:

29

THe VPE has what’s called a Grove port…

It’s actually quite easy to redirect the audio. Want TOSLINK? :wink: I mean who doesn’t like laser based audio with no ground loops.

HA Voice Preview Edition Toslink digital audio output - Share your Projects! - Home Assistant Community

30

It looks interesting, I had not explored what I might do with wires. THough it has a 3.5mm jack, right, so for a speaker you don’t really need it? (Though level/amy is an issue).

In the Echo case it’s just a matter of changing on_tts_end to do a play_media. I can’t recall if I tried really to figure out why the same code wouldn’t work on the VPE, but I know some things (like sensitivity settings) were hard coded to specific wake words, making it hard to remove a specific wake word (and meaning it might not be working for replacements, which I am using). The migration to a bunch of hard coded if/else set just looked more awkward than the M5 code, but I didn’t dig.

But redirecting to a nearby mini for the M5 is fine. I just don’t think the M5’s, at least the new ones, are for me. I really like the visual feedback of status. I may get one of the esp32-s3-box variants, but they had long deliveries when I ordered the other samples. With a screen I assume there’s a lot more feedback it could give. But I don’t get the idea it has as good of mic as the VPE, right?

If VNPE (Voice Non-Preview Edition) was on the horizon I might just wait, but am wondering if I should just buy a bunch of VPE’s and put where I have mini’s and see how well it works for a few weeks. That’s kind of pricey though if the verdict is “yuck” over time.

But today, at least, it’s the mic side… did you run across anyone finding a better mic array and a way to substitute that on any of these?

PS. I should add that music to me is mostly an afterthought. I got a great home theater and love it for video, but very rarely listen to pure music. That changes how I view a lot of things.

31

The esp32-s3 solutions are using only one mic. I have build quite a few of them. They are decent but the VPE has multiple mics and is far better. Also make sure you have the latest firmware installed on the voice pe

32

Yeah, last evening I just ordered two more VPE’s to start phasing them in.

I am running the “take control” version so it should be the latest. I’ve made a lot of changes and now getting pretty decent voice recognition as well as CA decisions. Here is the current pipeline (and advice welcomed):

  1. Wakeword is always on device.
  2. STT is Wyoming Faster Whisper running in python on windows with GPU and distil-large-V3
  3. CA is Gemma4:12b running in Ollama in Windows on GPU
  4. Both models stay cached all the time.
  5. Piper is the default, I would like better quality there but it is clear and understandable

Getting the larger model in STT (and making it acceptably fast by the GPU) helped a lot with not getting confusing speech. I shifted from docker to python on windows for STT to make it a bit more reliable headless. Gemma4:12b is supposed to be able to do voice recognition as well, but I failed to find a pipeline that would share the model for that.

The biggest problems now are related: (a) missed wake word, I just have to be close and facing the device even on the VPE, and (b) missed words between wake and end of command.

(a) I think is just the mic’s, it doesn’t live up to a google mini. The visual clue helps a lot, I don’t have to wait to see if it will respond if I get no blue light.

(b) I’m not sure. Sometimes “OK Hal, Turn on the hall light” becomes “the hall light”, but I really try to take a beat after Hal.

It’s also pretty awful at filtering out background noise, taking dialog on the TV and interweaving it with my voice. I looked a bit at pausing the TV if it hears the wakeword, but it doesn’t react fast enough so far.

Advice always welcome.

This does bring up a question: I can’t find a way to capture the audio (e.g. as a wav) for debugging, is there? I’d like to hear how much is me mumbling vs failed STT.

33

See Troubleshooting Assist - Home Assistant

You can get much better performance not using whisper and either using Nvidia-Parakeet or using Qwen3-ASR 1.7B via llama.cpp

34

I thought I had read all the docs but completely missed that. Thank you.

Relative to the others I am handicapped in that I need all the model runs to occur in windows - not docker, not linux, not a linux VM. Long convoluted reasons. I THINK the first of those requires a linux variant?

The second one looks like I could use the Local OpenAI STT and run it under my existing ollama installtion? Do you expect it to be better?

Wait… let’s define better: What I have now runs plenty fast, small fractions of a second for the STT portion. So I do not need better speed-type performance, but better speech recognition, notably teasing out my words from TV for example?

35

Either one can run in docker via wsl, but yes Qwen3-ASR would be runnable via your existing Ollama.

yes, to be clear I am the developer of OpenAI STT, in there includes a script that lets you take a collection of wav files and run through multiple STT / Whisper models and compare what they transcribed. Running on a collection of voice samples from various speakers, whisper is basically unusable (for me at least) as it is quite inaccurate especially for my wife.

36

I had that working once, but I really disliked all the hacks to keep wsl alive, or restarted; there’s no (at least on my windows) idle timeout you can turn off, you have to run something non systemd in windows scheduler and cross your fingers. It just felt fragile though it was working.

I keep trying to find a good alternative but I need to run Blue Iris on windows and it does not do well in any container, and I really don’t want to spin up a whole different hardware server. So… windows.

Ah… well thank you for that tool, I’ve used it and moved to another and back and forth, but not because of the integration.

If it will run in Ollama I can easily test it, and being able to automate some tests if I can capture badly transcribed audio would be quite helpful.

Thanks again.

37

If you are running things on windows and still in the experimental phase of which llm to use then you may have a look at lmstudio. That will give you a search option for the hugging face site. Details in llm’s and what goes in and out of the llm. It can also run as a service

About the wake word on the vpe. That should run local on the device. I noticed that for me it matters which wake word I use. Hey Nabu works best for me. It even get triggered from another room so I got the problem with multiple vpe’s that the wrong one triggers and based on its area it will do the action there.

38

I did use LM Studio for a while. I haven’t found ollama a chore to use and find models though.

I have both Hey Mycrof (which seems to work best) and a custom “Okay Hal” that I found which also works pretty well, maybe as well. My attempt at making a custom make word on one of the online sites failed badly.

39

I was delayed due to going out of town for someone’s birthday and I can’t get this to run (though I thought I had it running before). I don’t mean Ollama, but OpenAI STT. I’m getting an internal server error 500 which I think I have tracked down to Home Assistant core requiring OpenAI 2.21.0 and this requiring 2.45.0, and the (?) package management system refusing with

Unable to install package openai==2.45.0: 
   × No solution found when resolving dependencies: 
╰─▶ Because you require openai==2.45.0 and openai==2.21.0, 
we can conclude that your requirements are unsatisfiable.

I’m not good at reading git stuff but it appears you increased that requirement just yesterday +/- with commit 66ed9cf.

Will I require a later version of HA, or should I use an older version of the integration?

PS. I put in the prior version and it’s working. Experimenting to see how well, but so far hasn’t missed a word.

40

It probably was working, I did this to be compatible with the new HA version coming out next week and didn’t consider it would break, I will put up a fix.

edit: fix is up now