Voice Assistant, STT recognition, what else can I do?

Configuration Voice Assistant
1

I’m experimenting with local voice, I have an three M5 echo’s (two flavors) and a Voice Preview Edition.

I did some experimentation with an Ollama model but I can’t run it on my home assistant system, no GPU and it just doesn’t work well without. While I can run it on my desktop, I do not want to head down that split setup for anything permanent.

I really don’t want conversation – I want a specific command to be recognized. I’m happy with limited creativity.

So far I’ll fall the results underwhelming, though somewhat encouraging. Wake Word Detection is pretty good - not great, but pretty good.

Voice understanding is mediocre. I’m going to show a specific example but they are rather like this. I have no reason to think I mumble, by the way, and do not have a strong US accent (think Virginia suburban not rural).

I have lots of CPU power (12 cores assigned to HAOS on a fast machine), 32gb of memory. I have:

Conversation Agent = Home Assistant 
Speech to Text = Faster Whisper
Everything I can find set to english 

Whisper: 
Model = base
library = faster-whisper
custom model = unspecified 
custom model type = faster-whisper
beam size = 5 (i've also tried one) 
task = transcribe
vad clip = on
local files only = off

During the sample below should nothing interesting at the info level (below are two consecutive runs, the first fails the second works:

INFO:faster_whisper:Processing audio with duration 00:02.144
INFO:wyoming_faster_whisper.dispatch_handler: Turn on the network close it like
INFO:faster_whisper:Processing audio with duration 00:02.560
INFO:wyoming_faster_whisper.dispatch_handler: Turn on the network closet light.

I’m not saying it works awfully, but that I get too many like the above, where “closet light” became the nonsense “close it like”.

Is there something I should be doing differently in this setup for more accurate recognition?

I’ve experimented with various recommendations on model and what I have seems to work fastest so far, but none of them seem to really improve recognition accuracy. In particular, I’m disappointed it would come up with something like “close it like” when it’s comparing against specific HA related things – am I doing something wrong to keep it from tying recognition to relevant phrases?

Here’s the YAML for the one that failed from debug info:

stage: done
run:
  pipeline: 01kxpfzx18ka55wwrdtsk076dc
  language: en
  conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
  satellite_id: assist_satellite.vpe_1_assist_satellite
  tts_output:
    token: _KzsQoPM1XFkXlDYsIt2NQ.flac
    url: /api/tts_proxy/_KzsQoPM1XFkXlDYsIt2NQ.flac
    mime_type: audio/flac
    stream_response: false
events:
  - type: run-start
    data:
      pipeline: 01kxpfzx18ka55wwrdtsk076dc
      language: en
      conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
      satellite_id: assist_satellite.vpe_1_assist_satellite
      tts_output:
        token: _KzsQoPM1XFkXlDYsIt2NQ.flac
        url: /api/tts_proxy/_KzsQoPM1XFkXlDYsIt2NQ.flac
        mime_type: audio/flac
        stream_response: false
    timestamp: "2026-07-22T22:34:42.983129+00:00"
  - type: stt-start
    data:
      engine: stt.faster_whisper
      metadata:
        language: en
        format: wav
        codec: pcm
        bit_rate: 16
        sample_rate: 16000
        channel: 1
      audio_processing:
        requires_external_vad: true
        prefers_auto_gain_enabled: true
        prefers_noise_reduction_enabled: true
    timestamp: "2026-07-22T22:34:42.983371+00:00"
  - type: stt-vad-start
    data:
      timestamp: 1050
    timestamp: "2026-07-22T22:34:44.044444+00:00"
  - type: stt-vad-end
    data:
      timestamp: 2540
    timestamp: "2026-07-22T22:34:45.517941+00:00"
  - type: stt-end
    data:
      stt_output:
        text: " Turn on the network close it like"
    timestamp: "2026-07-22T22:34:45.961412+00:00"
  - type: intent-start
    data:
      engine: conversation.home_assistant
      language: en
      intent_input: " Turn on the network close it like"
      conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
      device_id: 29f30bd51156a47df2e04d4092dfb774
      satellite_id: assist_satellite.vpe_1_assist_satellite
      prefer_local_intents: true
    timestamp: "2026-07-22T22:34:45.961559+00:00"
  - type: intent-end
    data:
      processed_locally: true
      intent_output:
        response:
          speech:
            plain:
              speech: Sorry, I am not aware of any device called network close it like
              extra_data: null
          card: {}
          language: en
          response_type: error
          data:
            code: no_valid_targets
        conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
        continue_conversation: false
    timestamp: "2026-07-22T22:34:45.982009+00:00"
  - type: tts-start
    data:
      engine: tts.piper
      language: en_US
      voice: en_US-kristin-medium
      tts_input: Sorry, I am not aware of any device called network close it like
      acknowledge_override: false
    timestamp: "2026-07-22T22:34:45.982076+00:00"
  - type: tts-end
    data:
      tts_output:
        media_id: media-source://tts/-stream-/_KzsQoPM1XFkXlDYsIt2NQ.flac
        token: _KzsQoPM1XFkXlDYsIt2NQ.flac
        url: /api/tts_proxy/_KzsQoPM1XFkXlDYsIt2NQ.flac
        mime_type: audio/flac
    timestamp: "2026-07-22T22:34:45.982362+00:00"
  - type: run-end
    data: null
    timestamp: "2026-07-22T22:34:45.982429+00:00"
started: 2026-07-22T22:34:42.983Z
stt:
  engine: stt.faster_whisper
  metadata:
    language: en
    format: wav
    codec: pcm
    bit_rate: 16
    sample_rate: 16000
    channel: 1
  audio_processing:
    requires_external_vad: true
    prefers_auto_gain_enabled: true
    prefers_noise_reduction_enabled: true
  done: true
  stt_output:
    text: " Turn on the network close it like"
intent:
  engine: conversation.home_assistant
  language: en
  intent_input: " Turn on the network close it like"
  conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
  device_id: 29f30bd51156a47df2e04d4092dfb774
  satellite_id: assist_satellite.vpe_1_assist_satellite
  prefer_local_intents: true
  done: true
  processed_locally: true
  intent_output:
    response:
      speech:
        plain:
          speech: Sorry, I am not aware of any device called network close it like
          extra_data: null
      card: {}
      language: en
      response_type: error
      data:
        code: no_valid_targets
    conversation_id: 01KY5ZF1X6Y0AP7TX38BFZKHT4
    continue_conversation: false
tts:
  engine: tts.piper
  language: en_US
  voice: en_US-kristin-medium
  tts_input: Sorry, I am not aware of any device called network close it like
  acknowledge_override: false
  done: true
  tts_output:
    media_id: media-source://tts/-stream-/_KzsQoPM1XFkXlDYsIt2NQ.flac
    token: _KzsQoPM1XFkXlDYsIt2NQ.flac
    url: /api/tts_proxy/_KzsQoPM1XFkXlDYsIt2NQ.flac
    mime_type: audio/flac
finished: 2026-07-22T22:34:45.982Z
2

Hmmm… switching to Speech to Phrase seems to be a win…

As long as I’m looking for fairly precise requests, is there a downside?

3

Not really…

With the limits you’ve described it’s probably your best bet.

4

You mean not running a LLM?

5

Well, if turning on the closet light is your problem and it often recognized you speech as ‘close it like’ then go to the closet light entity and add an alias called ‘close it like’
That may help in this situation

6

It was an example. The next time it may be something completely different, often it is no recognized speech at all. None of the mistakes seem to have a recognizable pattern.

It definitely is a matter of distance. I’m spoiled by google mini’s all over the house. With these (even the VPE which is supposedly much better) I need to be close (say 6-8 feet or closer), facing the device, no background noise.

With the mini’s they can pick out voice with the TV on or other background, from anywhere in the house. If I recall back when I bought them they were about $20. I’m MORE than willing to pay $100 each if I could replace them each with a similar local only device, I’m not trying to be cheap – but I really hate taking a huge step back on voice recognition.

NOT intent recognition. I’m delighted to learn the right phrases. I do not need some (#&$@(&## AI trying to figure out “oh, when he said Den, he really meant dining room”.

7

This is because of the array of far field mics these devices have that the VPE and the Atoms do not.

Welcome to the device subsidy - it’s well known that these vendors (Goog, AMZ, etc.) sell the device at a loss to lock in the user. Unfortunately for Amazon they didnt capitalize on the user base… (see 13B loss for the AMZ hardware BU two years ago that nearly ended the program.)

If you want the mic experience the device would cost at least that - and there are a number of projects here where people are doing similar work. (Adding far field mics to vpe-like devices including the FP1) Most seem to be based on cannibalizing existing Amz Echo or Google Home devices to harvest the mic arrays.

The AI would do a better job tbh. It translates my Chicago-Louisiana-Texas twang perfectly fine and understands that ‘a Coke’ can indeed be Dr. Pepper.

8

Ok, $200. Seriously… up front cost is not the stumbling block really. It’s to get local-only, and recurring cost is relevant (see below).

Well, a good example (if not relevant to anything my smart home can do). If I ask for a “Coke” and got a Dr. Pepper it’s a failure on the AI’s part.

But the main reason for not using the AI is not that, as I suspect I can rein it in. The main reason is I need to set up a new, power hungry system. Let’s say it needs 300w to sit there and run a big AI model on a big GPU. Ignore that it’s maybe $1500 to buy, it’s also $1 to $2 per day to feed it power. Right now my HA is just an after thought in terms of power usage (a VM on a big PC running security cameras, network monitors, storage, etc.) And at some point these costs push down into UPS capacity, POE capacity, etc.

Of course, fi the satellites need big horsepower and I have 10 x 50w each or some such, that’s worse.

And yes I realize google (et al) want me in their ecosystem unable to get out and ads are a subsidy. I’m not trying to be cheap on buying, though I am trying to be aware of recurring costs.

9

you are WAY overestimating the power requirement.

Im running a RTX5700TI (16GB) adapter as an eGPU connected to a NUC 14 ProAI.

It uses 35w idle - which is 80% + of the time.

My DGX Spark (More than most ever need for home inference) averages 65w and bursts to 150.

It’s nowhere NEAR what you’re estimating.

And in TX - Coke == all carbonated beverages, and AI can discern that - a speech processor cannot.

10

I was going by my desktop. I’ll need to do some research. I see some systems (mini forums) have unified (?) memory that sounds interesting as a way to get huge models but give some back if needed elsewhere.

I’m just sensitive. I have a network closet that pulls a steady 280w (firewall, switches, HA, secuity, zwave, etc.), and a desktop that hangs in around 290 even with monitors off. I’ve been attributing a lot of that to it’s GPU ( RTX 3080 ti, so old and not big) but maybe that’s unfair. Hmmm… my measuring is including a little 8 port 10g switch, but I would think that’s small.

Those two together are a bit over $2/day for that background (the desktop is a lot more if I’m crunching photos or something, that’s background).

But yes… need to do some homework. Including if I could put a big GPU in that system that runs HA now (windows 11 pro) and share it to the VM running HA. I THINK I can, but haven’t tried.

But… if I order a “Coke” and get a “Pepsi” I’m irritated, but Dr. Pepper would have to go back. But we have to excuse foreign countries like Texas their quirks. (OK, not foreign yet, but I think they want to be :slight_smile: )

11

I learned something that wasn’t obvious. The wake-word 1 and 2 in the setup of these is tied to the voice assistant 1 and 2. I figured that since wake words are, sort of, a fallback, that the assistant 1 and 2 were also – but they are paired to the wake word. So you can set up two pipelines and test without making any changes other than the word you say. Neat.

I’ve been experimenting with STP, Faster Whisper, and ollama (qwen3:4b-instruct).

There doesn’t seem to be a “best”. STP is quickest but misses a lot. faster-whisper may be worse, both seem flakey at misunderstanding things. Ollama is interesting, it understands most things but is more creative - for example I have a kitchen_light (the main one), kitchen_sink_light, and kitchen_closet_light. The two dump pipelines generally get these right (if they understand at all), but frequently ollama decides I said “lights” instead for “Kitchen Light” and turns them all on. I’m not quite sure why STP and faster-whisper does not do the same, since all are in an area “kitchen” but ollama seems more prone.

The model for the 200 +/- exposed entities fits in my RTX3080ti with space to spare, even while using it for my two monitors with the model loaded only 8gb of the 12 available of dedicated memory are used. Other than initial load time, it’s cached and fast. But I’m going to need to buy something new if I want AI for voice and so have to decide between:

A new HA system with plenty of GPU horsepower, e.g. something like one of the minis forums mini workstations, or

Using my existing system (Z890 Intel Ultra 7 20 core with 128gb memory running security native in Windows 11 pro, and HA and a network monitoring system in VM’s, and adding a big GPU. It currently has a really old GTX970 just for transcoding of the security cameras, probably not really needed for that.

I’m tempted to stick with virtual and replace that GPU. I should have plenty of power (850w) and space (ASUS ProARt Z890) but have to look as I know there are a couple of network cards in there for 10g. But I like having HAOS on a VM where I can snapshot and rollback if they screw up an update (or if I do).

But I think I need to experiment more – really none of these are up to the google mini standards.

12

If you’re looking at running models locally, you should at least be using the latest models. Qwen3 is over a year old and unsloth/Qwen3.5-4B-GGUF · Hugging Face is significantly better.

13

Ah, thank you very much. Is there a good place to find the latest and greatest official recommendations? I’ve spent days now with lots of google searches and conversations with chatgpt and others to try to find specific recommendations, with often conflicting information.

I have ollama running on my desktop, and found I can quickly get a RTX5060ti that should be adequate and put it on the windows system (native) that houses HAOS (as a VM). But my desktop (RTX3080ti) should be good enough for testing, so far only hitting 8gb of 12 available.

14

home-assistant-datasets/reports at main · allenporter/home-assistant-datasets · GitHub is generally a good place to see the current models that have been tested with it. That said for that size of model the only real options are Gemma4 (E4B or 12B either as QAT) or Qwen3.5 (4B or 9B).

One advantage of Gemma is that it can handle both the LLM side and the STT side, and on the STT side it can be given a prompt which allows it to be biased towards home vocabulary. For example: VoiceAssistant/prompts/stt-prompt.md at a387b474c3df01b36a26cf796ecb4a468e35ac22 · NickM-27/VoiceAssistant · GitHub

15

Wow, I need a lot to learn, that all comes through as greek.

I tried downloading LM Studio and searching for the one you mentioned. I can find an unsloth qwen3.5-4B-GGUF but it doesn’t say hugging face, if I search for hugging face I see nothing similar. And I think I need the OpenAI integration to use it. Maybe. But nothing I’m reading is HA specific.

I suspect I need to go somewhere and learn which numbers mean what.

But until I get a lot smarter, is there a list of steps (or at least tools) I need to run it on windows, link it to HA in HAOS. I’m just rather stuck where to get started since most AI related sites are… well, just about AI in general use, not HA.

Obviously the one i found (Ollama and the older qwen) are from HA sites that are out of data.

Is Nabu Casa keeping a site updated with what they recommend, or is this all being done by the community and so you just have to stay in touch and know who is on track, and who is just screwing around?

16

But I have it working with that model (or I think it’s the right one, appears as qwen3.5-4b in lm studio). It fits nicely, about the same size as I was using. I’m not smart enough to tell the difference yet.

I think I’ll try the gemma4 ones also, and see if they answer better.

Thank you for the pointers.

17

Wow, did I. A lot has happened.

I bought a RTX5070ti 16gb, and replaced the GTX970 I had in that system (that card was not used for home assistant, it was there for video transcoding and hardly ever used).

My network closet (lots of stuff) was averaging 280-283w in the nighttime wee hours (nothing actively running). With the new card – 290w last night.

So at idle, a whopping 7 wats more. Idle. Now there’s a cost every time I ask HA to turn a light on potentially, but those are few and far between.

So… yes, I VASTLY over estimated it, I just assumed to get this working I would need a new box with a big card since I thought it needed to run in linux. What I did was run Ollama in windows in the hypervisor, and pointed the HA guest at it. No new box, big card (those things are huge!) but idle power draw is negligible.

Now getting a model I like is hard. There’s SO much conflicting advice, even how to connect (been experimenting unsuccessfully with the Extended OpenAI Conversation for example). At this instant I’m running qwen2.5:14b (because most things point to qwen and suggest staying away from the 3.5). Now that I understand the system prompt templates and such a bit better I think I’ll try others, including from the scoreboard above.

I doubt I need the 14b but it fits, curious to see if it works better than the small ones. It’s using 11.5/16gb.

Now to find other models and experiment a bit. I wish I could find a more natural speaking voice though, but I guess I need to read up more on piper.

Linwood

18

The E4b or equivalent is probably going to be plenty unless you plan on authoring code. I RARELY run anything over Qwen’s 7b for a daily driver - anything bigger I use for code. Your agent should specialize in tool use and speed.

Dont expose EVERYTHING - in fact as little as possible.

I suggest installing a framework - of course I suggest mine (ZenOS) but find the one that works for you - dont reinvent the wheel - prompts are hard and counterintuitive. The better your tools are the better your success will be.

19

At the moment I have about 170 of 1200 entities exposed, so it’s pretty pruned. And I really would like to add some sensors and controls, e.g. HVAC controls (a bit complicated as it’s a 4 zone system controlled by node-red).

Anyway… point taken on model size. I have both sitting here and trying to decide the best.

I found and read over a bit of ZenOS. I haven’t tried it because in brief reading (a) I’m not sure it’s the direction I want, and (b) it looks more invasive than integrations I can bolt on and I worry about screwing things up.

But re (b) I could just snapshot and try and rollback so it really doesn’t matter, however:

(a) What I want is simple – if I say “turn on the kitchen light” I want it to find the entity named “kitchen light” or “light.kitchen_light” if it uses entity_id, and turn it on.

That sounds so very simple but in many days of experimentation it doesn’t happen. A huge amount of the issues are in STT and/or audio capture. But a huge number are misunderstanding. I say “Turn off the Hall Light” and about 50% of the time it hears “Whole Light”. There are so many examples that I do not want to start putting in aliases (e.g. hallway and remember to say that not hall).

What I was HOPING is that the LLM would take the bad transcription and try to make it make sense. No one says “whole light” so find something close instead of deciding it means every light in the building (which is what it usually does).

Yours seems aimed at making it more capable, to do more complicated things. Honestly I get less and less sure every day that I want it to do that.

I want a dumb servant that does what it tell it, stretching only so far as to correct somewhat obvious STT mistakes.

Any guidance on that?

PS. I did get a VPE in addition to two M5 C-126 devices and a M5 original Echo. I don’t see a whole lot of difference in them, since I am testing only in a quiet area with negligable noise, and no spoken words (no TV for example). I think I speak plainly, I’ve done it all my life, I have negligible accent (or so both northern US and southern US folks tell me, despite me saying “folks”). The voice STT just sucks, so far. But I don’t see how to make that better.

20

This is what LLM solves at its core - full stop. You can see the examples early on in Friday’s party - takes copmpletely MANGLED words, mispronounciations whole bastardization of language and USUALLY gets it right.

BUT The minute you do that ^^^

If you DON’T have tools - then it’s dumber than a rock, and why did I go through all this crap just for a light switch :wink: