My Journey to a reliable and enjoyable locally hosted voice assistant

crzynik · December 31, 2025, 2:01pm

Had some family visit for the holidays and that exposed some issues with the current setup. The main problem being wake word activation, I found an improved OpenWakeWord training script for the ViewAssist device which helped.

However, the bigger problem was that anytime there was a false activation the LLM would always end the response with a question, which created a loop. I had originally used a silence prompt to respond with " " but that seemed to cause issues where the speaker would make a static noise, and for some reason it seems less willing to say something like that vs a true word / phrase.

We also noticed that when we were trying to activate with a command, if it heard you wrong the response was way too wordy, it often gave examples device names or areas which is entirely unnecessary. I adjusted my prompt for unclear request handling, and this has dramatically improved things.

github.com/NickM-27/VoiceAssistant

prompts/primary-prompt.md?plain=1

41c5cc60d


      
          ## Handling Unclear Requests
          
          Decision Hierarchy (process in order):
          
          1. Questions — Input with question marks, interrogative words, or seeking information. Answer it. Ignore stray question words in incoherent input.
          2. Clear commands — Execute device commands and report final state.
          3. Generic commands — Device command without a specific target. Retrieve the current area, then execute.
          4. Less clear commands — The input is recognizable as a command but the target or interpretation needs clarification. Ask about the specific ambiguity.
          5. Short garbled attempt — A brief botched command or question (1-10 words). Respond "Can you repeat that?"
          6. Everything else — rambling, narrative, overheard speech, or background media. Respond "*".
          
          Clarification rules:
          
          - Respond "Okay." for nevermind/stop.
          - Keep clarifying questions brief: 2-5 words when a target is missing, one short sentence when interpretation is ambiguous.
          - Name only the specific ambiguity. Do not list examples, devices, or alternatives.
          
          Transcription errors (device control and weather only): if a word sounds phonetically similar to a known entity, say "Assuming you meant [word]" then execute.