A quick story about where I am coming from - I once started my Smart Home journey in Google Home, first some lights, then some smart speakers and I got used to voice commands.
I became more aware of the data privacy issue that comes with sending audio and all other data to big tech servers. Then, local AI started to appear and I made it my goal to run my home via voice commands and keep my data locally.
Here is my solution (which I am quite happy with):
For STT, I am using Seeed’s reSpeaker XMOS XVF3800 with the XIAO ESP32-S3 and also tried the HA Voice Preview and the Satellite1 from FutureProofHomes. The Seeed device though, has the best speech recognition, in my opinion.
I kept my Google speakers with the mic turned off (and a piece of tape to cover the annoying red LED) for audio output, turned the STT devices to silent and wired it up using my Mic to MediaPlayer integration. To decrease the latency, pre-announce TTS is turned off in Music Assistant.
For the command processing, I am using my AI Plugin integration, which is connected to a local LLM on an RTX 3060 Ti running on a local server, but can also be connected to any cloud model or bigger local models on bigger GPUs.
The HA entities I am exposing using the native Model Context Protocol Server integration.
For the audio processing I am using the Home Assistant Cloud service, which gives me the lowest latency. I’ve been trying faster-whisper and piper on my local machine but I am not happy with the latency and transcription quality. But I am sure this can be fine-tuned to achieve complete local processing.
This all, of course, can be very specific to my configuration so I would love to see others trying it!
- AI Plugin: an Assist conversation agent
- Common commands (time, sensors, on/off, covers, volume) skip the LLM and answer in about 0.05 s.
- It streams replies, so speech starts after the first sentence, and its prompt layout keeps Ollama’s prefix cache warm.
- If the model claims an action it never took, it gets one corrective re-run.
- It also has voice timers, memory, web search, MCP servers, an ai_task entity and 6 languages.
- Best model I found for 8 GB: gemma-4-12B-it UD-Q3_K_XL. It scored 100% on my tool calling tests with a ~3.6 s median. qwen3.5:9b is faster at ~2.9 s.
- I’ve only tested it with Ollama. Reports from other backends are welcome.
2. Mic to MediaPlayer: any satellite as the mic, any media player as the speaker
- Works with Wyoming, ESPHome, the companion app and VoIP satellites.
- Replies play on your Sonos, Cast or DLNA speakers.
- Its announce service pauses music and resumes it afterwards.
- Follow-up listening still works.
- With both installed, AI Plugin filters out the satellite hearing its own reply, so it doesn’t loop.
