Experimental Gemini Live streaming voice flow for Home Assistant Voice PE

Share your Projects! Custom Integrations
, ,
1

Hi everyone,

I wanted to share a small experimental project I have been using locally with Home Assistant Voice PE.

The idea came from a limitation I kept feeling in the normal voice assistant flow: the device listens, records the full utterance, sends it, waits for processing, and only then starts responding. It works, but the interaction does not feel as natural as a streamed conversation.

I also built this because I wanted a comfortable Polish-language voice assistant for my home. I wanted something that works naturally in Polish, gives me a lot of control over what the assistant can do, and still integrates deeply with Home Assistant. This setup turned out to be a very good fit for that. In my local setup, it controls my home reliably and feels much closer to the kind of assistant interaction I was looking for.

So I built a first beta / proof of concept that connects Home Assistant Voice PE to Gemini Live through a local Home Assistant add-on.

The goal is simple:

  • lower perceived latency
  • more natural back-and-forth conversation
  • good non-English language support, including Polish
  • stream microphone audio instead of waiting for a full recording
  • stream response audio back to the device as soon as it is available
  • keep Home Assistant service calls in the loop
  • keep the assistant behavior configurable through prompts and settings

Short demo video, in Polish, showing the response speed I am getting locally:

There are two parts:

This is not an official Home Assistant, ESPHome, Nabu Casa, or Google project. It is also not something I would call production-ready. It is a first beta that was mostly vibe-coded, tested on my own setup, and published as inspiration.

That said, it works surprisingly well for me locally. The latency is low, the conversation flow feels much more natural, and Home Assistant commands can be executed through the Gemini Live session.

What I am hoping for with this post:

  1. Maybe someone will point me to an existing, cleaner solution that already solves this properly and that I can just install.
  2. Or maybe this can inspire someone more experienced with Home Assistant voice internals, ESPHome audio, or add-on packaging to build a more robust version.
  3. Or perhaps this can start a discussion about whether streaming voice interaction should be easier to support in the Home Assistant Voice PE ecosystem.

Current state:

  • works locally in my setup
  • requires custom firmware flashing
  • requires a Gemini API key
  • uses a local Home Assistant add-on as the proxy
  • supports streamed microphone audio and streamed response audio
  • exposes configurable prompt/language/voice options in the add-on
  • should be treated as experimental beta software

I am sharing it mainly because the concept feels useful: a more real-time, streaming voice assistant flow makes the device feel much more responsive and conversational.

Feedback, pointers to existing projects, warnings, and ideas are very welcome.

4 Likes
2

Nice! The response/latency is pretty quick and I love the natural way of responding.
Great concept! I hope someone will take this a bit further…

3

I love this idea! Unfortunately, I wasn't able to compile the voice-pe yaml. There seem to be compatibility issues with the current Home Assistant version. I have already tried and adjusted a lot, including integrating your project locally. It seems to be related to the sensor component (Component not found: sensor. - platform: rotary_encoder).

I think I have pretty much maxed out the usual voice-pe pipeline (STT-Gemini-STT) by using Elevenlabs, achieving a latency of about 1-2 seconds and continuous speech through the question mark trick. But live-speaking would be next level.

Could you give me a tip on how to fix this? I really want to try out your solution.

1 Like
4

Thank you for pointing that out, I have tried on clean env and in fact it was not building. I have pushed the fixes to the repo. Keep in mind that I have custom wakeup word there by defualt, you need to change it. It is more or less described in the README. For now it is "Jeffrey" but this is still under development I need more samples to train that.