I have written a custom integration that allows you to use Gemini in "Live" mode, that is it takes in audio in and returns audio, skipping the need for speech-to-text and text-to-speech.
Using this mode reduces the latency on responses. You also gain multi-lingual support and more natural sounding voice.
This integration plugs directly in to the Assist pipeline, it should work with any normal voice assistant hardware and works just fine in the browser debug mode too.
Apart from installing the custom integration, there is no special hardware or software required to use it.
Give it a go and let me know what you think. You should have the official Gemini integration setup first before trying this one out.
great integration, easy to install, this is exactly what I've been waiting for. Thank you so much for providing this!
It is truly very much faster than using the standard pipeline. Previously, I had to use ElevenLabs to get a similar latency. That doesn't seem necessary anymore. The current Gemini voices are not that bad either.
However, I haven't quite figured out how "Continuous Speak" works yet.
With a HA Voice PE, the voice input should basically remain active after the TTS output, right? Sometimes it stays inactive for me. I haven't figured out why yet, but I'm still brand new to this.
I have done some modifications recently to address this. I added a new callback that Gemini can call to end the conversation. So if you say something like “goodbye” or “that is all” to it then it will know to finish the conversation and has a mechanism for doing so.
I have also added a mechanism so that Gemini can show you the answer by writing out text. To use this simply ask Gemini to “show you”, and it will know to call the text display callback.
You will need the latest version installed to see these.