Hi everyone,
if you use the Home Assistant voice assistant and wish it felt as smooth and natural as ChatGPT Voice Mode, I'd like to invite you to test my new project: Pipecat Assist for Home Assistant.
I was missing a generic framework for modern voice assistants in HA. I didn't want to install a separate integration for every cloud, another one for STT, another for TTS, another for web search, another for MCP, and so on. I wanted one place where I could build a real realtime voice assistant, combine different model providers, and still control my home through Home Assistant's native mechanisms.
That led me to Pipecat, a framework for realtime voice and multimodal AI. I adapted it into Home Assistant as an add-on plus a custom component.
Repository:
https://github.com/kyvaith/pipecat-homeassistant
What it offers:
-
realtime speech-to-speech assistants, such as Gemini Live, OpenAI Realtime, and AWS Nova Sonic,
-
custom voice pipelines built from different cloud services: STT + LLM + TTS,
-
integrations for Gemini, OpenAI, Soniox, Deepgram, Speechmatics, Cartesia, Gradium, ElevenLabs, Google Cloud TTS, AWS Bedrock, Ollama, and OpenAI-compatible endpoints,
-
support for local AI tools,
-
multiple MCP servers, including the native Home Assistant MCP server,
-
web search as a pipeline step,
-
session memory,
-
audio input/output debugging,
-
a pipeline editor and visual Pipecat Flow editor,
-
a dedicated Lovelace WebRTC card with live transcript and audio animation,
-
HA Assist integration as Conversation / STT / TTS,
-
AI Tasks / AI Actions support where available in your HA version.
The main idea is simple: Pipecat Assist is not just another wrapper for one AI provider. It is a layer for building your own assistant from reusable blocks.
For example, you can run a simple pipeline like:
Gemini Live -> Home Assistant MCP
or a composed one like:
Speechmatics STT -> OpenAI Cloud LLM -> ElevenLabs TTS -> Home Assistant MCP
or extend it with:
Session Memory -> Web Search -> Pipecat Flow -> multiple MCP servers
Gemini Live is the default starting point because it works very well in practice and is easy to set up. Generate an API key in Google AI Studio, paste it into the Google Gemini Live integration, and you can start talking.
There are two main ways to use it:
-
Lovelace card
This is the most responsive path. It uses WebRTC, includes live transcription, smooth transcript scrolling, audio animation, and feels closest to a ChatGPT Voice-style experience. -
HA Assist
The custom component exposes Pipecat Assist as Conversation, Speech-to-text, and Text-to-speech, so you can select it in Home Assistant's Voice Assistant settings. I also built a dedicated bridge that lets HA Assist talk to speech-to-speech realtime assistants. That said, HA Assist itself is not a true full-duplex WebRTC client, so it cannot support natural barge-in while the assistant is speaking. The Lovelace card remains faster and more realtime.
Under the hood, there is quite a lot going on:
-
the add-on runs Pipecat and exposes a WebRTC endpoint,
-
the custom component auto-detects the add-on URL,
-
the integration registers Conversation / STT / TTS / AI Task entities,
-
Home Assistant MCP works through Supervisor, without manually copying tokens,
-
additional MCP servers can be connected,
-
pipelines are shared by the add-on test UI, Lovelace card, HA Assist, and future satellites,
-
Pipecat Flow allows conditional conversation graphs, for example asking follow-up questions, collecting parameters, and calling an HA service at the end.
This is still a project in active testing and iteration, but it is already usable. A lot of work has gone into it: the add-on UI, Lovelace card, HA Assist bridge, provider integrations, MCP support, flow editor, audio debugging, multi-arch image publishing, and more.
I'd really appreciate feedback from people who:
-
already use HA Assist,
-
have tested Gemini Live or OpenAI Realtime,
-
use voice satellites,
-
want to build more advanced voice flows,
-
run local models or OpenAI-compatible endpoints,
-
use MCP in automations.
Quick installation: -
Add the repository to the Home Assistant Add-on Store.
-
Add the repository to HACS as a custom integration.
-
Install the Pipecat Assist add-on.
-
Install the Pipecat Assist custom component.
-
Select Pipecat Assist in Voice Assistant settings for Conversation / STT / TTS.
-
Configure Gemini Live in the add-on, or build your own pipeline.
-
Add the Lovelace card and start talking.
The ESPHome custom componet (leveraging full duplex audio and websocket) is on its way, and I'll release it soon.
I'd love to hear your test results, bug reports, and ideas for where this ecosystem should go next.