I’m excited to share a project I’ve been working on lately: a custom Grok integration for Home Assistant that takes voice control to the next level. My goal was to combine the power of an LLM with the reliability of the native Home Assistant ecosystem, all while minimizing data traffic and operational costs.
The core idea is to drastically reduce the number of tokens sent to the LLM by offering an alternative interaction method to Home Assistant’s standard one. Grok doesn’t process every single request; instead, it acts as a smart router. It analyzes the user’s input, corrects any ASR (Automatic Speech Recognition) errors, and then makes a decision:
If the request is a simple conversational question (e.g., “tell me a joke”), Grok responds directly.
If it’s a smart home command (e.g., “turn on the living room lights”), Grok extracts only the essential command and sends it to Home Assistant for native processing.
This logic is implemented through a Pipeline.
The Pipeline
LLM Analysis (Grok): The user’s input is analyzed. If the request is a command, Grok doesn’t process the entire context; it just extracts the relevant part. For example, from “Could you please turn on the living room lights?” it extracts the concise command “turn on the living room lights.”
Command Tagging: The extracted command is then encapsulated in a special tag [[HA_LOCAL: {"text": "natural language command"}]]. This tag signals to the system that the request should be handled locally. This crucial step has allowed me to send far fewer tokens compared to a traditional integration, with a significant impact on cost and speed.
Native HA Processing: Home Assistant receives the concise command via the tag and handles it using the local conversation.home_assistant service. If for some reason the recognition fails, there’s a fallback option that allows for a second call using Home Assistant’s standard tools. This provides an additional chance for the command to be executed, increasing overall reliability. If you add Home Assistant’s native intent handling step (first locally, then to the LLM), the process becomes even more efficient.
In summary: This integration allows for a more fluid and natural conversation with an LLM, without needing to articulate words precisely to activate an intent.
I present to you my new project, xAI Conversation, much more elaborated than this one, which consists of two parts:
an agent for Assist conversation with peculiar features, such as conversation memory on the xAI server (meaning I only send the last message, not the whole conversation, as is the case with the Home Assistant standard), conversation continuity between the same user’s devices (switching from PC, tablet, to smartphone for the same user), and executing multiple actions in the same request;
b. an integrated code editor optimized for Home Assistant, in a card for your dashboard, for assistance, creation, and modification, based on xAI’s “Grok Code Fast 1” model.
I am interested in this. Are you going to make this HACS compatible? Do you have any installation and configuration instructions? I currently have the xAI Grok Conversation integration running, but it has significant limitations and would like to give this a try. Thanks.
I present to you my new project, xAI Conversation, much more elaborated than the previous one, which consists of two parts:
a. an assistant for Assist with peculiar features, such as conversation memory on the xAI server (meaning I only send the last message, not the whole conversation, as is the case with the Home Assistant standard), conversation continuity between the same user’s devices (switching from PC, tablet, to smartphone for the same user), and executing multiple actions in the same request;
b. an integrated code editor optimized for Home Assistant, in a card for your dashboard, for assistance, creation, and modification, based on xAI’s “Grok Code Fast 1” model.
just wanted to say i just found this and loaded things up (very simple process, download release, unzip, put unzipped custom_component content in custom_component folder, restart). I added it and loaded in my api key. I appreciate the drop down to pick from all the different models, it is all working very well and was super simple to setup.
Hi @driz, since you manually downloaded the version, I decided to share the latest version and its changelog with you, so you can decide whether to update your version.
Hi Sebastian, what to buy depends on many factors: your configuration, the environments where you’ll be implementing the voice service, etc. I’d suggest starting with your devices: smartphones, tablets, PCs, smartwatches. You’ll find a lot of information on how to configure them on this forum. But I promise I’ll let you know when I assemble my integrated system for communicating with LLMs.
photo analysis is mainly what i use it for, so i’ll definitely upgrade. However, if you are wanting feedback, i won’t be able to provide any until after 01DEC as Ill be on vacation.
thanks for your efforts on this!
edit: i don’t know how i plan to leverage this yet, but i am excited for xai_conversation.reload_pricing and xai_conversation.reset_token_stats
the new default of grok-2-vision-1212 may be out of my price range…
