Chapter 9: AI Inside Home Assistant
We have the pieces.
Now let’s put them into Home Assistant.
The main surfaces we care about are:
Assist
Assist Pipeline
Conversation Agents
LLM API
AI Task
MCP
Everything we have discussed so far lands somewhere in that stack.
Home Assistant Already Has the House
Before AI enters the picture, Home Assistant already knows about:
That is the substrate.
The AI layer gets access to useful pieces of it through the surfaces Home Assistant exposes.
Assist
Assist is the front door.
If Prefer handling commands locally is enabled, Home Assistant gets first crack at the request.
If it finds a local intent match, it handles the command there and stops.
If it does not, the request falls through to the configured Conversation Agent.
USER
│
▼
ASSIST
│
▼
LOCAL INTENT MATCH?
│
├── YES ──► HOME ASSISTANT
│
└── NO
│
▼
CONVERSATION AGENT
│
▼
MODEL
That gives you a very useful split:
known HA command → local deterministic handling
everything else → Conversation Agent
The existing intent surface is still sitting underneath all of this, including current-state access such as HassGetState.
Exposed Entities
One of the most practical controls is what the assistant can actually see.
ALL HA ENTITIES
│
▼
EXPOSED ENTITIES
│
▼
ASSIST / AGENT
Good names, sensible areas, aliases, and a restrained exposure set pay off immediately.
A model trying to control a handful of useful household entities does not improve when you hand it every battery voltage, diagnostic sensor, firmware entity, and miscellaneous helper in the installation.
This is especially noticeable with smaller local models.
There is a good real-world example in this thread:
My Journey to a reliable and enjoyable locally hosted voice assistant
The author is running HA Voice PE with local inference and exposes only a small subset of the entities in the installation to the assistant.
That is the kind of boring configuration decision that often matters more than another benchmark.
Assist Pipeline
For voice, the Assist Pipeline connects the stages.
WAKE WORD
│
▼
SPEECH TO TEXT
│
▼
ASSIST
│
▼
TEXT TO SPEECH
Each stage can be supplied independently.
For example:
wake word local
STT local
conversation cloud
TTS local
or:
wake word local
STT local
conversation local
TTS local
One local STT option worth knowing about is:
It is optimized around speech useful to Home Assistant rather than trying to be a universal transcription engine.
That same local voice thread above is worth reading if you want to see what happens when somebody actually assembles these pieces and lives with them.
Conversation Agents
This is where the model plugs into the Assist path.
Current examples include:
The basic shape is:
ASSIST
│
▼
CONVERSATION AGENT
│
▼
MODEL
Once the request reaches the model-backed Conversation Agent, Home Assistant capabilities can be made available through the LLM API.
CONVERSATION AGENT
│
▼
MODEL
│
▼
LLM API
│
├── ASSIST API
└── OTHER TOOLS
The model can be local or remote.
Home Assistant still owns the interaction.
Provider and Function Are Separate Decisions
A provider can fill more than one role.
For example, the official OpenAI integration can participate in:
Conversation
AI Task
Speech-to-text
Text-to-speech
That does not mean one provider has to own the whole stack.
A perfectly reasonable setup might be:
local STT
OpenAI conversation
local TTS
or:
local conversation
OpenAI AI Task
Same idea with other providers.
Pick the provider for the job.
Ollama Gives You the Local Version of the Same Socket
The official Ollama integration connects Home Assistant to an external Ollama server.
HOME ASSISTANT
│
▼
OLLAMA INTEGRATION
│
▼
OLLAMA SERVER
│
▼
LOCAL MODEL
Home Assistant remains the application.
Ollama runs the model.
The model you choose determines how well conversation, context handling, and tool use actually work.
The HA documentation also recommends keeping the exposed entity set manageable for local models.
Again, boring choices matter.
A small model faced with the semantic equivalent of a Home Assistant junk drawer can get entertaining.
Conversation Agents Are Not Only for Voice
This is an easy capability to miss.
A configured Conversation Agent can also be invoked from Home Assistant itself.
One community thread boils this down nicely. Someone wanted to use Ollama from a script for a morning overview.
The answer was essentially: use conversation.process and point it at the configured Conversation Agent.
Conceptually:
AUTOMATION / SCRIPT
│
▼
conversation.process
│
▼
CONVERSATION AGENT
│
▼
MODEL
That immediately gives you useful patterns like:
morning briefing
house summary
bounded question from an automation
generated notification text
The microphone is only one caller.
You Can Have More Than One Conversation Agent
Nothing says one model configuration has to handle every request.
There is a nice example here:
The solution ended up using two Ollama conversation entities against the same underlying model, configured differently for different jobs.
NORMAL VOICE
│
▼
FAST AGENT
MORNING BRIEF
│
▼
THINKING AGENT
No giant routing fabric required.
Sometimes two configured entities are enough.
LLM API
The LLM API is where the capability surface presented to the model comes together.
MODEL
│
▼
LLM API
│
├── ASSIST API
├── INTEGRATION TOOL
├── INTEGRATION TOOL
└── INTEGRATION TOOL
The built-in Assist API supplies the normal Home Assistant capability surface.
Other integrations can contribute more.
This is where the structures from the previous chapters become actual Home Assistant capability.
For example:
MODEL
│
▼
get_room_detail
│
▼
CUSTOM HA TOOL
│
▼
HOME ASSISTANT DATA
or:
MODEL
│
▼
search_household_index
│
▼
INDEX
Inventory, maintenance, documents, recipes, media, tasks, household knowledge, whatever your system actually needs.
AI Task
AI Task is the other major AI entry point.
AUTOMATION
│
▼
AI TASK
│
▼
MODEL
│
▼
RESULT
│
▼
AUTOMATION CONTINUES
This is Home Assistant asking AI to do a job.
No conversation required.
A simple example:
CAMERA IMAGE
│
▼
AI TASK
│
▼
PACKAGE PRESENT?
│
▼
RESULT
│
▼
HA AUTOMATION
Or:
HOME STATE
│
▼
AI TASK
│
▼
SUMMARY
│
▼
NOTIFICATION
Community Example: Camera Analysis
The locally hosted voice thread mentioned earlier goes beyond voice.
The author also combines Frigate camera imagery with AI so Home Assistant can answer useful questions about what is happening outside.
My Journey to a reliable and enjoyable locally hosted voice assistant
That gives us:
FRIGATE / CAMERA
│
▼
AI ANALYSIS
│
▼
HOME ASSISTANT
│
▼
ANSWER / AUTOMATION
Now the assistant can work with information Home Assistant could not derive from a simple binary sensor.
Community Example: AI Sky Observer
Another neat example is:
AI Sky Observer: Local AI Weather Observation for Home Assistant
It feeds outdoor camera imagery through local vision inference and publishes the observations back into Home Assistant as sensor entities.
CAMERA
│
▼
VISION MODEL
│
▼
HA SENSOR
│
▼
AUTOMATION
I really like this pattern.
The fuzzy observation happens once.
The result becomes ordinary Home Assistant state.
From that point dashboards, automations, templates, history, and everything else can use it normally.
That is a very HA way to use AI.
Structured AI Task Results
AI Tasks can also return structured data.
That lets you produce something shaped like:
package_present: true
vehicle_count: 2
door_blocked: false
and continue through ordinary Home Assistant logic.
FUZZY INPUT
│
▼
AI TASK
│
▼
STRUCTURED RESULT
│
▼
AUTOMATION
That is one of the cleaner AI patterns Home Assistant now offers.
A More Developed AI Task Pattern
You can take this farther than one camera image or one summary request.
This is one of the places where ZenOS uses the same Home Assistant primitives a little differently.
Friday does not get the firehose of Home Assistant events dumped into her context.
ZenOS has a summarization pipeline that turns those events into progressively smaller, more useful information.
Conceptually:
HOME ASSISTANT EVENTS
│
▼
KATAS
│
▼
LOCAL AI TASK
│
▼
CONSOLIDATE / PRUNE / WEIGHT
│
▼
ZEN SUMMARY
│
▼
FRIDAY'S LIVE CONTEXT
The first stage captures useful events into small structured packets.
A second stage merges recurring events, removes noise, elevates important signals, and produces the summary Friday actually needs.
So instead of loading something like:
motion on
motion off
temperature changed
motion on
door opened
motion off
power changed
temperature changed
...
the main agent can get something closer to:
Guest room has been occupied for 40 minutes.
Temperature is rising unusually quickly.
HVAC is running normally.
No action required yet.
The AI Task is not Friday.
It is doing background cognitive work for Home Assistant.
The result later becomes context for the main agent.
lots of HA signal
│
▼
background AI work
│
▼
small useful summary
│
▼
main agent
That means the model answering a voice request does not need to rediscover the last hour of the house every time someone says “Friday.”
If you want to see how that implementation works:
It is farther down the rabbit hole than “is there a package at the door?”, but it is still built from the same Home Assistant pieces.
Conversation Agent or AI Task?
If somebody is interacting with it:
USER
│
▼
CONVERSATION AGENT
If Home Assistant is asking AI to perform a job:
AUTOMATION
│
▼
AI TASK
You can use the same provider for both.
You do not have to.
Different Jobs Can Use Different AI
A Home Assistant setup can naturally look like:
VOICE CONVERSATION
│
▼
FAST MODEL
CAMERA AI TASK
│
▼
VISION MODEL
BACKGROUND SUMMARY
│
▼
LOCAL AI TASK
COMPLEX USER REQUEST
│
▼
LARGER MODEL
No reason every workload has to visit the same model.
Home Assistant often already knows what kind of job it is invoking, so a lot of routing can be explicit.
Sometimes selecting a different configured Conversation Agent or AI Task entity is enough.
MCP in Home Assistant
Home Assistant’s Model Context Protocol integration lets a Conversation Agent use capabilities from an external MCP server.
EXTERNAL MCP SERVER
│
▼
HOME ASSISTANT
│
▼
CONVERSATION AGENT
Home Assistant can also expose its own capabilities to an external MCP-capable agent.
EXTERNAL AGENT
│
▼
HA MCP SERVER
│
▼
HOME ASSISTANT
Community Example: Claude Code and HA MCP
There is already active community discussion around using Home Assistant’s MCP server with developer harnesses such as Claude Code.
Claude Code for Home Assistant
That can give the development harness a Home Assistant-facing capability surface while filesystem access, Git, shell, and other developer functions remain separate.
CLAUDE CODE
│
├── FILESYSTEM
├── GIT
├── SHELL
│
└── MCP
│
▼
HOME ASSISTANT
That is a useful real-world example of the HA MCP socket without pretending the external developer harness and Home Assistant are the same thing.
Three Main AI Paths
At this point most Home Assistant AI usage lands in one of three places.
Somebody is talking to it
USER
│
▼
ASSIST
│
▼
CONVERSATION AGENT
Home Assistant needs AI to do a job
AUTOMATION
│
▼
AI TASK
Another agent needs Home Assistant
EXTERNAL AGENT
│
▼
HOME ASSISTANT
When somebody says:
“I want AI in Home Assistant.”
a useful next question is:
Where?
A Simple Local Stack
VOICE / TEXT
│
▼
ASSIST
│
▼
CONVERSATION AGENT
│
▼
LOCAL INFERENCE
│
▼
LLM API
│
▼
HOME ASSISTANT
Add local STT and TTS if you want the entire voice path local.
A Simple Cloud Stack
VOICE / TEXT
│
▼
ASSIST
│
▼
CLOUD CONVERSATION AGENT
│
▼
LLM API
│
▼
HOME ASSISTANT
Same Home Assistant side.
Different inference endpoint.
A More Developed Stack
Once several pieces are in play:
HOME ASSISTANT
│
┌───────────────┼───────────────┐
▼ ▼ ▼
ASSIST AI TASKS AUTOMATIONS
│ │
▼ ▼
CONVERSATION MODEL TASK MODELS
│
▼
LLM API
│
┌────┼────────────┐
▼ ▼ ▼
ASSIST CUSTOM MCP
API TOOLS TOOLS
That is already a very capable home AI stack.
You may only need a few pieces of it.
Good.
Community Integrations Fill the Gaps
Home Assistant core is not going to cover every model server, provider, retrieval backend, or household capability somebody wants.
Community integrations can extend the useful surfaces with:
Conversation Agents
AI Task providers
local inference support
retrieval
memory
LLM API tools
When looking at one, figure out what it actually adds to Home Assistant.
Does it provide a Conversation Agent?
An AI Task entity?
More LLM tools?
Retrieval?
A local inference path?
That tells you where it belongs in the system.
Put It Together
The Home Assistant AI scaffold now looks something like:
HOME ASSISTANT
│
├── HOUSE STATE
│ ├── areas
│ ├── devices
│ ├── entities
│ ├── states
│ ├── events
│ └── history
│
├── ASSIST
│ ├── intents
│ ├── Conversation Agents
│ └── voice pipeline
│
├── LLM API
│ ├── Assist API
│ └── contributed tools
│
├── AI TASK
│ └── automation AI
│
└── MCP
├── external capabilities in
└── HA capabilities out
From there you choose the pieces the workload needs.
For ordinary voice:
Assist
+
good naming
+
areas
+
intentional entity exposure
For conversational AI:
Assist
+
Conversation Agent
For AI inside automations:
AI Task
For more agent capability:
LLM API extension
or
MCP
For an external agent using Home Assistant:
HA MCP Server
That gets you a long way before anything needs to become exotic.
And Now We Need Some Walls
We have finally connected the whole thing.
The model can receive useful household context.
It can act through Home Assistant.
AI can participate directly in automations.
External capabilities can join the agent.
External agents can reach back into Home Assistant.
Which means all of those nice arrows now cross trust boundaries.
So the last chapter is where we deal with:
identity
authentication
authorization
secrets
prompt injection
tool misuse
data exposure
privacy
logging
failure modes
fallbacks
recovery
least privilege
Most importantly:
What should still work when the AI is wrong, unavailable, or doing something you did not expect?
Next: