Chapter 3: Harnesses, Agents, and Where the Walls Actually Are
In Chapter 2 we separated the model from the machinery surrounding it.
Now we need to talk about that machinery.
Because this:
“I gave Claude access to Home Assistant.”
can describe several systems with completely different capabilities, security boundaries, and failure modes.
Did you connect a model to Home Assistant through Assist?
Did you install a custom conversation integration with additional tools?
Did you point Claude Code or OpenCode at your Home Assistant configuration directory?
Did you package one of those harnesses as a Home Assistant App, the artist formerly known as an add-on?
Did you install a tiny Home Assistant App that really acts as a doorway to an agent running somewhere else?
Did you give an external agent credentials and let it reach into Home Assistant from another machine?
All of those could reasonably be described as:
“AI connected to Home Assistant.”
They are very different architectures.
The thing that makes the difference is the harness.
The Model Is Still Just the Model
Take a capable language model and ask:
“Turn off everything downstairs except the lamp beside the couch.”
The model may understand the request perfectly.
It still cannot do anything.
It does not inherently know:
-
what entities exist
-
which areas are downstairs
-
which lamp you mean
-
whether you are allowed to control it
-
how Home Assistant controls the device
It can produce a lovely sentence saying:
“Sure, I’ll take care of that.”
And absolutely nothing may happen.
The model needs something around it that connects its reasoning to the environment.
That surrounding system is what we are going to call the harness.
So What Is a Harness?
A harness gives the model a working environment.
Depending on the system, it may provide:
-
instructions
-
context
-
tools
-
access to files
-
conversation state
-
memory
-
execution
-
permissions
-
approvals
-
retries
-
model selection
-
access to outside services
The simple interaction:
question → model → answer
can therefore become:
goal → model → action → result → model → next action
The model reasons.
The harness supplies the things the model can reason about and the actions it can request.
This is where AI starts moving from chatbot toward agent.
In Home Assistant, Assist Is a Harness
For Home Assistant users, there is a convenient way to understand this.
Assist, together with the conversation integration and LLM API behind it, functions as an AI harness.
Home Assistant receives the request.
It assembles information for the model.
It provides instructions.
It exposes approved entities and capabilities.
It describes tools the model can request.
The model decides what it wants to do.
Home Assistant performs the approved operation.
The result can then return to the model.
Conceptually:
HUMAN
│
▼
ASSIST / CONVERSATION INTEGRATION
│
├── instructions
├── HA context
├── exposed entities
├── tools / intents
└── conversation state
│
▼
MODEL
│
▼
TOOL REQUEST
│
▼
HOME ASSISTANT
The important part is that the model is not simply dropped into Home Assistant and handed the keys.
The harness gives it a shaped interface.
That shape determines what the model can see and what it can ask Home Assistant to do.
And that immediately leads to one of the biggest ideas in this chapter:
The harness helps define the security boundary.
Meet the Wall
Suppose I tell Assist:
“Turn off the kitchen lights.”
The model may decide that a kitchen light needs to be controlled.
Home Assistant has intentionally exposed a way for that request to be made.
Now suppose I tell the same model:
“Rewrite my Home Assistant authentication configuration.”
That is a very different request.
A normal Assist-style harness does not inherently give the model general filesystem access, arbitrary shell execution, or unrestricted administrative control over the Home Assistant installation.
If those capabilities are not exposed, the model cannot use them through that path.
The model can be extremely intelligent.
It still cannot walk through a door the harness did not give it.
Conceptually:
MODEL
│
▼
HA HARNESS
│
┌───────┴────────┐
│ │
ALLOWED NOT EXPOSED
│ │
▼ X
lights, climate arbitrary host
media, intents administration
That wall is useful.
It separates:
what the model may understand
from:
what the system allows the model to do
Those are very different questions.
Now Point Claude Code or OpenCode at /config
Here is where things get interesting.
Instead of using Assist, suppose you run Claude Code, OpenCode, Codex, or another developer-oriented harness and point it at your Home Assistant configuration directory.
Now the working environment changes.
A coding harness may provide capabilities such as:
-
reading files
-
writing files
-
searching files
-
editing configuration
-
running shell commands
-
executing tests
-
using version control
-
calling external tools
Our architecture now looks more like:
MODEL
│
▼
CLAUDE CODE / OPENCODE / SIMILAR
│
├── read files
├── edit files
├── search files
├── run commands
└── other harness tools
│
▼
/config
At first glance this may look like:
“Assist, but more capable.”
That framing misses the important part.
This is a different security model.
Assist primarily operates through capabilities intentionally exposed by Home Assistant.
A developer harness pointed at /config operates through filesystem permissions, shell permissions, API credentials, and whatever other access its runtime has been granted.
The wall moved.
Same Home Assistant, Different Door
Compare these two systems.
Assist
MODEL
│
▼
ASSIST
│
▼
LLM TOOLS / INTENTS
│
▼
EXPOSED HA CAPABILITIES
Home Assistant defines the interface.
The model operates through that interface.
Developer Harness
MODEL
│
▼
CLAUDE CODE / OPENCODE
│
▼
FILESYSTEM / SHELL
│
▼
HOME ASSISTANT CONFIGURATION
The operating environment defines much more of the interface.
If the process can read a file, the harness may be able to read it.
If the process can modify a file, the harness may be able to modify it.
If the process can execute a command, the harness may be able to execute it, subject to whatever controls the harness adds.
That can be extraordinarily useful.
It can also be extraordinarily powerful.
What Can That Mean in Practice?
Imagine you ask:
“Why does this automation keep turning the bedroom lights off?”
An Assist-based agent may be able to inspect relevant state and reason using the Home Assistant tools it has been given.
A developer harness with access to /config might instead be able to:
-
Search the YAML.
-
Find the automation definition.
-
Read included files.
-
Inspect related templates.
-
Modify the automation.
-
Run a configuration check.
-
Compare the result.
-
Reload or restart something if its permissions allow it.
That is an enormous increase in capability.
It is also an enormous increase in authority.
The model did not suddenly become smarter.
We changed the harness.
More specifically:
We moved the wall.
What If the Harness Lives Inside Home Assistant?
Now we get another architecture.
Home Assistant Apps, formerly called add-ons, let you run specialized containers alongside Home Assistant.
That means someone can package Claude Code, OpenCode, or another agent harness as a Home Assistant App and give it access to selected Home Assistant resources.
From the user’s perspective, this can look very integrated:
HOME ASSISTANT
│
├── Assist
│
└── Agent App
But those two paths may have completely different privilege surfaces.
Assist might receive:
exposed entities
approved tools
conversation context
Home Assistant intents
The agent App might receive some combination of:
/config mounted read/write
Home Assistant API access
network access
shell execution
other mounted storage
additional runtime privileges
So the architecture is closer to:
HOME ASSISTANT ENVIRONMENT
│
├──── ASSIST HARNESS
│ │
│ └── approved HA capabilities
│
└──── AGENT APP
│
├── mounted filesystem
├── HA API
├── network
└── agent tools
Both may appear to be “inside Home Assistant.”
Their security boundaries can be completely different.
That distinction matters a lot.
Packaging is not permissioning.
Putting a developer agent inside a Home Assistant App does not automatically make it obey the same boundaries as Assist.
If the container can write /config, then the filesystem itself may become part of the agent’s control surface.
If it holds a powerful API credential, that credential becomes part of the control surface.
If it has broad network access, that becomes part of the control surface too.
Where the agent runs tells you surprisingly little.
What it can reach tells you much more.
The Tiny Box May Not Be the Agent
There is another pattern showing up increasingly often.
You install something tiny.
It uses almost no memory.
It needs no GPU.
The instructions say:
“You don’t need powerful local AI hardware to run this.”
That may be completely true.
But before concluding that somebody has somehow squeezed a capable general-purpose agent into a tiny container, stop and ask:
What is actually running here?
Sometimes the local component is primarily:
-
a user interface
-
an API bridge
-
a proxy
-
a message relay
-
an MCP client or server
-
an authentication layer
-
a control surface
The actual harness, model, memory, orchestration, or all of the above may be running somewhere else.
Conceptually:
HOME ASSISTANT
│
▼
SMALL LOCAL APP
│
│ network request
▼
REMOTE AGENT SERVICE
│
├── harness
├── model
├── memory
├── tools
└── orchestration
There is nothing inherently wrong with this.
It can be a perfectly reasonable architecture.
Cloud AI works through the same broad idea.
The important issue is understanding where the system actually lives.
Small Local Footprint Does Not Mean Local Intelligence
This gives us another useful rule:
Small local footprint does not imply local intelligence.
Nor does:
installed in Home Assistant
necessarily imply:
controlled by Home Assistant
If the local App is mostly a doorway to a remote agent, the interesting security questions move upstream.
Ask:
-
Where is the harness actually running?
-
Where does inference happen?
-
What information leaves my network?
-
What credentials does the local component hold?
-
What information gets sent upstream?
-
What tools can the remote agent request?
-
Which side actually enforces permissions?
-
Can the remote service change behavior without the local App changing?
-
Is the remote side open to inspection?
-
What happens if the remote service is compromised?
Those questions do not imply that the architecture is bad.
They tell you what architecture you actually installed.
That is the goal.
A Note About Tiny Local Models
Before somebody reaches for the reply button:
Yes.
There really are useful AI models designed to run on very small hardware.
A Raspberry Pi, small accelerator, embedded computer, or even smaller device can perform remarkably useful machine-learning workloads.
Examples include:
-
wake-word detection
-
object detection
-
classification
-
constrained speech recognition
-
embeddings
-
simple routing
-
narrow anomaly detection
These models often work precisely because they are designed for a specialized problem.
Home Assistant’s own local voice work provides a good example of the principle.
If the problem can be narrowed from:
“Understand arbitrary human language about anything.”
to:
“Recognize this constrained family of home-control phrases.”
the computational requirements change dramatically.
That is real AI.
It is useful AI.
It is also a different problem from running a capable general-purpose household reasoning agent.
A tiny specialist model may be excellent at:
“Did someone say the wake word?”
It does not follow that it can also efficiently handle:
“Look at everything happening in the house, determine why the guest room is uncomfortable, consider recent activity and weather, decide whether anything needs adjustment, then use several tools to deal with it.”
Different workload.
Different model requirements.
Different hardware requirements.
So when you see:
“This AI agent runs great on a Raspberry Pi.”
do not assume anything yet.
Find out whether the Pi is running:
-
a small specialist model
-
the full general-purpose model
-
only the harness
-
only the interface
-
a router
-
a proxy to another machine
-
a client for a cloud service
The statement may be completely accurate.
It simply does not tell you where the intelligence lives.
Find the Harness
At this point we have enough information to establish a useful diagnostic habit.
Whenever you encounter an AI system:
Find the harness.
Then ask:
Where is it running?
Then:
What can it reach?
Then:
Who controls the keys?
Those three questions will tell you an enormous amount.
Because this:
MODEL
│
▼
LOCAL APP
│
▼
REMOTE HARNESS
has a very different trust boundary from:
MODEL
│
▼
LOCAL HARNESS
│
▼
HA TOOLS
and both are different again from:
MODEL
│
▼
DEVELOPER HARNESS
│
▼
FILESYSTEM + SHELL
The UI may look almost identical.
The security architecture is not.
The Four Doors
We can now describe four common ways an AI agent might interact with Home Assistant.
Door 1: Native Assist Harness
MODEL
│
▼
ASSIST
│
▼
APPROVED HA TOOLS
│
▼
HOME
The model interacts through deliberately exposed Home Assistant capabilities.
Door 2: Agent App With Local Harness
MODEL
│
▼
AGENT HARNESS IN HA APP
│
├── mounted files
├── HA API
├── network
└── app permissions
│
▼
HOME ASSISTANT SYSTEM
The container configuration, mounted resources, credentials, and harness policy help define the boundary.
Door 3: External Agent
MODEL
│
▼
EXTERNAL HARNESS
│
├── filesystem
├── shell
├── HA API
└── external tools
│
▼
HOME ASSISTANT
The runtime environment, credentials, network access, and host permissions define much of the boundary.
Door 4: Local Shell, Remote Agent
HOME ASSISTANT
│
▼
LOCAL APP / BRIDGE
│
▼
NETWORK
│
▼
REMOTE HARNESS
│
MODEL
│
TOOLS
The thing installed locally is primarily the doorway.
The real agent environment lives elsewhere.
All four can be legitimate.
They are not interchangeable.
Harness Permissions and Host Permissions Are Different Layers
This distinction becomes critical with developer harnesses, Apps, and remote agent bridges.
A harness might say:
“Ask before running a shell command.”
That is a harness policy.
A container might say:
“This directory is mounted read-only.”
That is a runtime boundary.
Linux might say:
“This process can access these resources.”
That is a host permission.
Home Assistant might say:
“This credential may perform these API operations.”
That is an application permission.
A remote platform might additionally enforce:
“This account may use these remote tools.”
That is another policy layer.
Several can exist simultaneously.
MODEL
│
▼
HARNESS POLICY
│
▼
CONTAINER / RUNTIME BOUNDARY
│
▼
HOST / OS PERMISSIONS
│
▼
APPLICATION PERMISSIONS
│
▼
RESOURCE
The important question is:
Where is the restriction actually enforced?
A line in a system prompt saying:
“Please do not edit secrets.”
is not equivalent to making the secrets file inaccessible.
One is an instruction.
The other is a control.
That distinction will return when we get to security.
Repeatedly.
Probably with flashing lights.
A More Capable Home Assistant Integration Can Move the Wall Too
The choice is not limited to:
Assist
or:
give Claude shell access and pray.
A custom Home Assistant integration can create a much richer harness while still exposing deliberately shaped tools.
It might provide capabilities such as:
-
inspect an automation
-
validate configuration
-
retrieve household knowledge
-
search documents
-
query maintenance records
-
inspect energy history
-
call external services
-
use memory
-
route between models
Instead of giving an agent:
“Here is
/config. Have fun.”
you might give it tools like:
get_automation
validate_automation
propose_automation_change
apply_approved_change
Those tools could accomplish a similar goal while exposing a much narrower capability surface.
Conceptually:
MODEL
│
▼
CAPABLE HA HARNESS
│
┌──────────┼──────────┐
│ │ │
HA TOOLS MEMORY EXTERNAL TOOLS
│ │ │
└──────────┼──────────┘
▼
CONTROLLED WORLD
That is harness design.
It is also security design.
Tools Give the Agent Hands
The model provides reasoning.
Tools provide capability.
A Home Assistant harness might expose:
-
query a sensor
-
turn on a light
-
adjust climate
-
control media
-
add an item to a list
A richer Home Assistant harness might add:
-
search documents
-
inspect configuration
-
access calendars
-
query databases
-
send messages
-
call outside services
A developer harness might provide:
-
read
-
edit
-
patch
-
shell
-
git
-
search
-
web access
The model generally does not perform these operations directly.
It asks the harness to perform them.
MODEL
│
│ "call this tool"
▼
HARNESS
│
│ permission check
▼
TOOL
│
▼
WORLD
That means the useful question is not simply:
“Can Claude do this?”
Ask:
Does the harness give Claude a way to do this?
Then ask:
What prevents that capability from reaching farther than intended?
Now we are asking useful architectural questions.
Same Model, Different Harness, Very Different System
Imagine the same underlying model in several environments.
System A: Chat
The model receives:
“The living room is 78°F.”
It can respond with text.
System B: Assist
It receives Home Assistant context and approved climate tools.
It can inspect state and request an adjustment.
System C: Expanded Home Assistant Harness
It receives richer context, memory, additional tools, and policy controls.
It can reason across more of the household environment.
System D: Developer Harness
It can inspect configuration files, edit them, and perhaps execute Home Assistant-related commands.
System E: Remote Harness
The local Home Assistant component gathers information and sends the job to an agent environment running somewhere else.
The model may even be identical across all five.
The surrounding system is not.
The harness defines the working world.
This Is Where We Start Calling It an Agent
Unfortunately, we have reached the word agent.
People use “agent” to describe everything from a chatbot with one tool to a persistent system controlling other agents.
For this primer, we will continue using the practical definition from Chapter 2:
An agent is an AI system that can pursue a goal by selecting actions and using available tools or capabilities.
An agent might receive:
“Turn off everything downstairs.”
It interprets the goal, identifies relevant devices, and requests actions.
A developer agent might receive:
“Find the bug causing my automation to fail and fix it.”
It can inspect files, run commands, make changes, observe the result, and continue working.
The basic loop is the same.
GOAL
│
▼
MODEL
│
▼
SELECT ACTION
│
▼
HARNESS EXECUTES
│
▼
RESULT
│
└────────────► MODEL
Observe.
Decide.
Act.
Observe again.
Agent Does Not Mean Autonomous
This distinction is worth protecting early.
An agent does not necessarily roam freely doing whatever occurs to it.
An agent may:
-
operate only when asked
-
have read-only tools
-
require approval for actions
-
control only selected entities
-
have no shell access
-
run for a single task
-
stop at clearly defined boundaries
Agency describes the ability to choose actions toward a goal.
Authority describes what actions the surrounding system permits.
Those are separate properties.
An agent can therefore be highly capable at reasoning while having very little authority.
That is often exactly what we want.
Coding Agents Make Harnesses Easy to See
Claude Code, Codex, and OpenCode are useful examples because the environment around the model is obvious.
To perform useful coding work, the harness may need:
-
files
-
shell access
-
source control
-
search
-
tests
-
project instructions
-
documentation
-
external tools
-
subagents
That harness is doing a huge amount of the work that makes the model useful.
Point that same style of harness at Home Assistant configuration and you have effectively given the model a developer-shaped interface to the system.
Use Assist and you have given the model an occupant-shaped interface to the system.
That is perhaps the simplest way to understand the distinction.
Assist lets the AI use the house.
A developer harness may let the AI work on the house.
Both are useful.
They should not automatically have the same keys.
Instructions Are Part of the Harness Too
Tools are only one part of the environment.
The harness also provides instructions.
Those might include:
-
system prompts
-
household rules
-
project conventions
-
tool descriptions
-
entity descriptions
-
safety guidance
-
user preferences
-
workflows
A developer harness might have instructions saying:
Run validation before applying a change.
A Home Assistant harness might say:
Never unlock an exterior door without confirmation.
These instructions can strongly affect behavior.
They should still be distinguished from enforcement.
If the agent does not possess an unlock tool, it cannot unlock the door through that path.
If it possesses the tool but the prompt simply asks it not to use it, we are relying on model behavior.
Those are different controls.
Context Is Also Part of the Harness
The harness determines what the model gets to know.
For Home Assistant that might include:
-
exposed entity state
-
areas
-
device names
-
conversation history
-
tool definitions
-
sensor data
-
retrieved household information
A basic harness might send:
sensor.living_room_temperature: 78
binary_sensor.living_room_occupancy: on
media_player.living_room_tv: playing
light.living_room: on
A more sophisticated harness might provide:
“Living room is occupied. Television is active. Temperature is 78°F. Lighting is currently configured for evening television viewing.”
Those prompts describe roughly the same environment.
They give the model very different material to reason over.
This is where harness design starts blending into context engineering.
A strong harness does not merely make information available.
It helps put useful information into a useful shape.
Some Harnesses Add Memory
A conversation model can only reason over information available in its current context.
A harness can add longer-lived memory.
That might include:
-
prior conversations
-
household preferences
-
previous decisions
-
task history
-
summaries
-
documents
-
structured records
Now useful information from previous interactions can be brought back when needed.
The model did not acquire permanent memory.
The harness gave it access to one.
Same model.
Different machine.
Some Harnesses Add Subagents
Once one agent becomes useful, somebody inevitably asks:
“What if my agent had agents?”
Welcome to subagents.
A primary agent can delegate narrower work to another agent.
For example:
Primary agent: Determine why energy use is unusually high.
It might ask:
Energy agent: Examine current and historical consumption.
HVAC agent: Inspect climate systems.
Device agent: Look for unusual loads.
Those agents might have different:
-
prompts
-
models
-
tools
-
permissions
-
context
Now we have another architectural choice:
one agent using many tools
or:
multiple specialized agents
We will come back to that when we discuss routing and orchestration.
Some Agents Hang Around
A typical conversation has an obvious interaction cycle.
You ask something.
The agent responds.
Eventually the interaction ends.
Other systems are designed to remain available.
They may:
-
maintain sessions
-
receive events
-
run scheduled work
-
listen on communication channels
-
maintain state
-
invoke tools over time
-
coordinate other agents
That creates a persistent agent runtime.
The difference is roughly:
“I opened an AI application.”
versus:
“There is an AI agent service running.”
That becomes particularly interesting in home automation.
Home Assistant Is Already Persistent
The home does not stop because nobody is chatting with Assist.
At 2:17 AM:
-
a door opens
-
the freezer warms up
-
a leak sensor trips
-
somebody arrives home
-
an automation fails
Home Assistant already lives in a persistent, event-driven environment.
Traditional automations are very good at responding to those events.
If AI becomes a deeper participant in the home, its harness may eventually need to participate on similar timescales.
That does not automatically mean autonomous AI control.
It means an agent architecture may need to support more than a single conversation turn.
Harness Versus Persistent Runtime
This gives us another useful distinction.
Harness
Provides a model with instructions, context, tools, permissions, and a working loop.
Persistent runtime
Keeps one or more agents available across time and events.
You can have a harness without a persistent runtime.
You can also place a harness inside one.
PERSISTENT RUNTIME
│
▼
HARNESS
│
┌────┴────┐
│ │
MODEL TOOLS
Home Assistant already supplies much of the persistence if the agent is built into Home Assistant.
A Home Assistant App may supply another persistent runtime alongside it.
An external platform may provide its own persistence and connect back into Home Assistant.
A tiny local App may simply provide connectivity to a persistent runtime somewhere else.
Those differences matter because they change where the walls are.
So What Is the Agent, Exactly?
Depending on the product, “agent” may mean:
-
a configured model
-
a conversation agent
-
a worker inside a harness
-
a persistent service
-
a complete application
-
one member of a multi-agent system
The terminology is not consistent enough to tell us much by itself.
So when somebody says:
“I connected an agent to Home Assistant.”
ask:
What model does it use?
What harness surrounds it?
Where is that harness actually running?
What context does it receive?
What tools can it call?
Which identity does it operate as?
What can that identity access?
What data leaves the local machine?
Where are permissions enforced?
Does it have access through Home Assistant, through the container, through the operating system, through a remote service, or some combination?
Those questions tell us what the agent can actually do.
This Is the Big Idea
We spend a lot of time asking:
“Which model should I use?”
That remains an important question.
Now we have several more:
Where is the harness?
What should the model be allowed to touch?
What enforces that boundary?
Where does the information go?
A well-designed harness can give a model exactly enough capability to perform useful work.
A poorly bounded harness can accidentally give the same model far more authority than the job requires.
A remote harness may also make some of those boundaries harder to inspect.
That does not make local good and remote bad.
It means you should know what you installed.
The Stack So Far
Our Home Assistant mental model now looks something like this:
HUMAN / EVENT
│
▼
AI HARNESS
│
┌──────────────┼──────────────┐
│ │ │
INSTRUCTIONS CONTEXT CAPABILITIES
│ │ │
└──────────────┼──────────────┘
│
▼
MODEL
│
▼
TOOL REQUEST
│
▼
SECURITY BOUNDARY
│
▼
ALLOWED ACTION
But the harness may be:
inside Assist
inside an HA App
on another local machine
in a cloud service
Change the harness and we may change:
-
what the model knows
-
what tools it sees
-
what actions it can request
-
what permissions apply
-
where those permissions are enforced
-
where information is processed
-
how much of the system it can reach
Same model.
Different machine.
The Useful Question Has Changed
When you encounter a new AI product, integration, or agent, asking:
“What model does it use?”
is still useful.
Now add:
What is wrapped around that model?
Where is the harness?
Where is the model?
Does anything leave my network?
What context does it receive?
What tools does it have?
Does it have filesystem access?
Does it have shell access?
Can it control Home Assistant entities?
Can it modify Home Assistant itself?
Is it running inside an App?
What directories are mounted?
What credentials does it possess?
Where are the actual permission checks?
Can the remote side change independently of the component I installed?
What happens if the model asks for something outside its intended job?
Those questions reveal the architecture.
And once we understand the harness, we are ready for the next question.
The model still has to run somewhere.
Maybe somebody else runs it in a cloud data center.
Maybe you run it on a GPU under your desk.
Maybe the tiny thing you installed is talking to a monster sitting three feet away.
Maybe it is talking to a monster three thousand miles away.
Maybe you eventually have several machines humming in a closet and have begun casually using the phrase “inference farm.”
So next:
Where does the thinking actually happen?