Chapter 2: Okay, Which AI Thing Are We Talking About?
Now that we have established that “AI” is not one thing, we need to name some of the pieces.
This matters because the terminology around modern AI has become a bit of a mess.
People will say:
“I installed an AI.”
Okay.
What did you install?
A model?
An inference server?
A conversation agent?
A coding agent?
A tool harness?
A persistent agent runtime?
A router?
A retrieval system?
Some combination of those?
All of these things may involve AI. They occupy very different places in the system.
Worse, many products bundle several of them together, which makes the boundaries harder to see.
So before we start comparing products, architectures, or hardware, let’s build ourselves a vocabulary.
Start With the Model
At the center of most of the systems we are going to discuss is a model.
A model is the thing doing the inference.
For our purposes, you can think of inference as:
input goes in → model processes it → output comes out
The nature of that input and output depends on the model.
A large language model, or LLM, primarily works with language.
A vision-language model, or VLM, can reason about images as well as language.
A speech recognition model can turn audio into text.
A text-to-speech model can turn text into audio.
An embedding model can turn information into numerical representations that make certain kinds of search and comparison possible.
There are many other kinds of models, but these are some of the ones we are likely to encounter around Home Assistant.
Models also vary enormously in capability.
Some are tiny and specialized.
Some are enormous general-purpose models capable of reasoning across many kinds of problems.
Some accept multiple kinds of input.
Some can use tools.
Some are designed specifically for coding, speech, vision, classification, or embeddings.
The important thing to understand right now is this:
The model is a component.
It is usually not the whole AI system.
A Model Needs Somewhere to Run
When you send a request to a model, something has to load that model, feed it the request, perform the computation, and return the result.
That is the job of an inference service.
If you use a cloud AI provider, the inference infrastructure belongs to somebody else.
You send a request across an API.
Their infrastructure runs the model.
You receive the result.
Conceptually:
your application → provider API → model → result
If you run models yourself, software such as an inference server performs the same general job on your hardware.
Conceptually:
your application → local inference service → local model → result
We will spend much more time on inference later because this is where model size, memory, GPUs, quantization, context size, performance, and quite a lot of electricity enter the story.
For now, just remember:
Model = the thing doing the inference
Inference service = the thing running the model
Those are different things.
This is why saying:
“I run Ollama.”
does not yet tell us what model you are running.
Likewise:
“I use OpenAI.”
does not tell us which model, how it is configured, what application is calling it, or what that application allows it to do.
Then There Is the Application Around the Model
Suppose we take a capable language model and give it this prompt:
“Fix the bug in my application.”
The model cannot magically inspect your filesystem.
It cannot automatically open your source files.
It cannot run your test suite.
It cannot edit code unless something gives it the ability to do those things.
So we build software around the model.
That software may provide:
-
access to files
-
tools
-
a terminal
-
search
-
memory
-
project instructions
-
session state
-
approvals
-
retries
-
tool results
-
model selection
-
context management
-
orchestration
This surrounding software is often described as an agent harness.
You will also hear terms like agent framework, agent runtime, coding agent, agent environment, or simply agent.
The terminology is still settling.
The basic idea is more important than the label.
The harness gives the model somewhere to work and things it is allowed to do.
This is why a raw model and something like Claude Code or OpenCode are fundamentally different experiences.
Claude Code and OpenCode are applications built around models. They provide access to files, tools, execution environments, instructions, sessions, and other machinery that lets a model actually work on a software project.
The model supplies intelligence.
The harness supplies the working environment.
This Is Where People Start Calling Things Agents
Unfortunately, we have now reached the word agent.
“Agent” is currently used so broadly that two people can say it and mean completely different things.
For this primer, I am going to use a practical definition:
An agent is an AI system that can pursue a goal by selecting actions and using available tools or capabilities.
That is intentionally broad.
An agent might receive:
“Turn off everything downstairs.”
It determines which devices are relevant and invokes the necessary Home Assistant tools.
Another agent might receive:
“Figure out why this integration is failing and fix it.”
It reads logs, examines code, searches documentation, edits files, runs tests, and repeats that process until it reaches a result.
Both qualify as agents.
Their environments and authority are dramatically different.
That distinction will become extremely important once we start giving agents access to our homes.
Tool Use Changes Everything
A language model by itself can generate text.
Give it tools, and now it can affect other systems.
A tool might allow an agent to:
-
turn on a light
-
query a sensor
-
search the web
-
read a file
-
send a message
-
call an API
-
run a command
-
query a database
-
create a calendar event
The model does not usually execute the underlying action directly.
Instead, it produces a structured request saying, essentially:
“I would like to use this tool with these arguments.”
The surrounding system validates that request, executes the tool, and returns the result to the model.
Conceptually:
human → agent → model
model → requests tool
harness → executes tool
tool result → model
model → next decision
That loop may happen once.
It may happen twenty times.
This is where AI begins to look less like a chatbot and more like software capable of doing work.
It is also where we need to start paying very serious attention to permissions.
We will get there.
Oh, we will absolutely get there.
Context Is What the Model Gets to Know
A model has no magical awareness of the system around it.
It operates on the information made available during inference.
That information is broadly called context.
Context may include:
-
your current request
-
system instructions
-
previous conversation
-
Home Assistant state
-
device information
-
tool definitions
-
retrieved documents
-
memories
-
search results
-
files
-
results from previous tool calls
This is another reason two systems using exactly the same model can behave very differently.
One may provide the model with a giant undifferentiated dump of data.
Another may provide a small, carefully selected representation of exactly what matters.
Same model.
Very different system.
We touched on this in Chapter 1.
Later we are going to spend an entire section on it because context engineering is one of the central disciplines in building useful AI systems.
Memory Is Not the Same Thing as Context
This distinction trips people up constantly.
The model has context available during the current inference.
Memory is some mechanism for retaining useful information beyond that immediate interaction and making it available again later.
That memory might live in:
-
a database
-
conversation history
-
documents
-
summaries
-
structured records
-
a vector store
-
Home Assistant itself
The model does not need to permanently contain that information.
The surrounding system can retrieve it when needed and put it back into context.
So:
context = what the model can see now
memory = how the system can make useful information available again later
Keep that distinction in your pocket.
It will save us considerable confusion when we reach RAG.
Retrieval Gives the Model More Information
Sometimes the system has far more information available than we could reasonably stuff into every request.
Perhaps we have:
-
manuals
-
Home Assistant documentation
-
household records
-
maintenance history
-
thousands of notes
-
years of logs
Instead of giving all of that to the model every time, we can retrieve the parts that appear relevant to the current task.
One common approach is Retrieval-Augmented Generation, usually shortened to RAG.
Very roughly:
question → search useful information → add results to context → model answers
Modern systems can make that considerably more sophisticated.
The agent may decide when retrieval is needed.
It may search several sources.
It may evaluate the results.
It may search again.
It may combine ordinary keyword search, databases, knowledge graphs, embeddings, reranking, or other mechanisms.
We will deal with all of that later.
Including, inevitably:
What the hell is a vector, Victor?
For now:
Retrieval is how we find useful information.
Context is how we give that information to the model.
Routers Decide Where Work Goes
Another component that is becoming increasingly common is the router.
And no, for once we are not talking about the box blinking angrily next to your fiber connection.
An AI router decides where a request should go.
Imagine a system with access to:
-
a tiny fast local model
-
a larger local reasoning model
-
a cloud model
-
a vision model
-
a specialist coding model
There is no reason every task needs to go to the same place.
A router might decide:
“This is a simple intent. Send it to the fast local model.”
or:
“This requires vision. Send the image to the multimodal service.”
or:
“This is complicated and the local models are struggling. Escalate it to the cloud model.”
Routing can be based on capability, speed, cost, privacy, availability, workload, or other policies.
Conceptually:
request → router → appropriate model or service
This becomes particularly useful once your AI installation graduates from a model to a small collection of things humming ominously in a closet.
We will revisit routers when we talk about local AI infrastructure.
Some Agents Hang Around
A coding agent usually exists in a fairly obvious working session.
You ask it to work on something.
It works.
Eventually the job ends.
Another category of system is designed to remain available over longer periods.
These systems may maintain sessions, connect to messaging platforms, expose tools, manage multiple agents, receive events, and keep an AI assistant available as a persistent service.
OpenClaw is a good example of this architecture. Its current design centers on a long-running gateway that connects communication channels, agent sessions, tools, and model providers.
Systems in this category start looking less like:
“Here is an AI application I opened.”
and more like:
“There is an agent service running somewhere.”
That difference matters enormously for home automation.
A home is already a persistent environment.
Events happen whether somebody has a chatbot open or not.
Doors open.
People arrive.
Temperatures change.
Automations fire.
Devices fail.
An AI system intended to become part of that environment may eventually need to operate on similar timescales.
That leads us toward persistent agents, event-driven agents, schedulers, supervisors, and orchestration.
We are not going down that rabbit hole quite yet.
I can already hear it digging.
Then We Get Systems Around the Agents
Once you have multiple agents, another layer can appear.
Something has to decide:
-
which agent gets a job
-
what jobs exist
-
what state the work is in
-
whether an agent failed
-
whether another agent should take over
-
how work is approved
-
which resources an agent can access
This is orchestration.
At this point, we are no longer merely wrapping a model.
We are coordinating systems that themselves wrap models.
You can think of the rough progression like this:
model → inference service → harness → agent → orchestration
Do not treat that as a rigid industry standard.
Real products cross boundaries.
Some bundle several layers together.
Some use different terminology.
Some will undoubtedly rename the entire stack next Tuesday.
The diagram is a mental model.
That is what we need.
Put the Pieces Together
A reasonably complete AI system might now look something like this:
HUMAN
│
Voice / Text / UI
│
▼
AGENT / HARNESS
│
┌────────────┴────────────┐
│ │
CONTEXT TOOLS
│ │
└────────────┬────────────┘
│
▼
MODEL
│
▼
INFERENCE SERVICE
│
▼
RESULT
Add retrieval:
HUMAN
│
▼
AGENT / HARNESS
│
┌─────────────┼─────────────┐
│ │ │
RETRIEVAL CONTEXT TOOLS
│ │ │
└─────────────┼─────────────┘
▼
MODEL
│
▼
INFERENCE SERVICE
Add several models:
REQUEST
│
▼
ROUTER
│
┌─────────────┼─────────────┐
│ │ │
FAST LLM BIG LLM VLM
│ │ │
└─────────────┼─────────────┘
▼
RESULT
Then add persistent agents, speech services, databases, Home Assistant, MCP servers, external tools, memory, multiple inference machines, agent orchestration, permissions, and identity.
Congratulations.
You now have an architecture diagram that looks like someone dropped a bowl of spaghetti on Visio.
This is normal.
So Where Does Home Assistant Fit?
Almost anywhere.
Home Assistant might be:
The source of context
Here is the current state of the house.
The provider of tools
Here are the things the agent is permitted to do.
The automation engine
This event happened. Ask AI to classify it.
The conversation surface
The human is talking to Assist.
The destination of an action
The external agent decided to ask Home Assistant to turn something on.
The orchestrator of a bounded AI task
Send this image for analysis, get the result, continue the automation.
Or some combination of those.
There is no single architecture called AI in Home Assistant.
There are a lot of architectures that connect these pieces in different ways.
The Vocabulary We Will Use
Throughout the rest of this primer, I will generally use these terms:
Model
The trained neural network performing inference.
Inference service
The software and infrastructure that runs a model.
Agent harness
The environment around a model that supplies tools, context, instructions, execution, and workflow.
Agent
An AI system capable of pursuing a goal by choosing actions and using available capabilities.
Tool
A defined capability the agent can request.
Context
The information available to the model during a particular inference.
Memory
A mechanism for retaining useful information and making it available again later.
Retrieval
Finding relevant information to place into context.
Router
A component that decides which model, service, or agent should receive a task.
Orchestrator
A system that coordinates work across agents, tasks, or services.
These definitions are deliberately practical rather than doctrinal.
If somebody publishes an academic paper tomorrow with a more precise taxonomy, wonderful.
We are trying to automate a house.
We need words that let us talk about which damn box is doing what.
One Last Distinction
There is a useful question you can now ask whenever somebody shows you a new AI product:
Which part of the stack is this?
Is it the model?
Does it run models?
Does it wrap models with tools?
Does it provide memory?
Does it route requests?
Does it manage persistent agents?
Does it orchestrate several agents?
Does it connect those agents to other systems?
Often the answer will be:
“Several of those.”
That is fine.
Now you know which questions to ask.
And that brings us to the next layer of confusion:
If the model is only one part of the system, what exactly are things like Codex, Claude Code, and OpenCode doing?
Welcome to the harness…