Chapter 4: Where Does the Thinking Actually Happen?
So far we have separated the model from the harness.
Now we need to answer a very practical question:
Where does the model actually run?
Because once the harness decides:
“I need a model to think about this.”
something somewhere has to do the computation.
That might be:
- a cloud service
- another computer in your house
- the same computer running Home Assistant
- a cluster of machines
- a specialized speech processor
- a tiny model running directly on low-power hardware
And increasingly, the answer may be:
“Several of those, depending on the job.”
Welcome to inference.
Inference Is the Actual Thinking Work
In Chapter 2 we defined inference as the process of running input through a trained model to produce an output.
The model exists.
Inference is using it.
If you ask:
“What is happening in the living room?”
the harness prepares the request.
Then an inference service takes that request, runs the model, and returns the result.
Conceptually:
HARNESS
│
▼
INFERENCE SERVICE
│
▼
MODEL
│
▼
RESULT
The harness and inference service might live on the same machine.
They might be separated by your LAN.
They might be separated by half the planet.
To the harness, the basic interaction can look surprisingly similar.
Cloud Inference
Cloud inference is probably the model most people encounter first.
Your application sends a request to a provider.
The provider runs the model on its infrastructure.
The result comes back.
HOME ASSISTANT
│
▼
HARNESS
│
▼
INTERNET
│
▼
CLOUD PROVIDER
│
▼
MODEL
Home Assistant currently supports cloud AI providers through integrations including OpenAI, Google, and Anthropic for AI-related capabilities. Home Assistant also supports local inference through Ollama.
Cloud inference has some obvious attractions.
You do not have to buy the hardware.
You do not have to load the model.
You do not have to manage GPU drivers.
You do not have to figure out why CUDA has decided that today is a character-building exercise.
You call an API.
Someone else’s machines do the work.
What You Are Really Buying From the Cloud
When you pay for cloud AI, you are buying more than access to a model.
You are also effectively renting:
- compute
- memory
- accelerators
- model hosting
- scaling
- deployment
- updates
- availability infrastructure
That can be an extremely good trade.
A model that would require substantial local hardware may be available to you through a simple API call.
It also means your application depends on:
- network connectivity
- provider availability
- provider pricing
- provider policy
- whatever data is sent to that provider
None of those automatically make cloud inference a bad choice.
They are simply architectural properties.
Remember the habit from Chapter 3:
Find where the work is actually happening.
Local Inference
Local inference means the model is running on hardware you control.
That might be:
HOME ASSISTANT
│
▼
OLLAMA
│
▼
LOCAL MODEL
or:
HOME ASSISTANT
│
▼
LAN
│
▼
INFERENCE SERVER
│
▼
GPU
Home Assistant’s Ollama integration is designed specifically to connect Home Assistant to an Ollama server running language models.
Other software can perform the same general inference-server role.
For example, llama.cpp includes a lightweight HTTP server with OpenAI-compatible APIs and is designed to run LLM and VLM inference across a wide range of hardware.
vLLM is another inference and serving system, aimed heavily at fast, high-throughput model serving.
Do not worry yet about which one you should use.
That comes later.
The important point is that:
Ollama, llama.cpp, and vLLM are not the models.
They are systems for running models.
Local Does Not Mean “On Home Assistant”
This distinction causes a surprising amount of confusion.
Suppose Home Assistant is running on a small Intel box.
Your LLM is running on a workstation with a large GPU downstairs.
You connect them over your LAN.
That is still local inference.
HOME ASSISTANT
│
│ local network
▼
GPU WORKSTATION
│
▼
INFERENCE SERVICE
│
▼
MODEL
The model does not have to live inside Home Assistant.
In many cases, separating the two is actually useful.
Home Assistant can remain a boring, reliable appliance.
The AI hardware can be noisy, expensive, experimental, frequently rebooted, and full of questionable decisions involving beta drivers.
This is a healthy relationship.
Local Also Does Not Mean Private by Magic
Running the model locally can reduce the amount of information sent to an external AI provider.
That does not automatically make the complete system local.
Remember Chapter 3.
Your local model might still be used by a harness that:
- searches the web
- calls remote APIs
- uses cloud memory
- connects to external tools
- sends telemetry
- delegates some tasks elsewhere
So this:
LOCAL MODEL
does not necessarily mean:
LOCAL SYSTEM
When privacy matters, trace the whole path.
Where does the context go?
Where do tool results go?
Where does memory live?
Where does inference happen?
Those are separate questions.
Fully Local Is a System Property
A fully local architecture might look like:
VOICE
│
▼
LOCAL SPEECH RECOGNITION
│
▼
HOME ASSISTANT
│
▼
LOCAL HARNESS
│
▼
LOCAL INFERENCE
│
▼
LOCAL MODEL
│
▼
HOME ASSISTANT TOOLS
│
▼
LOCAL TEXT TO SPEECH
Nothing in the primary interaction path has to leave the local network.
Home Assistant explicitly supports fully local voice configurations. Its Speech-to-Phrase system can run locally on lower-powered hardware by limiting itself to focused home-control language rather than arbitrary general speech.
That brings us to an important distinction.
Speech Services Are Not LLM Inference Services
When somebody says:
“My AI voice system runs fine on a Raspberry Pi.”
we still need to ask:
Which part?
A voice assistant is a pipeline.
Very roughly:
MICROPHONE
│
▼
WAKE WORD
│
▼
SPEECH TO TEXT
│
▼
CONVERSATION / AGENT
│
▼
TEXT TO SPEECH
│
▼
SPEAKER
Those are different computational jobs.
Home Assistant itself treats text-to-speech as a separate building-block integration supplied by TTS providers.
Speech-to-Phrase is another example of a specialized service. It performs focused local speech recognition and intentionally does not provide full general speech-to-text capability.
So this:
“Speech recognition works great on this little machine.”
does not imply:
“A general-purpose reasoning model will also work great on this little machine.”
Different model.
Different workload.
Different performance requirements.
Specialized Models Change the Hardware Equation
This connects directly to the tiny-model discussion from Chapter 3.
A specialized model can be designed for a narrow task.
For example:
“Did someone say the wake word?”
or:
“Which of these known home-control phrases did the user say?”
or:
“Is there a person in this image?”
or:
“Turn this text into an embedding.”
Those tasks can often be solved with models far smaller than a general-purpose LLM.
Home Assistant’s Speech-to-Phrase deliberately limits the possible language it needs to recognize, which allows it to operate efficiently on lower-powered systems.
That is a feature.
The model is smaller because the problem is smaller.
General Reasoning Is a Different Workload
Now compare:
“Turn on the kitchen light.”
with:
“The house seems unusually warm upstairs, we have guests arriving in two hours, nobody is currently using those rooms, and electricity prices are high. Figure out what makes sense.”
That second request may require:
- a larger context
- tool definitions
- current Home Assistant state
- several tool calls
- reasoning across multiple facts
- possibly external information
- multiple model passes
The workload is no longer:
recognize one phrase
It is:
reason about a changing environment
That tends to demand significantly more model capability and compute.
Which is why:
tiny specialist AI
and:
general household agent
should not be treated as the same hardware problem.
Latency Is More Than Tokens Per Second
Once people start running local models, a number appears everywhere:
tokens per second
Useful number.
Not the whole story.
Imagine asking:
“Turn the fan off.”
Suppose the model can generate its answer at 50 tokens per second.
Sounds fast.
But first it may need to process:
- the system prompt
- conversation history
- Home Assistant context
- entity information
- tool definitions
- your request
The system might spend several seconds processing all of that before generating the first token.
So the actual experience may look like:
REQUEST
│
▼
PROMPT PROCESSING
│
│ several seconds
▼
FIRST TOKEN
│
▼
GENERATION
For an interactive home assistant, time to first useful action can matter more than headline generation speed.
We will dig into this much more when we talk hardware and context.
For now, remember:
Fast generation does not automatically mean fast interaction.
Context Costs Compute
This connects two major parts of the primer.
Context is not free.
The model has to process the context you give it.
If your harness sends:
- hundreds of entities
- long tool descriptions
- conversation history
- household documentation
- giant system prompts
the inference service has to process that information.
This is why:
“The question was simple.”
does not necessarily mean:
“The inference workload was simple.”
Your spoken request might contain five words.
The actual prompt arriving at the model might contain thousands.
That difference is one of the reasons AI performance inside Home Assistant can surprise people.
Local Inference Servers Have Different Goals
Not every local inference server is optimized for the same thing.
llama.cpp emphasizes broad hardware support and relatively straightforward local inference. Its server provides REST and OpenAI-compatible interfaces.
vLLM is designed around efficient serving and high throughput, and supports both offline inference and online serving.
Ollama focuses heavily on making local model management and serving approachable, and Home Assistant provides an official integration for connecting to it.
These systems overlap.
They are not identical.
Later, when we design a local inference farm, we will care about questions such as:
- one user or many users?
- one model or several?
- interactive latency or throughput?
- CPU, GPU, or mixed acceleration?
- how much context?
- how many simultaneous requests?
- how frequently do models change?
Those questions determine the infrastructure.
One GPU Is Not Automatically a Farm
Let us keep the terminology under control before someone buys a rack.
A perfectly good local AI installation might be:
ONE MACHINE
│
├── speech
├── LLM
└── embeddings
Or:
HA BOX
│
▼
GPU BOX
That may be all you ever need.
A farm becomes useful when workloads start separating.
For example:
HOME ASSISTANT
│
┌─────────┼─────────┐
│ │ │
▼ ▼ ▼
SPEECH FAST LLM BIG LLM
NODE NODE NODE
Now different machines or services handle different jobs.
We will build this properly in Chapter 5.
For now, the important idea is:
The AI system does not have to be one model on one computer.
Hybrid Inference
Now we arrive at what will probably become an extremely common architecture.
Hybrid inference means some work stays local while other work goes elsewhere.
For example:
REQUEST
│
▼
HARNESS
│
┌───────┴───────┐
│ │
SIMPLE / LOCAL COMPLEX
│ │
▼ ▼
LOCAL MODEL CLOUD MODEL
Maybe:
“Turn off the bedroom light.”
stays local.
But:
“Research the best way to redesign my HVAC automation based on tomorrow’s weather and our recent energy consumption.”
goes to a larger remote model.
Or perhaps privacy determines the route.
Household state remains local.
General knowledge questions go to the cloud.
Or cost determines the route.
Cheap work stays local.
Rare difficult tasks use a premium model.
Or availability determines the route.
Cloud service unavailable?
Fall back locally.
There are many possibilities.
Local and Cloud Are Deployment Choices
This is worth making explicit.
Local and cloud are not types of intelligence.
They describe where inference happens.
The same model family might even be available in both places.
So:
LLM versus VLM
describes model capability.
local versus cloud
describes deployment.
Assist versus OpenCode
describes harness.
Ollama versus vLLM
describes serving infrastructure.
These axes are independent.
That is why an AI architecture can seem confusing until you separate the layers.
Hybrid Does Not Mean Half Local
You will sometimes see a system described as “local AI” because some component runs locally.
Be careful with that phrase.
Imagine:
LOCAL WAKE WORD
│
LOCAL SPEECH TO TEXT
│
REMOTE LLM
│
LOCAL HOME ASSISTANT
│
LOCAL TTS
Most components are local.
The reasoning is not.
Or:
LOCAL LLM
│
REMOTE MEMORY
│
REMOTE WEB TOOLS
│
LOCAL HOME ASSISTANT
The model is local.
The full system is not.
Neither architecture is inherently wrong.
Just describe them accurately.
Why Hybrid Often Makes Sense
A home produces a wide range of AI workloads.
Some are tiny:
“Did somebody say the wake word?”
Some need speed:
“Turn on the light.”
Some need vision:
“What was left on the driveway?”
Some need broader reasoning:
“Why has energy use increased this week?”
Some happen constantly.
Others happen twice a month.
Building one machine large enough to handle the most expensive possible workload may be wasteful.
Sending every tiny task to a huge cloud model may also be wasteful.
A hybrid system lets different tools solve different classes of problems.
This is where routers eventually enter the story.
Not the box with the blinking Ethernet ports.
The other kind.
Home Assistant Is Particularly Well Suited to Hybrid AI
Home Assistant already acts as an integration layer for many independent services.
Voice is already a pipeline.
AI Tasks can be supplied by different AI integrations, including local Ollama or cloud providers.
Conversation agents can similarly connect Home Assistant to different model providers.
That architecture naturally supports the idea that:
different AI jobs may use different AI services.
The smart home does not need one giant brain.
It can use several specialized capabilities.
That should feel familiar.
Home Assistant has been doing essentially the same architectural trick with devices and services for years.
Do Not Put Reliability-Critical Automation Behind Inference
We should also connect this back to Chapter 1.
Suppose your network goes down.
Your cloud model disappears.
Or your local inference machine is rebooting after a driver update.
Your lights should still work.
Your leak detector should still work.
Your smoke alarm should still work.
Your basic climate automation should still work.
This is why:
Keep the deterministic parts deterministic.
AI can enhance the system.
It should not casually become a hard dependency for functions that were already reliable without it.
A hybrid architecture makes this especially easy to design well.
Home Assistant remains Home Assistant.
Inference becomes another service it can use when appropriate.
Think in Services
Instead of imagining:
THE AI
start imagining services.
Perhaps your home eventually has:
HOME ASSISTANT
│
├── speech recognition service
│
├── text-to-speech service
│
├── fast local LLM service
│
├── large reasoning service
│
├── vision service
│
└── embedding service
Some may run on the same machine.
Some may run elsewhere.
Some may be cloud APIs.
Some may be tiny local models.
Home Assistant and your harness decide which service is useful for the current job.
Once you think this way, the hardware question changes from:
“What computer do I need for AI?”
to:
What workloads do I want to run locally, and what resources do those workloads require?
Now we finally have a good hardware question.
Four Questions Before Buying Anything
Before you start shopping for GPUs, answer these:
1. What needs to be local?
Everything?
Speech?
Household context?
Only basic commands?
Only sensitive information?
2. What needs to be fast?
Voice interaction?
Background analysis?
Image processing?
Scheduled jobs?
Interactive latency matters enormously for some workloads and almost not at all for others.
3. What models do those jobs require?
Tiny specialized model?
Small LLM?
Large reasoning model?
Vision model?
Several models?
4. How many things need to run at once?
One conversation?
Several voice satellites?
Background AI Tasks?
Vision jobs?
Multiple agents?
This is where capacity planning begins.
Not with:
“How many TOPS does this box have?”
We will get there.
The Stack So Far
Our architecture now looks something like this:
HUMAN / EVENT
│
▼
HARNESS
│
▼
ROUTE
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
SPEECH LOCAL LLM CLOUD LLM
SERVICE SERVICE SERVICE
│ │ │
└─────────────┼─────────────┘
│
▼
HOME ASSISTANT
We now know:
- what the model is
- what the harness is
- what inference is
- where inference can happen
- why speech and general reasoning are different workloads
- why local and cloud are deployment choices
- why hybrid architectures are useful
Which means we can finally talk about the thing everyone wanted to talk about in the first place.
Hardware.
Because if you have decided:
“I want this particular workload to run locally.”
the next question becomes:
What does the machine running it actually need?
And that brings us to: