Chapter 7: Context, Memory, RAG, ARAG, and What the Hell Is a Vector, Victor?
We have now done something very clever.
We selected a model.
We built an inference service.
We gave it a harness.
We may even have a router deciding which model gets which job.
Wonderful.
Now ask it:
“Where did we put the spare water filters?”
And unless somebody supplied that information, the model has absolutely no idea.
This is one of the most important things to understand about AI systems:
The model only knows what is inside the model and what you give it for this request.
Your house is not secretly stored inside the LLM.
Neither are your receipts.
Neither is your Home Assistant history.
Neither is the conversation you had with it six months ago unless the system preserved that information and made it available again.
This chapter is about how we do that.
Context
Context is the information available to the model while it is handling the current request.
That may include:
system instructions
+
user request
+
conversation history
+
Home Assistant state
+
tool definitions
+
retrieved information
+
memory
Conceptually:
CONTEXT
│
▼
MODEL
│
▼
RESPONSE
Change the context and you may get a very different result from exactly the same model.
This is why the model is only one component of the system.
Context Engineering
Once you understand that, a lot of modern AI architecture becomes easier to understand.
If context affects what the model can do, deciding:
- what information to include
- what to leave out
- what to summarize
- what to retrieve
- what history to preserve
- which tools to expose
becomes an engineering problem.
That is context engineering.
Home Assistant makes this especially interesting because a developed home can produce a ridiculous amount of information.
You could dump:
1,500 entities
hundreds of attributes
device metadata
battery states
firmware versions
sensor history
weather
power readings
automations
into every prompt.
Please don’t.
More Context Is Not Automatically Better
Modern models can support enormous context windows.
That does not mean your goal should be:
“Fill it.”
More context can mean:
- more prompt-processing time
- more memory use
- more cloud cost
- more irrelevant information
- more conflicting information
- more work for the model to sort through
A better goal is:
Give the model enough relevant information to do the job well.
That often means less information, chosen better.
Home Assistant Gives Us a Head Start
Home Assistant data already has structure.
We know things such as:
entity belongs to device
device belongs to area
sensor has state
state has history
We may also know where the request came from.
If someone says:
“It’s too warm in here.”
the harness may already know:
- which satellite heard them
- which room they are in
- current room temperature
- HVAC state
- nearby windows
- occupancy
That lets us reduce a giant home state to a small useful slice of context.
This is why useful summaries, entity metadata, area relationships, and semantic home models become so valuable to an AI system.
Breadcrumbs Beat Dumps
One of the patterns I like most is:
summary first, detail available on demand
For example:
HOME OVERVIEW
Living Room:
occupied
lights on
74°F
detail available
Kitchen:
vacant
dishwasher running
detail available
The model gets enough information to understand what is going on.
If it needs more:
get kitchen detail
it can retrieve the deeper information.
That scales much better than dumping the entire raw house into every prompt.
Semantic Summaries Are Powerful
Suppose a room exposes:
state: engaged
reason: television + seated presence
That may be much more useful to an LLM than:
motion sensor = off
chair sensor = on
TV power = 91W
media player = playing
The raw signals still exist.
The summary gives the model a useful semantic representation.
If it needs the raw details, it can follow a tool or retrieval path down to them.
That is context engineering.
Context Is Temporary
Context normally exists for the current interaction.
The model does not automatically preserve everything it sees forever.
Your harness may preserve conversation history.
Another system may save important facts.
The information may be retrieved again later.
But the underlying LLM did not suddenly grow long-term memory.
Which brings us to:
memory
What Is Memory?
For our purposes:
Memory is a mechanism for preserving useful information and making it available again later.
Architecturally, “the AI remembered” often looks more like this:
USER SAYS SOMETHING
│
▼
SYSTEM STORES IT
│
▼
TIME PASSES
│
▼
SYSTEM FINDS IT
│
▼
PUTS IT IN CONTEXT
│
▼
MODEL USES IT
From the human side:
“It remembered.”
From the system side:
“We stored something useful and retrieved it later.”
What Should Become Memory?
Do not remember everything.
Useful memory might include:
- durable preferences
- household terminology
- where something is stored
- important prior decisions
- facts that are difficult to retrieve elsewhere
- summaries of previous conversations
For example:
"the good lamp" = light.living_room_reading
or:
spare HVAC filters are in garage cabinet 3
Those may be useful later.
But if someone says:
“I’m cold.”
that probably should not immediately become:
PERMANENT FACT:
User prefers house at 82°F
Memory needs judgment.
Use the Source You Already Have
This is probably the most important practical point in this chapter.
A lot of household information already has a home.
If you want to know:
“What is the kitchen temperature?”
Home Assistant already knows.
If you want:
“When did I replace the filter?”
your maintenance system may already know.
If you want:
“What is on the shopping list?”
your shopping system may already know.
If you want:
“What recipe did we use last time?”
your recipe system may already know.
My preference is to leave information in the system that owns it and give the harness a good way to retrieve it.
The Source Is Only Useful If the Agent Can Reach It
There is a caveat.
The information can be perfectly organized in the authoritative system and still be useless to the model if the harness has no good way to get it.
A useful path looks like:
AUTHORITATIVE SYSTEM
│
▼
GOOD TOOL
│
▼
HARNESS
│
▼
MODEL
That good tool might be:
- a native Home Assistant tool
- an API wrapper
- an indexed search tool
- an MCP tool
- a custom integration
We will spend a lot more time on the tool side in Chapter 8.
For now:
Data existing somewhere and an agent being able to use it are two different things.
My Preference: Give Home Assistant Better Retrieval First
If the information already belongs to the house, I prefer giving the Home Assistant harness a useful retrieval method before bolting another system beside it.
Conceptually:
HOME ASSISTANT DATA
│
▼
INDEX
│
▼
RETRIEVAL TOOL
│
▼
HARNESS
│
▼
MODEL
The index might cover:
- entities
- areas
- devices
- household summaries
- history
- documents known to the home
- structured household knowledge
Now the model does not need all of that information stuffed into every prompt.
The harness can ask for what it needs.
For example:
“Find what we know about the guest-room air conditioner.”
or:
“Find maintenance information related to the water heater.”
or:
“Give me the detail behind the kitchen summary.”
The important part is that retrieval remains a capability of the Home Assistant working environment.
That is a very natural fit.
This Is Still Retrieval-Augmented Generation
RAG sounds much more exotic than it needs to.
RAG means:
Retrieval-Augmented Generation
In plain English:
Find useful information and give it to the model for this request.
So this:
USER QUESTION
│
▼
HA INDEX TOOL
│
▼
RELEVANT HOUSE INFORMATION
│
▼
MODEL CONTEXT
│
▼
ANSWER
is absolutely the same broad idea.
The retrieval backend does not have to be a vector database.
That point matters.
Retrieval Does Not Automatically Mean Vector Database
Sometimes retrieval is simply:
Home Assistant state lookup
or:
history query
or:
indexed search
or:
API call
or:
document lookup
If someone asks:
“Is the garage door open?”
use the garage-door state.
If someone asks:
“What was the bedroom temperature at 3 AM?”
use history.
If someone asks:
“Find the maintenance notes for the air handler.”
an index may be exactly the right answer.
Use the structure you already have.
So When Does Semantic Retrieval Help?
Now imagine you have:
appliance manuals
warranties
receipts
house notes
network documentation
maintenance notes
Then someone asks:
“How do I reset the filter warning on the refrigerator?”
The document may actually say:
“Clearing the water filter indicator.”
Exact search may or may not find that.
Semantic retrieval lets the system search by meaning instead.
That is where embeddings and vector search become useful.
What the Hell Is a Vector, Victor?
This is about as deep as we need to go.
An embedding model converts something such as text into a list of numbers.
"reset refrigerator filter warning"
│
▼
EMBEDDING MODEL
│
▼
[0.18, -0.42, 0.91, ...]
That list is a vector.
Texts with similar meanings tend to produce vectors that are closer together.
So:
“reset refrigerator filter warning”
can match:
“clearing the water filter indicator”
even though the wording is different.
That gives us search by meaning.
Victor may now return to his seat.
Do I Need to Build a Vector Database?
Probably not.
At least not to get started.
There is an entire discipline hiding underneath that little box.
Things like:
- embedding models
- indexing
- metadata
- chunking
- similarity thresholds
- hybrid search
- tuning
- lifecycle management
can turn into a black art of their own.
That is advanced territory.
For this primer, you only need to understand:
KNOWLEDGE
│
▼
RETRIEVAL SYSTEM
│
▼
RELEVANT INFORMATION
│
▼
MODEL CONTEXT
Whatever lives inside that RETRIEVAL SYSTEM box can come later.
A Real Home Assistant Example: Weaviate
There is already a useful example of this approach in the Home Assistant community.
Skye Harris’ Local OpenAI LLM custom integration includes experimental RAG support using Weaviate.
Conceptually:
USER MESSAGE
│
▼
LOCAL OPENAI LLM
│
▼
WEAVIATE
│
▼
MATCHING KNOWLEDGE
│
▼
ADD TO CURRENT CONTEXT
│
▼
MODEL
The integration queries Weaviate using the current user message and injects matching content into the current conversation context.
Importantly, the integration itself describes this as retrieval, not general-purpose persistent agent memory.
That distinction matters.
Same Goal, Different Direction
The HA-native index approach and a Weaviate-backed RAG approach solve related problems from different directions.
An indexed HA approach may look like:
HOME ASSISTANT KNOWLEDGE
│
▼
HA INDEX
│
▼
RETRIEVAL TOOL
│
▼
HARNESS
A Weaviate-backed approach might look like:
UNSTRUCTURED KNOWLEDGE
│
▼
WEAVIATE
│
▼
CONTEXT INJECTION
│
▼
MODEL
Neither is inherently the correct answer.
They fit different information shapes.
And They Are Not Mutually Exclusive
This is important.
You can absolutely use both.
For example:
HARNESS
│
┌───────────┴───────────┐
│ │
▼ ▼
HA INDEX TOOL WEAVIATE RAG
house-native semantic/document
knowledge knowledge
│ │
└───────────┬───────────┘
│
▼
CONTEXT
Now the harness has several ways to get information.
For exact or structured household knowledge:
HA INDEX / TOOLS
For fuzzy semantic document retrieval:
VECTOR RAG
For current device state:
DIRECT HA TOOL
Use the right retrieval path for the information.
A Simple Decision Flow
When the agent needs information, start here:
DOES THE INFORMATION ALREADY LIVE
IN AN AUTHORITATIVE SYSTEM?
│
├── YES
│ │
│ ▼
│ IS THERE A GOOD TOOL
│ OR INDEX TO RETRIEVE IT?
│ │
│ ├── YES → USE IT
│ │
│ └── NO → BUILD A BETTER PATH
│
└── NO
│
▼
IS THIS MOSTLY
DOCUMENT / KNOWLEDGE DATA?
│
▼
RETRIEVAL / RAG MAY HELP
Then ask:
“Do I actually need semantic search?”
If ordinary search works, use ordinary search.
If meaning matters more than exact wording, semantic retrieval may help.
RAG Does Not Train the Model
This is another common misunderstanding.
If you add your refrigerator manual to a retrieval system, the model has not learned the refrigerator manual.
Its weights have not changed.
Instead:
QUESTION
│
▼
RETRIEVE MANUAL SECTION
│
▼
PUT IT IN CONTEXT
│
▼
MODEL ANSWERS
That is useful because household knowledge changes.
Replace the refrigerator.
Replace the document.
The next retrieval can use the new information.
No model retraining required.
Keep the Source
If the model answers:
“The pump has a five-year warranty.”
you should ideally know where that information came from.
Useful source information might include:
document
device
date
version
page
URL
That helps with:
- trust
- debugging
- stale information
- citations
A response backed by:
dishwasher_warranty.pdf
is a lot more useful than:
“The AI says so.”
Homes Change
Devices get replaced.
People move things.
Documents become obsolete.
Preferences change.
Suppose retrieval contains:
thermostat_manual_old.pdf
thermostat_manual_current.pdf
A semantic system can retrieve a wonderfully relevant answer from the wrong manual.
So context systems benefit from relationships such as:
device
room
date
version
person
Again, Home Assistant already gives us a lot of useful structure.
Use it.
RAG Can Stay Simple
A perfectly useful household RAG flow can be:
KNOWLEDGE
│
▼
INDEX
│
▼
SEARCH
│
▼
RELEVANT INFORMATION
│
▼
MODEL
That handles a tremendous number of problems.
You do not need to begin by designing the retrieval engine itself.
If an integration, harness, or document platform already provides that capability, start there.
The retrieval engine underneath it can remain somebody else’s black art until you have a reason to care.
So What Is ARAG?
Once RAG makes sense, agentic RAG is easy to understand.
Basic RAG is roughly:
QUESTION
│
▼
SEARCH
│
▼
ANSWER
Agentic RAG lets the agent decide what information it needs while solving the task.
For example:
“Is the dishwasher still under warranty, and would this leaking pump probably be covered?”
The agent might decide it needs:
receipt
warranty
model number
maintenance history
Then retrieve those things.
QUESTION
│
▼
AGENT
│
├── find receipt
├── find warranty
├── check model
└── check maintenance
│
▼
ANSWER
Now retrieval is part of the reasoning loop.
Where Agentic Retrieval Gets Interesting at Home
A household question can cross several systems.
For example:
“Why has the bedroom been warmer at night this week?”
The agent might need:
HOME ASSISTANT
temperature history
HVAC state
window state
MAINTENANCE
recent HVAC work
MEMORY
someone mentioned closing a vent
WEATHER
outside temperature history
Those sources might be reached through:
- direct tools
- indexed HA retrieval
- semantic RAG
- APIs
- external services
And there is no reason the agent has to choose only one mechanism.
That is where these systems become interesting.
Context, Memory, Retrieval, and RAG
These terms get blurred together constantly.
Keep this picture in your head:
Context
What the model can see right now.
Memory
Information preserved so it can be used again later.
Retrieval
Finding useful information.
RAG
Retrieving external knowledge and placing it into the model’s context.
Agentic RAG
Letting the agent decide what it needs to retrieve while working.
And underneath all of them:
Tools
The paths the harness has to actually reach those systems.
We will deal with those properly next.
Memory Should Be Selective
Memory deserves the same restraint as context.
Useful memory should have some notion of:
- relevance
- durability
- source
- ownership
- confidence
- expiration
Some information belongs forever.
Some belongs for an hour.
Some should not be stored at all.
The goal is:
Make useful information available when it matters.
A Practical Home Architecture
A capable household system might eventually look like:
USER
│
▼
HARNESS
│
┌──────────────┼──────────────┐
│ │ │
▼ ▼ ▼
HA TOOLS HA INDEX MEMORY
│ │
│ │
│ ┌────┴────┐
│ │ │
│ ▼ ▼
│ HOUSE DOCUMENT
│ KNOWLEDGE RAG
│ │
└──────────────┬────┘
│
▼
CONTEXT
│
▼
MODEL
Some information comes directly from Home Assistant.
Some comes from an indexed Home Assistant knowledge layer.
Some may come from semantic document retrieval.
Some comes from memory.
The model does not particularly care which backend produced it.
It cares that the right information arrived.
The Better Question
Earlier in the series we asked:
“Which model should I use?”
Now we can ask:
What information does this model need for this job, where does that information live, and does the harness have a good way to get it?
That is a much better question.
Before You Build Another Thing
Ask yourself:
Does Home Assistant already know this?
Give the harness a good way to retrieve it.
Does another authoritative system already own it?
Use a good tool to reach that system.
Do I need fuzzy document or semantic search?
Use an existing RAG capability if one fits.
Do I need persistent remembered facts?
Add selective memory.
Does the agent need several sources while solving the task?
That is where agentic retrieval starts becoming useful.
Do I need to design my own vector infrastructure?
Probably not yet.
Simple is still better.
Context Is Where the System Becomes Yours
A frontier model may know an enormous amount about the world.
It knows very little about your world.
Your house becomes useful to it through:
context
+
tools
+
retrieval
+
memory
That is how:
“the lamp”
becomes your lamp.
How:
“the dishwasher”
becomes your dishwasher.
How:
“we replaced that last year”
becomes something the system can actually look up.
The general capability comes from the model.
The household intelligence comes from the system around it.
And Now We Need to Let It Touch Things
At this point our agent can:
- understand requests
- choose models
- retrieve relevant information
- use memory
- reason over household context
Excellent.
Eventually it is going to want to do something.
And doing something means connecting the model to capabilities outside itself.
That means:
tools
APIs
MCP
agent-to-agent communication
integration plumbing
And, importantly:
who exactly is allowed to call what?
So next: