Chapter 6: Routers and Model Selection
We ended Chapter 5 with a nice problem.
We have options.
Maybe the house now has:
fast local model
larger reasoning model
vision model
cloud models
A request arrives:
“Turn off the office light.”
Then:
“What kind of bird is sitting on the feeder?”
Then:
“Look at the last six months of power use and figure out why overnight consumption increased.”
Those are very different jobs.
They probably should not all go to the same model.
Welcome to routing.
And once again, I owe the networking people an apology for borrowing one of their words.
First, What Does Home Assistant Need From a Model?
Before choosing models, define the job.
A Home Assistant model usually operates inside a harness.
That harness may provide:
- Home Assistant state
- entities
- areas
- devices
- tool definitions
- conversation history
- household context
- retrieved information
- instructions about what the model may do
Then we expect the model to correctly interpret all of that and take useful action.
That means a great Home Assistant model is not simply:
“The model with the highest benchmark score.”
For HA, I care particularly about a few things.
Instruction Following
Your harness may contain instructions governing:
- tool use
- permitted actions
- response style
- household terminology
- when to ask questions
- when to act
A model that casually ignores those instructions becomes frustrating very quickly.
For an agent, instruction following is part of the control surface.
Tool Calling
This one matters enormously.
A useful Home Assistant model frequently needs to do something like:
USER
│
▼
MODEL
│
│ chooses tool
▼
HOME ASSISTANT
│
▼
RESULT
│
▼
MODEL
The model needs to:
- understand the request
- choose the correct tool
- provide valid arguments
- interpret the result
- continue or answer
Consider:
“Turn off the downstairs lights except the reading lamp.”
Writing a pleasant sentence is easy.
Correctly identifying the downstairs lights, excluding the reading lamp, and calling the right tools is the useful part.
A smaller model with reliable tool use can be much more valuable around the house than a larger model that occasionally gets creative with function arguments.
Structured Output
Agent systems spend a lot of time exchanging structured data.
That may mean:
- JSON
- schemas
- tool arguments
- machine-readable responses
If the harness asks for:
{
"entity_id": "light.kitchen",
"action": "turn_off"
}
you would prefer the model not respond:
“Certainly! Here is the JSON you requested!”
followed by three paragraphs about kitchen lighting.
Strong structured-output behavior makes agent plumbing much easier.
Context
Home Assistant can generate a lot of context.
A household agent may eventually receive:
system instructions
+
home overview
+
tool definitions
+
room state
+
conversation history
+
retrieved knowledge
+
user request
That can become large.
So context-window size matters.
The model’s ability to actually use that context matters too.
A huge advertised context window does not guarantee that information buried deep inside it will influence the model as reliably as you expect.
Test your workload.
Reasoning
Some requests are nearly direct mappings:
“Turn off the kitchen light.”
Others require interpretation:
“It’s getting stuffy in here.”
Or:
“We’re going to bed.”
Or:
“Something downstairs has been using more power lately. Figure out what changed.”
The reasoning requirement grows rapidly once you move beyond command translation.
This is one reason I like having both a fast model and a heavier reasoning model available.
Latency
A model can be brilliant and still make a terrible voice assistant.
If someone says:
“Turn on the kitchen light.”
and the model contemplates the nature of illumination for twelve seconds, nobody will care about its benchmark score.
Interactive Home Assistant use rewards:
- fast prompt ingestion
- low time to first token
- fast tool selection
- short total action time
Background AI Tasks can tolerate more latency.
That difference becomes useful when routing.
Multimodal Capability
If you want the model to interpret:
- cameras
- photographs
- scanned documents
- diagrams
- screenshots
you need multimodal capability somewhere in the system.
For example:
“What is sitting on the porch?”
may require a vision-language model.
You can also use a specialist vision model to describe an image, then send that description to another model.
That is another routing choice.
The Chat Template Matters
Here is one of those things people discover after spending several hours wondering why a perfectly good model suddenly seems drunk.
Models do not all expect conversations to be formatted identically.
They may expect particular:
- system markers
- role markers
- tool-call formats
- reasoning formats
- multimodal placeholders
This is commonly described through the model’s chat template.
Modern inference engines often manage it for you.
You still need to know it exists.
Using the wrong template can make a very capable model behave badly.
Base, Instruct, Chat, Reasoning…
You will encounter names containing things such as:
Base
Instruct
IT
Chat
Reasoning
A base model is generally the underlying pretrained model before the instruction or conversational post-training intended for assistant behavior.
For Home Assistant conversation and agent work, you usually want the appropriate instruction-tuned or agent-capable variant.
Do not install:
ReallyCoolModel-14B-Base
then conclude that ReallyCoolModel is an idiot.
You may have selected the wrong member of the family.
Where Do I Find All This?
For open models, one of your first stops should be:
Hugging Face
Hugging Face is a huge repository of models, model documentation, derived models, quantizations, datasets, and deployment tooling.
A model page may tell you:
- publisher
- architecture
- parameter count
- context size
- license
- modalities
- tool support
- supported inference frameworks
- chat template
- evaluation results
- model files
- quantized variants
Start with the model card.
Yes.
Documentation.
I am sorry.
Read it anyway.
Then look at the actual files and model tree.
The original model might require hardware you do not own while a good GGUF, AWQ, GPTQ, FP8, or another optimized representation fits nicely into the machine you already have.
Do Not Choose a Model From One Benchmark
Benchmarks are useful.
They are also easy to overvalue.
For a Home Assistant model, test Home Assistant-like work.
For example:
“Turn off every light in the guest area except the bathroom.”
“Use the available tools to determine whether the garage was left open.”
“The bedroom is too warm. Do something reasonable.”
“Is anyone still awake downstairs?”
Then inspect what happened.
Did the model:
- choose the right tool?
- invent entities?
- follow instructions?
- use the supplied context?
- recover from failed tool calls?
- respond quickly enough?
- talk when it should have acted?
- act when it should have asked?
That tells you whether the model fits your house.
Models Worth Looking At Right Now
Giant warning label:
THIS SECTION IS DATED AUGUST 2026
This field changes absurdly quickly.
New models are appearing while this article is being written.
So treat these as:
families I would investigate today
rather than:
Nathan’s Eternal Approved Model List Engraved Upon Stone Tablets
Learn how to evaluate them.
Then go see what exists when you actually build.
Qwen
Qwen is one of the first open-model families I would check for serious local Home Assistant work.
As of this writing, Qwen 3.6 is a particularly interesting generation for local inference, with multimodal variants and sizes that fit nicely into the hardware classes we discussed in Chapter 5.
Qwen has also pushed heavily into:
- reasoning
- coding
- agentic behavior
- multimodal work
And because the universe apparently wanted to help make my point, Qwen 3.8 is already beginning to appear while this article is being written.
So do not memorize:
“Qwen 3.6 is the answer.”
Memorize:
Go see what Qwen has now.
Ministral 3
Yes, that spelling is intentional.
Mistral is the company and broader model family.
Ministral is its smaller edge-oriented family.
Ministral 3 deserves particular attention around Home Assistant because it targets local and edge deployment.
The current family includes practical small and medium sizes with instruction and reasoning variants, multimodal capability, function calling, and structured-output support.
That is a very Home Assistant-shaped feature set.
If you have 8, 12, 16, or 24 GB-class local hardware, Ministral belongs on the test list.
Gemma
Google’s Gemma family also belongs on the local evaluation list.
Gemma has consistently targeted efficient deployment, and current generations provide multimodal and instruction-tuned options across useful local sizes.
That makes Gemma interesting for:
- conversational agents
- vision
- tool use
- fast local inference
- specialist workloads
Again, check the current generation rather than blindly using the version number printed here.
gpt-oss
OpenAI’s open-weight gpt-oss family is another interesting option, particularly for reasoning and agentic workloads.
The smaller 20B-class model is relevant because optimized representations can bring it into the memory range of serious consumer hardware.
That makes it useful for the sort of local reasoning and tool-oriented work we care about here.
And Then Check Again
Seriously.
Check again.
Model selection is becoming an ongoing maintenance task.
This is one reason we should avoid wiring Home Assistant directly to one model name forever.
Instead of:
HOME ASSISTANT
│
▼
SomeModel-27B-Q4_K_M-v12
we want to move toward:
HOME ASSISTANT
│
▼
home-fast
Today home-fast may be Qwen.
Next month it might be Ministral.
Six months later it might be something that does not exist while I am typing this.
That is where routing becomes useful.
What Is an AI Router?
At its simplest:
A router decides where an AI request should go.
For example:
REQUEST
│
▼
ROUTER
│
├── FAST LOCAL MODEL
├── LARGE LOCAL MODEL
├── VISION MODEL
└── CLOUD MODEL
That choice may consider:
- capability
- task type
- latency
- cost
- privacy
- modality
- context length
- availability
- current load
Routing can be completely deterministic.
For example:
image request
│
▼
vision model
private household reasoning
│
▼
large local model
simple conversation
│
▼
fast local model
Do not immediately build an AI model whose only job is deciding which AI model should decide things.
Rules are fast.
Rules are predictable.
Rules are cheap.
Use them when they work.
Router, Gateway, Load Balancer, Orchestrator
These words get mixed together constantly.
For our purposes:
Router
Which model or service gets this request?
Gateway
What common endpoint do clients use to reach these services?
Load Balancer
Which equivalent worker gets this request?
Orchestrator
How do multiple jobs, tools, agents, or services cooperate to complete a goal?
A product can perform several of these roles.
Understand the jobs rather than arguing doctrine.
OpenRouter
One of the easiest ways to understand this architecture is OpenRouter.
OpenRouter provides one OpenAI-compatible gateway in front of hundreds of models across many providers.
Conceptually:
APPLICATION
│
▼
OPENROUTER
│
┌───┼───────────────┐
▼ ▼ ▼
MODEL A MODEL B
MODEL C
One API can now reach many different model families.
That alone is useful.
OpenRouter also introduces two different routing decisions.
Model Routing vs Provider Routing
Suppose you want a particular model.
Several providers may serve it.
MODEL X
/ | \
/ | \
PROVIDER A B PROVIDER C
There are now two questions:
Which model should answer?
and:
Who should serve that model?
Those are separate decisions.
Provider Routing
Providers serving the same model may differ in:
- price
- latency
- throughput
- availability
- data policies
- regional availability
OpenRouter can route among those providers and provide failover if one becomes unavailable.
So:
REQUEST
│
▼
MODEL X
│
▼
PROVIDER ROUTER
/ | \
A B C
You chose the brain.
The router chose where that brain runs.
Model Routing
The routing layer can also choose which model handles the request.
OpenRouter’s current Auto Router can inspect the task and select from a changing pool of candidate models.
Conceptually:
REQUEST
│
▼
CLASSIFY WORK
│
▼
SELECT MODEL
│
▼
SELECT PROVIDER
That can be useful when workloads vary widely.
OpenRouter’s current automatic routing even changes its candidate preferences over time based on recent model usage patterns.
There is our model-churn problem again.
Routing helps isolate us from it.
Okay, So Who Do We Escalate To?
Now we get to the cloud model families.
The same things we evaluated locally still matter:
- instruction following
- tool use
- structured output
- context
- reasoning
- multimodality
- latency
Cloud adds a few more questions:
- price
- rate limits
- service reliability
- data handling
- retention terms
- geographic requirements
- provider availability
And once again:
Check what is current.
This list will move too.
OpenAI
As of August 2026, OpenAI’s current GPT-5.6 family gives us a useful example of an escalation ladder:
LUNA
fast / economical
│
▼
TERRA
balanced
│
▼
SOL
heavy reasoning
That maps nicely onto routing.
For example:
cloud-fast
│
▼
Luna
cloud-general
│
▼
Terra
cloud-deep
│
▼
Sol
The model names will change.
The pattern will survive:
fast, general, deep
That is the part worth architecting around.
Anthropic
Anthropic’s Claude family is another obvious cloud escalation target, particularly for agentic work.
Current Claude models are strong candidates for things such as:
- complex tool use
- coding
- planning
- longer-running agent work
- difficult multi-step tasks
That does not mean:
“Claude is the best model.”
It means:
Claude is another strong capability family worth evaluating for the jobs you actually have.
Google, Mistral, Qwen, and Whoever Shipped Yesterday
Other provider families may deserve routes too.
Google’s Gemini family, hosted Mistral models, hosted Qwen variants, and new providers will continue moving in and out of the interesting set.
OpenRouter makes experimenting with those models much easier because your client can remain pointed at one gateway.
This is the point:
Choose capability for the workload, not a logo for the house.
Online Is Not Bad
Let me make one thing very clear.
Online isn’t bad.
And:
Free online isn’t inherently bad either.
What matters is understanding exactly what you are giving away in exchange for free. Full stop.
A free endpoint may be completely reasonable for your use.
A paid endpoint may have terms you dislike.
Paying money does not magically create privacy.
The useful questions are:
- What data leaves?
- Who receives it?
- What do they retain?
- What may they use it for?
- What guarantees apply to this tier?
- What did I receive in exchange?
Make that decision knowingly.
We will spend much more time on privacy and security later.
For now, remember that routing changes where data travels.
That makes routing a policy decision too.
Routing Is Policy
Consider two logical services.
First:
home-private
Its routing policy might be:
home-private
│
├── local-node-a
└── local-node-b
No cloud fallback.
Now:
home-best
might be:
home-best
│
├── large-local
└── OpenRouter
Cloud escalation is allowed.
Those routes intentionally mean different things.
If a request moves from:
LOCAL
to:
CLOUD
the context associated with that request may move too.
So I strongly discourage:
“If local dies, automatically send everything somewhere online.”
That may be technically resilient.
It may also defeat the reason you routed the workload locally.
Local by Policy, Cloud by Choice
A hybrid system might look like:
routine home request
│
▼
fast local model
private household reasoning
│
▼
large local model
image request
│
▼
local VLM
difficult general reasoning
│
▼
OpenRouter
│
▼
chosen cloud model
Local and cloud are resources.
Use the resource appropriate to the job.
So Why Would I Want LiteLLM?
OpenRouter gives us a managed gateway outside our perimeter.
Now imagine you want a similar control point inside your network.
That is where something like LiteLLM becomes interesting.
Conceptually:
HOME ASSISTANT
│
▼
LITELLM
│
┌───┼───────────────┐
▼ ▼ ▼
LOCAL LOCAL OPENROUTER
Home Assistant sees one endpoint.
You control what exists behind it.
LiteLLM can provide a common API over many model backends and apply things such as:
- routing
- fallback
- credentials
- budgets
- limits
- logging
More importantly for our architecture, it can sit between the harness and both internal and external inference.
That gives us a hybrid control point.
Give Capabilities Stable Names
This is one of my favorite tricks.
Do not make Home Assistant care that your current model is:
SomeWonderfulModel-27B-Q4_K_M-v4
Expose:
home-fast
home-reasoning
home-vision
home-private
home-best
Those names represent capabilities.
For example:
home-fast
│
▼
small local model
home-private
│
▼
best suitable local model
home-best
│
▼
OpenRouter escalation route
Now you can replace the model behind home-fast next month.
Home Assistant does not care.
Your automations do not care.
Your harness does not care.
This is a very useful kind of boring.
Where Does LlamaSwap Fit?
Now we go one layer deeper.
Suppose you own one substantial inference box.
On disk it has:
fast conversational model
large reasoning model
coding model
vision model
embedding model
Lovely.
You may have enough disk for all of them.
You probably do not have enough accelerator memory to keep every one loaded simultaneously.
Enter LlamaSwap.
LlamaSwap sits in front of local inference servers and can load models according to the requested model ID.
Think of it as a:
local model residency manager
Model Residency
Suppose the large reasoning model is currently loaded.
Then a request arrives for:
home-vision
The local node may need to do something like:
REQUEST
│
▼
MODEL ALREADY LOADED?
│
├── YES → SERVE
│
└── NO
│
▼
LOAD / SWAP
│
▼
SERVE
LlamaSwap handles that local problem.
LiteLLM may have already decided:
“This request belongs on the heavy inference node.”
Different layer.
Different job.
Three Useful Layers
This gives us a useful mental model:
LlamaSwap manages local model residency.
LiteLLM provides the internal gateway and policy layer.
OpenRouter provides external model and provider reach.
There is some functional overlap.
That is fine.
Software rarely observes our diagrams and stays politely inside the box we drew for it.
The architecture still helps explain why you might use each component.
From Smol to “Why Do You Have That in Your House?”
Let’s scale this from reasonable to questionable.
Smol
One model.
HOME ASSISTANT
│
▼
OLLAMA / LLAMA.CPP
│
▼
MODEL
Perfectly valid.
Stop here if this solves your problem.
Local Plus Cloud
Now you want escalation.
HOME ASSISTANT
│
▼
LITELLM
│
┌──┴───────────┐
▼ ▼
LOCAL MODEL OPENROUTER
Local work can remain local.
Cloud-capable routes can escalate.
Very useful.
Several Local Models
Now the local machine has more models than accelerator memory.
HOME ASSISTANT
│
▼
LITELLM
│
▼
LLAMA-SWAP
│
┌────┼────────┐
▼ ▼ ▼
FAST VISION REASONING
LlamaSwap manages residency.
LiteLLM exposes stable capability names.
Growing a Farm
Now there are several inference nodes.
HOME ASSISTANT
│
▼
LITELLM
│
┌────────────┼────────────┐
│ │ │
▼ ▼ ▼
FAST NODE HEAVY NODE OPENROUTER
│ │
▼ ▼
LLAMA-SWAP LLAMA-SWAP
Perhaps:
FAST NODE
interactive models
HEAVY NODE
reasoning / coding / larger models
Maybe another node handles speech or vision.
At this point you have a legitimate little inference farm.
Why Do You Have That in Your House?
Eventually:
HOME ASSISTANT
│
▼
HARNESS
│
▼
LITELLM
POLICY / GATEWAY
│
┌─────────────────────┼─────────────────────┐
│ │ │
▼ ▼ ▼
FAST INFERENCE HEAVY INFERENCE OPENROUTER
NODE NODE │
│ │ ▼
▼ ▼ CLOUD MODELS
LLAMA-SWAP LLAMA-SWAP
│ │
┌───┼────┐ ┌───┼──────┐
▼ ▼ ▼ ▼ ▼ ▼
SMALL VLM EMBED LARGE CODE MoE
LLM
Congratulations.
You have built an inference fabric.
Normal people may ask questions.
Remember, This Is a House
We have now reached the point where our diagrams are beginning to look suspiciously like data-center architecture.
That is useful.
It shows what is possible.
It is not automatically a recommendation.
A house has different priorities.
I generally care about:
- simplicity
- reliability
- low latency
- low power
- understandable failure modes
- low maintenance
If one inference server and one model handle your needs:
Stop.
If LiteLLM plus one local endpoint and OpenRouter handle your needs:
Stop there.
Add LlamaSwap when model residency becomes an actual problem.
Add inference nodes when workloads require them.
Add load balancing when you genuinely have loads to balance.
Every additional component means another:
- configuration
- dependency
- update
- log
- failure mode
- thing you may someday troubleshoot because somebody just wants the bedroom light off
There is absolutely nothing wrong with building the beautiful rack, matching the patch cables, adding RGB, and making the thing look like a small regional cloud provider.
If that brings you joy, fantastic.
It is not required.
My preference is:
the smallest, simplest collection of services that reliably does the job.
We are exploring the upper end because understanding the layers helps you make good choices.
You do not need to deploy every layer.
We are borrowing techniques from data centers. We are not trying to turn your home into one.
Simple is better.
Let the workload earn the complexity.
A Tiny LiteLLM Example
This is not an installation guide.
I just want to make the abstraction real.
Conceptually:
model_list:
- model_name: home-fast
litellm_params:
model: openai/local-fast
api_base: http://fast-node:8080/v1
- model_name: home-reasoning
litellm_params:
model: openai/local-heavy
api_base: http://heavy-node:8080/v1
- model_name: home-cloud
litellm_params:
model: openrouter/current-cloud-model
Home Assistant sees:
home-fast
home-reasoning
home-cloud
The infrastructure behind those names is free to change.
Use the current LiteLLM documentation when you actually configure it.
This field moves too quickly for me to tattoo YAML onto the article.
A Tiny LlamaSwap Example
The same principle applies locally.
Conceptually:
models:
fast:
cmd: >
llama-server
--model /models/fast.gguf
--port ${PORT}
reasoning:
cmd: >
llama-server
--model /models/reasoning.gguf
--port ${PORT}
The client requests a model.
LlamaSwap starts or swaps the appropriate backend.
Again, use the current project documentation for actual deployment.
We are learning architecture here.
Do I Need All This?
No.
Please do not install infrastructure because the diagram looked cool.
If you have:
ONE GPU
ONE MODEL
you probably do not need an AI routing fabric.
If you have:
ONE GPU
FIVE MODELS
model swapping becomes interesting.
If you have:
MULTIPLE NODES
MULTIPLE MODEL CLASSES
LOCAL AND CLOUD
DIFFERENT POLICY REQUIREMENTS
a gateway and routing layer can become extremely useful.
Complexity should earn its keep.
Load Balancing Is Yet Another Problem
Suppose you have two identical fast inference workers.
FAST A
FAST B
Now the question is:
Which equivalent worker should receive the request?
That is closer to load balancing.
A larger architecture might contain:
REQUEST
│
▼
ROUTER
Which capability?
│
▼
LOAD BALANCER
Which worker?
│
▼
INFERENCE
Most homes will never need this.
Some of you are already shopping for another GPU.
Fallbacks Need Thought
Fallbacks sound wonderful.
Local model dies?
Use another model.
Provider dies?
Use another provider.
Node crashes?
Send the work elsewhere.
Great.
Now consider:
PRIMARY
private-local
with:
FALLBACK
random-cloud-provider
You just changed the trust boundary.
So define fallback chains intentionally.
For example:
home-private
│
├── local-heavy-a
└── local-heavy-b
versus:
home-general
│
├── local-fast
├── local-heavy
└── OpenRouter
The route name now communicates something useful.
Routing Can Save Money Too
Cloud models vary significantly in cost.
So do jobs.
This:
“Turn off the office light.”
probably does not require the most expensive frontier reasoning model available.
Perhaps:
simple request
│
▼
fast inexpensive model
while:
hard reasoning
│
▼
frontier model
The expensive model becomes a specialist.
That can reduce cost and latency at the same time.
The Harness Can Help Route
Remember Chapter 3.
The harness often already knows something about the request.
It may know that this is a:
conversation request
camera request
background AI Task
developer task
private household query
That metadata may be enough to make a deterministic routing decision.
You do not necessarily need another LLM examining every request.
Use information you already have.
It is faster, cheaper, and easier to debug.
Sometimes Routing Becomes Orchestration
Consider:
“Research three replacement dishwashers, compare their warranties, and tell me whether they will fit the existing opening.”
That request may require:
- web research
- household measurements
- retrieval
- reasoning
- perhaps images
- several tools
Now we may have several jobs requiring different capabilities.
That starts becoming orchestration.
Remember:
ROUTER
Where should this job go?
ORCHESTRATOR
How should several jobs cooperate?
We will encounter that distinction again.
Build Stable Interfaces Around Unstable Models
This may be the most important lesson in this chapter.
Models change insanely fast.
Build:
HOME ASSISTANT
│
▼
home-fast
instead of:
HOME ASSISTANT
│
▼
SomeVendor-Model-27B-Q4-v17-FINAL-FINAL2
Today home-fast might be Qwen.
Tomorrow Ministral.
Next year something else.
The capability remains stable.
The implementation evolves underneath it.
That is exactly the sort of problem abstraction is supposed to solve.
So Which Model Should I Use?
As much as everyone hates this answer:
It depends on the job and what is good right now.
For Home Assistant, evaluate:
- instruction following
- tool calling
- structured output
- context handling
- reasoning
- latency
- multimodal capability
For online services, also evaluate:
- price
- reliability
- data handling
- provider policy
Use:
- Hugging Face model cards
- model documentation
- current inference-engine support
- community testing
- OpenRouter’s current model/provider information
- your own evaluations
And check again periodically.
Model choice is no longer a one-time decision.
The router lets the rest of your architecture survive that.
Now We Have a Different Problem
Excellent.
We selected a model.
The router knows where to send the request.
Our inference farm is happily converting electricity into heat.
There is only one small issue.
The model still does not magically know:
- your house
- your terminology
- your documents
- what happened yesterday
- why the bedroom is warm
- what “the good lamp” means
- what you told it six months ago
A brilliant model without the right information is still guessing.
Which means it is time to talk about:
context
memory
retrieval
RAG
ARAG
and, inevitably: