What Should Power the Intelligence in a Home That Behaves Well?

Share your Projects!
, , , ,
1

I’ve been thinking about a question that feels like a natural next step in the work I’ve been doing with Home Assistant and Homes That Behave Well:

What should actually power the intelligence in the Home?

Home Assistant is still at the center of everything I’m building. HTBW isn’t intended to replace it. It adds structure and governance around it, helping the Home understand things like rooms, people, devices, capabilities and the relationships between them instead of expecting every automation to figure those things out for itself.

I got another reminder of why that matters this weekend. I have routines that find the Sonos speakers assigned to a room while excluding the Sonos Beam used with the television. It worked until Music Assistant created another representation of the Beam that no longer matched the exclusion logic. Nothing was really broken with Sonos or Music Assistant. The fragile part was my automation trying to infer intent from the entities it discovered.

HTBW should eventually make that unnecessary. If I’ve explicitly told the Home which speaker is the general audio output for the Primary Bedroom and which one belongs to the television, an automation shouldn’t have to rediscover that relationship every time it runs.

But knowing the structure of the Home is only part of what I’m interested in.

Enter Home Steward

I recently started another project called Home Steward.

The distinction between the two is important to me. HTBW governs the Home. Home Steward reasons about it.

I don’t want to hand control of the house to an LLM. If I ask Home Steward why the bedroom shades didn’t open this morning, I want it to be able to understand the Home’s configuration, state, history and decisions and help explain what happened. That’s very different from giving an AI model unrestricted authority to operate the house.

And the opportunity goes well beyond traditional Home Assistant entities.

With explicit household authorization, Home Steward could eventually reason over documents on the NAS, equipment manuals, warranties, calendars, email, financial information, camera observations and other household knowledge. Each source should have its own governed connection and scope. Connecting one source shouldn’t give Home Steward permission to see everything else.

That’s also why local-first matters so much to me.

Local doesn’t have to mean running everything on the Home Assistant server. It means the processing remains inside a household-controlled boundary. I’m perfectly comfortable with a separate AI appliance sitting on the home network and doing the heavy lifting.

Which brings me to the question I’d like to put to the Home Assistant community.

What would you put in the house?

There are suddenly a lot of interesting choices.

Home Assistant can work with models running elsewhere on the local network, including through Ollama, so the AI compute doesn’t need to live on the Home Assistant server itself. At the other end of the spectrum, machines such as NVIDIA DGX Spark are now being designed specifically for local AI workloads and offer up to 128 GB of unified memory. Here’s one look at Home Assistant and Ollama, and NVIDIA’s DGX Spark shows just how far the other end of the local-compute spectrum has moved.

There are AMD Ryzen AI Max systems, conventional NVIDIA GPU machines, Macs with large amounts of unified memory, mini PCs and probably several options I haven’t considered.

But I’m not sure the answer is simply buy the biggest model I can afford.

One of my biggest requirements is perceived latency. When somebody speaks to the Home, it needs to respond naturally. I care less about a benchmark telling me how many tokens per second a machine can produce than I do about how long it takes from the end of the question to the beginning of a useful answer.

And Home Steward may eventually have several very different jobs: household reasoning, RAG over private documents, voice interaction, embeddings, vision from cameras, longer-running research and perhaps even development assistance. Current local-AI hardware guidance makes an important point here: those workloads don’t necessarily want the same hardware.

Maybe one machine should do all of that.

Maybe that’s completely the wrong approach.

I’m not the first person asking

One thing I’ve found encouraging is that people in the Home Assistant community are already experimenting with many pieces of this problem.

There are discussions about compact, low-power hardware for local LLMs, local inference and RAG, and even running useful local models on an inexpensive Intel N150 mini PC. Others are reporting what happens when they move to systems with dedicated GPU memory and begin caring about response time, context size, power and the number of entities being exposed to the model.

What I like about those conversations is that there doesn’t seem to be one obvious answer. A machine that can asynchronously prepare a morning summary is solving a different problem from one that needs to understand a spoken question, retrieve private household context, reason over it and begin answering quickly enough that the interaction still feels natural.

Home Assistant itself seems to be drawing a similar distinction. Its Building the AI-powered local smart home article describes AI as something people can selectively use for capabilities like summarization and image recognition while keeping it away from mission-critical operations. Its earlier AI agents for the smart home article is even more direct about the tension between probabilistic AI and deterministic home control.

That resonates with what I’m trying to do with HTBW and Home Steward.

Let AI reason. Let the governed Home decide what it’s allowed to do.

So, Home Assistant community, what am I missing?

This is the part where I’d really like to learn from people who are already doing this.

If you were building a local AI appliance for a Home Assistant-based home today, what would you build or buy?

What hardware have you actually deployed? What models have worked well? What does voice latency feel like in real use? Are you using Ollama, llama.cpp, vLLM or something else? Have you separated voice, vision, RAG and general reasoning onto different systems?

And I’m especially interested in the things that aren’t already on my list.

Maybe it’s an appliance. Maybe it’s an open-source project. Maybe somebody in the self-hosted or robotics communities has already solved part of this problem. Maybe there’s an architecture that makes more sense than the one I’m considering.

The Reference Home gives me a place to actually test these ideas rather than just draw diagrams around them. I’d like to take the candidates, put them into a real Home Assistant environment with real rooms, people, voice interactions, cameras, devices and household information, and document what actually happens.
Not just, here’s the AI box I bought.

I’d rather be able to say: here were the requirements, here’s what we tried, here’s what worked, here’s what failed, and here’s what surprised us.

So before I buy the box, I thought I’d ask.

What should be on the list?

Thanks in advance!
Tom.

What I’m Reading

If you’re interested in the same questions, here are a few of the conversations informing my thinking:

2

Hi Tom

A well articulated post and I am glad that someone is thinking along similar lines to myself.

There is no doubt that to achieve this is a lot of time and money on hardware. I have invested and upgraded from my faithful NUC running HA for about seven years. I went to an AMD HX470 based mini PC. I installed Proxmox and setup HA as a VM running HAOS. This was a design choice because I want to spend minimal time on managing the HA machinations. I would rather spend time on building an autonomous home.

I added two LXC’s one for Lemonade Server and one for Frigate.

The Lemonade Server runs two LLM’s. A Qwen based LLM on Llama.cpp running on the iGPU. This is dedicated to all things home. The second LLM is smaller and runs on the NPI for Frigate. I also have a Hailo8 for object detection.

The first thing I built is called Abode AI Tools. It is the interface for voice pipeline. I found intents less than adequate (sorry guys) and when it used the LLM backend it sent everything to the LLM, it was thousands of tokens of information on every turn. So Abode AI tools does just the right amount and voice responses beyond intents are less than 3s response.

The Second thing I built is called Abode AI Autonomy Steward. It is a middleware layer and yet incomplete. However, It is designed to monitor everything happening in the home and provide advice and information about the home mainly via voice to the people in the home. This is yet unfinished as I realised that an Autonomous house does not necessarily mean a house run by AI. Some systems can be completely autonomous as they can be modeled and once modeled don’t need an AI to control it.

I am well down the track of an autonomous HVAC system. Its foundation is based on the latest research but the idea is based on comfort control and split system airconditioners not ducted. Unlike traditional HVAC systems it does not control temperature, it controls comfort levels which are set by the user. I won’t go into the maths but it is based around temperature and humidity, looks at the outside environment as well as the room and power usage. It learns the rooms thermal properties over a period of time and responds as needed to maintain a comfortable environment while being as power efficient as possible.

Along side this is a power tariffs module that is a system of truth for what power costs at any point in time and a forecast ahead.

The HVAC component is a fantastic example of an autonomous system that does not require AI to run.

In the end AI is the assistant to the home, it provides monitoring and assists the residents in providing the best environment for them to live in not running the home. It is presence aware therefore knows where people are when they ask for assistance.

I am also building local assistant/voice satellites with sensor inbuilt, some with displays to assist with this journey. I found the voice PE and associated devices lacking in both sound and voice communication so have taken a different route in using XMOS XVF3800 chips teamed with an ESP-P4 and C6 chips with sensors and a TAS5805M and a 25W speaker running over POE. One has a display and is desktop and the other does not have a display and and is ceiling mounted.

Everything is always local only.

Just my two cents worth.

3

Thank you. This is remarkably close to several conclusions I’ve been reaching, particularly that an autonomous Home is not the same thing as an AI-controlled Home.Your HVAC example is exactly the distinction I think matters: where behavior can be modeled reliably, it should be deterministic. The role I’m exploring for AI through Home Steward is much closer to what you describe with your Autonomy Steward: understanding what is happening across the Home, recognizing what may matter, helping residents understand it, and leaving deterministic Home operation where it belongs.I’m particularly interested in your Abode AI Tools experience. The observation that sending the entire Home to the LLM on every interaction creates thousands of unnecessary tokens reinforces something we’re exploring around providing only the minimum governed context needed for the question.We’re also independently experimenting with XVF3800-based voice endpoints, although I’ve intentionally separated each room’s microphone/input relationship from its speaker/output relationship so the Home can govern both independently.I would love to compare notes as both projects develop. The work is public here: [Home Steward repo].I suspect we’ve arrived at many of the same questions from somewhat different directions.

4

Really interesting thread, and the HVAC example is the one that lands for me as a homeowner. My heating and cooling runs on plain schedules, setpoints, and a couple of room sensors — that part has to be deterministic, because the house being warm when I wake up is not something I want an AI to decide on the fly.

Where I actually want the smarts is the layer on top: telling me why the upstairs bedroom runs cold at night, or suggesting a setpoint tweak before a cold snap, and letting me approve it. Voice is the same — I want the lights to come on fast even if the local AI box is busy or down.

So my rule of thumb has become: the house must keep working with the AI switched off, and the AI earns its place by explaining and suggesting, not by being in the loop for everyday stuff. Curious how others here draw that line for their own setups.

5

I am in the final stages of testing the component ready for a release candidate. What I haven’t tested as yet is multi-head systems or two separate systems working together in the same large room. you are welcome to have a look at it. am happy for feedback and any bugs or issues please log them in github. There is comprehensive documentation explaining all the maths and control theory if you are interested and the experience after a few months will be a lot smoother then the first few days as it learns and models the rooms characteristics.

https://github.com/meremortals70/abode-hvac-coordinator

There is a sister application I posted on here a couple of months ago this can provide information on energy costs to the hvac coordinator but is not required.

I am in the process of a final code review on the power tariffs component.

6

I think we are on similar lines. the approach of has taken is a middleware layer between ha and the llm. I am still in early stages of it and wow, your documentation is next level I have spent a lot of the day reading :slight_smile:

My AI-tools has been running for a few months in my home and I am happy with it, just did some updates to code efficiency and bringing it into line with the ha 2026.10.0 changes for llm based tools over the weekend.

7

Thanks for the kind words about the documentation. That discipline probably comes from spending roughly 40 years working around enterprise systems and architecture. I’ve learned to “begin with the end in mind” and document the outcome before getting too attached to the technology.

I’m not actually developing Home Steward yet. I’m focused on HTBW, and that’s giving me the luxury of letting Home Steward develop as an idea before it develops as software. As I work through home control, identity, context, explainability and governance, I keep discovering things that help define what Home Steward should and shouldn’t eventually be. Rather than letting those thoughts disappear, I capture them as scenarios, outcomes, requirements and guardrails.

I also inventoried my Reference Home and asked a deceptively simple question: “Given everything I’m already telling Home Assistant, what additional value could I get from that information?” Much of the Home Steward scenario library grew from that exercise.

I’m conscious of the bias that creates. Naturally, I think most deeply about systems and integrations I actually live with. Home Assistant development documentation and some of my Asset Intelligence work help broaden that, but there are undoubtedly valuable use cases I haven’t encountered because they are not currently part of my home. That’s one reason I’m enjoying conversations like this it helps me explore the ‘art of the possible’.

And yes, I think there’s a real similarity to your middleware architecture. That’s intentionally where the eventual Home Steward integration sits: between Home Assistant and the LLM. I’m just trying to spend a lot of time defining the outcomes and boundaries before I start writing the software.

8

Mike, I keep coming back to your examples because they’re remarkably close to the boundary I’ve been trying to define.

I completely agree that “turn on the lights” shouldn’t require an LLM. That’s actually one of the problems I’m working on with Homes That Behave Well. I want the Home to understand the room, the person, the context and the capabilities available there well enough that ordinary interactions remain fast and deterministic. If I unplug the AI box, the Home shouldn’t suddenly become dumb.

The cold-bedroom example is where I think this gets really interesting. I’m treating “Why was the bedroom cold last night?” as an explainability problem. The Home should be able to show what it knew, what happened and why.

But then I want to be able to ask, “So what should I do about it?” That’s where I see Home Steward eventually coming in. It could reason across the information I’ve authorized it to use and look for relationships that aren’t obvious when I’m staring at individual Home Assistant entities: weather, room temperatures, occupancy, HVAC performance, energy consumption, equipment documentation, historical behavior and whatever other relevant evidence is available.

And I especially agree with your point about letting the human approve the recommendation. The answer doesn’t have to be “I’ll change the thermostat.” It could be, “Here are the three things I think could be contributing to this, here’s the evidence for each, and here’s what I’d investigate or change first.” I still get to make the decision. If I approve an action, then the Home can carry it out through the same governed control mechanisms it already uses.

That human-in-the-middle step is important to me. More intelligence shouldn’t automatically mean more authority. Home Steward may eventually be able to know and reason about considerably more of the household, but that doesn’t inherently give it permission to act.

So I think your rule is a really good one: the Home must keep working with AI switched off. I’m increasingly thinking my companion rule is: AI earns its place by helping the household understand, discover and improve, not by becoming a dependency for everyday control or quietly taking control away from the people who live there.

9

Tom

This is where I have also struggled, the boundary between a neural network for the home and AI. I am still learning towards llm based for the reasoning but as demonstrated with the HVAC component and as you mentioned yesterday. If a process can be modelled there isn’t a need for an llm to run that process autonomously. as for sifting through all the data to explain why it made that choice I still think the llm is the best way to do that. However, I am still looking to implement some sort of memory graph to assist the llm in learning the home.

10

Jason,

I think we’re getting very close to the same boundary. The part that has helped me resolve the “how does AI explain the autonomous system?” question is the Explainability work we’ve done in HTBW.

Rather than asking the LLM to sift through every sensor state, log entry and entity after the fact and infer why a deterministic system made a decision, HTBW creates and logs a Decision Trace around the decision itself: what triggered the evaluation, which governed facts mattered, what constraints applied, what decision was reached, what action or non-action followed, and what outcome was observed. In HTBW this serves as a quick way to get an answer to ‘why did x happen’ or ‘why didn’t x happen’ and it is built at the time an even occurs. The same principle can apply whether the governed decision originated in an automation, a script, or another deterministic Home behavior.

As a byproduct, it gives the LLM a much smaller and much more trustworthy input. Its job becomes explaining and synthesizing the evidence rather than reconstructing causation from thousands of raw Home Assistant states.

If I ask, “Why did the HVAC start cooling?”, HTBW should already know why. The LLM can turn that trace into a natural explanation. If I then ask, “Did it work?”, “What did it cost?”, or “Has this been happening more often?”, that’s where the AI reasoning layer becomes increasingly valuable across multiple traces and other authorized evidence.

I think a graph still becomes useful, but I’d separate the governed Home graph from AI memory. Rooms, Assets, People, capabilities, Stewardship relationships and current Truth shouldn’t have to be inferred by the LLM. The Home should already know those things. The AI can then reason over that governed structure and its authorized extensions.

Your point about limiting the context sent to the model really resonates with this. A Decision Trace plus the relevant facts may give us a far better answer, with lower latency and fewer tokens, than dumping the entire Home into the model on every voice turn.

This is the piece I was referring to when I said I don’t think the LLM should have to infer why the Home acted from thousands of raw state changes.

Explainability Architecture

This describes the broader Explainability model: how the Home can answer questions about what it did, didn’t do, or understood, using governed evidence rather than inventing a plausible explanation after the fact.

Decision Trace Model

This is the important part for the deterministic systems we’re discussing. A Decision Trace captures the relevant evidence around a decision when it occurs: what triggered the evaluation, the facts and constraints that mattered, the decision reached, the resulting action or non-action, and the available outcome evidence.

An explanation doesn’t have to be pre-written for every possible future question. If someone later asks something we didn’t anticipate, Explainability can assemble an answer from the retained Decision Trace and other governed evidence. The distinction I’m trying to preserve is that the reason for the original decision isn’t being invented later by the LLM.

Explainability Persistence and Retention ADR

This gets into the temporal problem. Explainability needs evidence, but keeping every decision and every supporting fact forever creates its own privacy problem. We treat retention as both a floor and a ceiling. If evidence has aged out, the Home should say that rather than quietly filling the gap.

Historical Explainability and Historical Truth ADR

This is optional deeper reading, but it’s relevant to your memory-graph question because it distinguishes:

what the Home contemporaneously knew and recorded

from:

what can still be reconstructed later from retained evidence.

I know this is getting really deep in the weeds, but I feel that making Explainability a core requirement of HTBW naturally creates a much cleaner input for Home Steward. Rather than throwing all of the Home’s raw data at the LLM and risking a “forest for the trees” result, we can give it the relevant governed evidence and let the model do what it’s good at: explain, synthesize, compare, and reason across it.

This is still theory until I finish more of HTBW and start testing it with local models, but based on what you’re building and what I’ve been architecting, it feels like we’ve arrived at many of the same conclusions from different directions.

11

great point. Looking more at what you have it has convinced me more and more we are looking at mainly the same thing but I am taking a federated approach, not saying it is better than your unity approach where you use HA for everything. I personally thing HA is great for what is at the moment and history of the environment but not the best place for identity, assets, documents etc so here is something I have worked on to try and explain the two approaches to achieve the same aim.

htbw-vs-abode-topology
htbw-vs-abode-topology1600×1400 213 KB

12

Jason, I really like this, and I think your diagram has helped me see that we’re probably even closer than “unitary versus federated” makes it appear.

A little history may help explain why my side ended up looking the way it does.

I didn’t start with the premise that Home Assistant should own everything. I started several years ago with a much simpler question: what does it mean for a Home to behave well?

That eventually became the Behaves Well Framework, which is the governance model underneath HTBW. The residential work then became one of the places where I could pressure-test that Framework against a real lived Home.

As I worked through that, I kept finding responsibilities that a conventional smart home doesn’t really model.

Identity was one of the first big ones. Home Assistant is very good at knowing that an entity changed state, but that isn’t the same thing as sufficiently resolving which person is interacting with the Home. That becomes particularly important with voice. I couldn’t find a meaningful way to say, for example, “Tom may ask for this, David may ask for that, and an unresolved speaker may not receive either,” so Identity and person-scoped authorization became explicit parts of HTBW.

Then came Assets. I realized the Home contained many things that mattered but weren’t smart devices at all: paintings, furnishings, appliances, collectibles, documents, warranties, family possessions and eventually anything the household deliberately declares worth knowing or caring about. I created a different Integration called ‘Asset Intelligence’ who’s functionality is being pulled into HTBW.

That led into ownership, location, provenance, manuals, insurance, maintenance cycles, custody, environmental requirements and the supporting documents around those Assets.

From there Stewardship became unavoidable. Once the Home knows what something is and what it requires, there is a separate question: what must be cared for, by whom, and how do we know the obligation was actually fulfilled?

Explainability followed for much the same reason. If the Home is going to make increasingly sophisticated deterministic decisions, I don’t think “the automation fired” is a sufficient explanation. The household should be able to ask why something happened, why it didn’t happen, what evidence mattered, what constraint won, and later whether the decision actually produced the intended outcome.

That was roughly the original finite scope: govern the Home, its people and its Assets; let deterministic Home Assistant behavior do what can be modeled; preserve enough evidence to explain it; and keep authority explicit.

Home Steward actually came later.

Once we had all of that governed structure, I started asking what became possible if a private local intelligence could reason over it, along with separately authorized household documents and external knowledge.

That’s where things exploded.

Suddenly the questions weren’t only:

“Why didn’t the shades close?”

but:

“Has this been happening more often?”
“When did it start?”
“What changed around then?”
“Walk me through running the cleaning cycle on the Miele coffee service?”
“Did the fix actually work?”
“What is this Home costing us to operate?”
“What do these room conditions mean for this particular painting?”
“Has anything changed from this person’s normal pattern?”
“Has Home Assistant added something that means HTBW no longer needs to own this implementation?”

That’s really why I’ve resisted letting the LLM become the Home’s model or authority. I want the Home to know what it knows independently. Then the AI can do the things AI is particularly good at: retrieval, synthesis, comparison, pattern recognition, investigation, explanation and advisory.

And I think this is where your federated approach and mine may actually converge.

I’m completely comfortable with UniFi remaining authoritative for what UniFi does, Music Assistant for what Music Assistant does, an external document store retaining the document bytes, and a private-AI appliance running inference outside Home Assistant. I’m not trying to recreate all of those systems inside HA.

The distinction I’m making is that HTBW governs what those things mean to the Home and how they relate to one another.

So the NAS can own the PDF bytes, but HTBW can know that the PDF is a conservation report for a particular painting. UniFi can own the camera evidence, but HTBW can know that a camera participates in a particular Room. A private-AI platform can own inference, without becoming the source of Truth or action authority.

I might describe it less as unitary versus federated and more as:

federated specialist systems with a governed household semantic model.

Your diagram has actually helped me see that distinction more clearly.

The Framework is here if you want the origin story behind the architecture:

https://behaveswellframework.com/

And the Home Steward scenario work is increasingly becoming the other end of that journey: we preserve the household experiences first, then let recurring requirements tell us what architecture is actually justified.

I think we’re attacking remarkably similar problems from different directions, which is why this exchange has been so useful.

Here is the full framework image

HTBW-CORE-003-Framework_36x24_200DPI
HTBW-CORE-003-Framework_36x24_200DPI1920×1280 768 KB

I haven’t updated this diagram yet to show Home Steward because I don’t see Home Steward as another Framework responsibility alongside Concierge. It’s an intelligence capability that can reason across the governed Home these responsibilities collectively describe.

What became interesting was realizing that the same private intelligence could also reason across separately authorized information outside the native Home model: household documents, owner’s manuals, utility bills, financial information, email, calendars, media, external knowledge and other personal datasets.

The important distinction I’m trying to preserve is that access to that additional knowledge doesn’t give the LLM authority over the Home. Identity helps establish who is involved, while Operational Trust, source authorization and the other Framework responsibilities continue to govern what information may be used, disclosed or ultimately acted upon.

That’s really how Home Steward emerged. It wasn’t another layer I set out to add to the Framework. Once the governed Home existed, it became the natural question: what could private intelligence help the household understand if it could reason across all of this without becoming the authority over it?

13

I’ve been reading through your Abode AI Tools repo. This is really helpful because it gives me something concrete behind the context-minimization point you made.

I’m particularly interested in eventually comparing what your bounded tool layer needs to provide the LLM from raw Home Assistant versus what happens when HTBW can hand it governed context and a Decision Trace directly.

If the same question can be answered with substantially less context, lower latency, and a smaller local model because the Home has already captured the meaning of the event, that would be really useful evidence for both of us.

I’m going to spend some time reading through what you’ve built rather than speculate from the diagram.

14

Tom I am loving this conversation and feeding off it. To day I have uploaded my architecture and decisions so far on steward as I mentioned this is a work in progress and I agree with you, what knows the house for me it is the steward the middleware layer that sits between ha and my other systems of truth. I have left the llm to do what it is good at as well. I guess I have taken it slightly in a different direction as when the llm recognises a pattern I tells me about it as a notification and I can approve its suggestion or deny. either way it goes into the steward, not the llm, as a rule or constraint. if it is a rule then the steward together with the llm can continue to operate that rule without my intervention. If it is a constraint then it will not and the llm will only ask me again if it has new compelling evidence. so it is the steward governing the home not the llm.

I am also getting my head around your voice identity. I had thought about a similar approach but had not yet resolved how I did a voice print.

and yes I am working my way through your youtube.

15

Jason, I found the new AI Autonomy Steward repo, and this makes your architecture much clearer. The Approve / Deny / Never model and especially the append-only constraint layer are really interesting. I like the property that the running system can learn a new restriction but can’t silently remove one.

One thing jumped out at me, though, because of the point you made earlier about reducing LLM context for voice. The Steward README says that every event wakes the App and it pulls the complete current Home Assistant state fresh, the whole picture every time.

Do you expect that to remain the long-term design, or is that mainly where the current implementation is while you’re building out the domain specialists?

That’s exactly the area I’m approaching differently with HTBW. My hope is that Foundation, Truth and Decision Trace can give the reasoner a much smaller semantic package representing what changed, what it means, and which governed evidence is actually relevant, rather than making the reasoner rediscover significance from the whole HA state on each pass.

Your voice-context experience makes me wonder whether we may eventually converge there too.

I’m also very interested in the append-only constraint idea. Our closest analogue is Operational Trust, but I’ve intentionally avoided using a single model confidence score as authority. I’d enjoy comparing those two approaches as the implementations mature.

And on Voice Identity, you’re right that I ended up treating the voiceprint as evidence of Person, not Person identity itself. That distinction turned out to be important.

On the voice identity piece, there may be another part of the journey you’d find interesting because the architecture came directly out of an implementation failure.

We started with a pretty simple idea: enroll a person with voice samples, build a Voiceprint, and let HTBW use the voice match to help identify who is speaking.

The important distinction we made early was:

a Voiceprint match is evidence of identity. It is not identity.

So the flow isn’t:

voice → match → “this is Tom.”

It’s closer to:

voice utterance → Voiceprint comparison → Voice Evidence → HTBW Identity → combine with other available evidence → Person resolution

The voice system isn’t allowed to decide who the Person is.

For example, HTBW Identity might eventually have several pieces of evidence available around an interaction:

  • which room microphone received the request;
  • presence evidence for that room;
  • device/BLE evidence associated with household People;
  • interaction/session context;
  • Voice Evidence.

Identity then decides whether the combined evidence is sufficient to resolve a Person for the particular purpose. A strong voice match can be important evidence, but it isn’t allowed to bypass the rest of Identity or become authentication by itself.

We also deliberately compare an incoming utterance against all enrolled Voiceprints rather than asking the voice system to determine which Person is speaking. The voice component returns the complete comparison result using governed ordinal match levels rather than handing Identity a raw model score and saying “this is Tom.”

The implementation journey is where this got interesting.

Our first attempt was to put the speaker encoder directly into the Home Assistant Integration. That failed for a useful reason: the ML runtime/dependencies needed for the encoder simply weren’t a good fit inside the HA Integration boundary.

Rather than fight Home Assistant, we changed the architecture.

Voice Evidence is now a separate Home Assistant App (as you called out in the diagram you produced comparing our approaches) with a deliberately narrow responsibility:

receive voice sample → encode it → compare it against every eligible Voiceprint → return the complete Voice Evidence

The App does not:

  • resolve the Person;
  • use Home context;
  • consider presence;
  • decide authorization;
  • decide whether an action is permitted;
  • stop after finding the first plausible match.

HTBW Identity does the identity reasoning.

That separation also gave us a much cleaner lifecycle.

Enrollment captures ten samples for a Person and passes those samples, with the stable Person identifier, to the Voice Evidence App. The App owns the Voiceprint lifecycle after receipt.

One especially useful consequence is encoder migration. A Voiceprint is derived evidence created by a particular encoder. If we later replace the encoder, an old Voiceprint shouldn’t quietly remain authoritative as though nothing changed.

Where the household has chosen to retain the enrollment recordings, the App can rebuild the Voiceprint using the new encoder. That gives us a path to auto-heal derived Voice Evidence after an encoder change without asking everyone in the household to re-enroll unnecessarily.

If the household chose not to retain those recordings, then obviously we can’t reconstruct something we deliberately no longer possess.

That’s another recurring pattern I’ve run into:

retain the governed source when appropriate → treat ML output as derived evidence → make the derived artifact reconstructable when possible.

I suspect this may relate to your Steward architecture more broadly. The ML component can be very good at a narrow specialist task without gaining the authority to interpret what that result means for the Home.

In our case:

Voice Evidence App asks:
“How strongly does this utterance compare with each enrolled Voiceprint?”

Identity asks:
“Given Voice Evidence and the other governed evidence available right now, do I know sufficiently who this Person is?”

Operational Trust asks later:
“Given who is sufficiently resolved, is this particular disclosure or action permitted?”

Those are deliberately three different questions.

And honestly, we didn’t arrive at the App boundary because the original architecture looked prettier. We arrived there because trying to embed the encoder in the Integration didn’t hold up against the reality of Home Assistant. The failure made the architecture better.

Before I turn this into another page of architecture discussion, I just want to say how much I’m enjoying the conversation.

This has always been one of my favorite parts of working with good teams. Somebody puts a thesis on the table. We explain why we believe it. Somebody else challenges it, comes at the same outcome from a different direction, and forces us to look again at our own assumptions. If we’re doing it right, nobody has to “win” the debate. We both come out of it better educated.

That’s what I’m finding here. Some of the places where we’ve independently arrived at almost the same conclusion are fascinating. But honestly, the places where we’ve made different architectural choices may be even more valuable to me, because they’re making me ask whether I’ve chosen something because it’s actually the right boundary or simply because it’s the way I’ve been thinking about the problem.

I also really appreciate finding someone else who genuinely thinks in frameworks. The technology is interesting, but the framework is what lets you reason about why something belongs where it does, test that assumption against implementation, and change course when the evidence disagrees.

That’s very much how I’ve tried to approach BehavesWell and HTBW, and I can see the same discipline in what you’re building.

So genuinely, thank you for the exchange and for opening up your work. This is exactly the kind of conversation I hoped would happen when I started putting all of this work into the public domain.

This is exactly the kind of exchange I love, and hopefully it gets us closer to what started this whole conversation: a private AI appliance that doesn’t bankrupt the household and is repeatable beyond our own homes.

16

I too am loving the teamwork. you are correct my architecture document was lacking with information of where I am heading. If you look at the architecture.md in the documents folder it has more info and some diagrams which explains the vigil layer. this will help answer your question regarding waking the steward

17

Jason, yes, this answers the question. Vigil is the piece I was missing from the README. I especially like the distinction between raw-state wake signals and derived state, and the separation of continuous awareness from expensive deliberation. That’s much closer to where I’m heading than I realized. I want to think harder about where our approaches differ on what meaning should exist before the salience judgment, because I suspect that’s where the interesting comparison now is.

18

I am just updating the architecture again as I realised there was a need for another domain that caught anything that was modeled in the other domains. I haven’t looked at this for a few months as I was concentrating on other projects but this discussion has my thinking cap back on and refining my architecture

19

Something I do when I get this far down in the weeds is take a few HUGE steps backward and ask myself, “What am I not asking?” These are the two that came to mind for me tonight (it’s almost 8 PM here in Dallas):

1. We’ve both spent a lot of effort explaining decisions, but how do we eventually explain what the system failed to notice? If Vigil decides something isn’t salient and Steward never sees it, what evidence lets us later understand a false negative? I think Home Steward has the same unanswered problem.

And:

2. We’ve both separated authority from reasoning, but what happens when two appropriately authorized household members want incompatible outcomes? Neither person is wrong and neither lacks authority. At that point, we’re no longer solving authorization. We’re arbitrating legitimate competing intent.

20

Guys.

Im going to offer some constructive criticism as I see you both appear to be falling into a trap.

Im the creator of Zenos - Friday’s Party, have your agents go read it, Friday's Party: Creating a Private, Agentic AI using Voice Assistant tools
I cover most of the tenets you are circling, autonomy, safety, limits, including testing in actual stress test environments in an environment actually running in production in multiple households. If it’s ‘hard to read’ fair, it was not originally a how to - there’s links to an architectural guide in the supporting git and I even cover some of these there too. Your agents should be able to consume them.

Collaboration is awesome. Keep that part. But I don’t see your agents challenging your assumptions well. I see a lot of ideas and I see your agents agreeing with each other about the decisions but…

I also see a lot of explaining away the rough edges (where I think you’re going to face plant) without actually solving the identified problems. You have an increasingly complex spec that I can’t tell if it works or not because I don’t see any examples.

I would suggest asking your agents how to simplify your specifications and reduce them to what we in the business call an ‘MVP’ (no, not like Jordan, he is the GOAT, fight me)

MVP == Minimally Viable Product.

The absolute minimum feature set you can build and actually call it a success. Define the smallest atomic piece you can actually work on and produce a product.

Then once you have a working prototype there test heavily and iterate. Code agents do this really well when given good simple requirements if you have a good definition of done. Done does not include designing the entire enchilada on one shot.

If you do not, of your requirements are too broad or lacking technical specifications, the code agent will code the equivalent of ‘rambling’ and you never get to goal. You may be building the world before the agents even can wake up.

I don’t want to squash any of your ideas at all but I will ask both if you’re challenging the assumptions of your agents enough and that error may lead you into something you can’t recover easily. (it’s the failure mode of more than a lot of AI projects I work on in my day job and I don’t like to see folks fail.)

Now. What should a well behaved home do? Everything. But. Not a damn thing I didn’t explicitly allow… :smiling_face_with_sunglasses: