Home Mind — Conversation Agent with Persistent Semantic Memory
TL;DR: A self-hosted HA conversation agent that remembers context across sessions using semantic search and embeddings. Two Docker containers, Claude Haiku API, HACS-installable, AGPL-3.0. Daily-driving for 2+ weeks.
The Problem
Every AI conversation agent for HA is stateless. You tell it your NOx baseline is 100 ppm, it acknowledges — and the next conversation it tells you 100 ppm “appears elevated” again. The community workarounds (todo lists as memory, template sensors) are keyword-based and don’t scale.
What It Looks Like in Practice
Session 1 — teaching a baseline:
User: What's the NOx sensor reading?
Claude: [uses get_state tool] The NOx sensor shows 100 ppm.
User: Is that high?
Claude: Typically, indoor NOx levels are 20-40 ppm, so 100 seems elevated...
User: Actually, 100 is completely normal for my home. Please remember that.
Claude: I understand - I'll remember that 100 ppm is within normal parameters
for your home's NOx levels.
Session 2 — new conversation, no reminders:
User: How's the air quality?
Claude: [checks NOx sensor] The NOx sensor shows 98 ppm, which is within
normal parameters for your home.
Note that the user asked about “air quality” — not “NOx.” The semantic search matched the intent to the stored fact.
Session 3 — anomaly detection using the learned baseline:
User: The temperature sensor shows 25°C - is that unusual?
Claude: Yes, 25°C is very unusual for your living room! Based on what I know
about your home, your normal morning temperature range is 20-21°C.
This is 4-5°C above your baseline, which suggests:
• Heating system issue
• Window left open with sun exposure
• Unusual heat source
Would you like me to check the temperature history to see when
this spike started?
These are real interactions from my setup — not fabricated demos.
What Home Mind Does
Home Mind is a conversation agent that registers with HA Assist and maintains persistent memory across sessions. It works with voice satellites, the companion app, and anything that uses conversation.process.
Memory is semantic, not keyword-based. Facts are stored as embeddings in a vector database. When you ask about “air quality,” it retrieves your NOx baseline — even though you never used the phrase “air quality” when teaching it. This is the core difference from keyword/regex approaches.
Architecture
HA Assist (Voice/Text)
→ HA Custom Component (Python, conversation agent)
→ Home Mind Server (Express/TypeScript, port 3100)
→ Claude API (Haiku 4.5 for conversation + tool calls)
→ Shodh Memory (semantic search, embeddings, port 3030)
→ HA REST API (entity states, service calls)
Two Docker containers:
- home-mind-server — Express/TypeScript API that handles conversation, tool execution, and fact extraction
- shodh — Memory backend with embedding generation, semantic search, and Hebbian learning
Request Flow
- User message arrives via HA Assist → custom component → server
- Server queries Shodh for relevant facts (semantic similarity to current message)
- System prompt is built: static instructions (prompt-cached) + dynamic context (retrieved facts + current datetime)
- Claude processes the message with 5 HA tools available:
get_state,get_entities,search_entities,call_service,get_history - Response streams back to HA
- In the background (fire-and-forget): a second Claude call (Haiku 3.5) extracts facts from the conversation and stores them in Shodh, automatically replacing conflicting older facts
Memory Model
Facts are categorized: baseline, preference, identity, device, pattern, correction. Each maps to a Shodh memory type with different decay characteristics.
Key behaviors:
- Hebbian learning — memories strengthen each time they’re accessed
- Natural decay — unused facts fade over time; daily-use facts persist indefinitely
- Smart replacement — the extractor identifies when a new fact supersedes an old one (e.g., “bedtime is 10:30pm” replaces “bedtime is 11pm”)
- Passive extraction — no need to say “remember this”; facts are extracted from natural conversation
What It Costs
Requires an Anthropic API key (Claude Haiku). In 2+ weeks of daily use, cost has been negligible — single-digit cents per day for a household. Memory storage and semantic search are fully local; the only external call is to Claude’s API.
Privacy
Conversation text and entity names are sent to Claude’s API for processing. Device states are fetched locally via HA’s REST API and only included in the prompt when relevant. All memory storage and semantic search runs on your hardware.
Hardware
Running on a Proxmox VM with an Intel i5-9500T. Resource usage is minimal — Shodh uses a local embedding model but it’s lightweight. No GPU needed.
Installation
Available via HACS (custom repository) or manual install. Docker Compose for the server side.
Repo + full installation guide: GitHub - hoornet/home-mind: AI assistant for Home Assistant with cognitive memory - Adds learning capabilities, persistent memory, and voice control to your smart home
Current State
v0.6.0. Single-user only for now (multi-user via OIDC is planned). Conversation history is in-memory (lost on restart, but long-term facts persist in Shodh). Been running stable as my primary conversation agent for over 2 weeks.
This is my first public project. Looking for feedback and early testers — especially from people running voice satellites or multi-room setups. Issues and PRs welcome.
