Friday's Party: Creating a Private, Agentic AI using Voice Assistant tools

NathanCu · June 4, 2026, 3:07am

Catch up time!

If you're new here, you may be wondering

what exactly the hell is going on?

That's fair. TOTALLY,

This started simple: could I build something that feels less like a chatbot and more like a member of the household? Also teaching tool - Story WAAAAY up there - feel free.

Most systems know facts. People live in context.

Now - If I tell you we're having friends over Saturday night, you immediately start thinking about food, music, food, shopping, guests, more food, schedules, "where's my wok?", and if you're my wife, whether the house is ready. Nobody explains how those things connect. They just... DO.

Current assistants don't really work that way. They answer questions and commands. In fairness, OK some of the commercial ones based on models can get a little farther than that if you're willing to hand over a frankly alarming amount of information about your life. Where you are. When you're home. What you buy. How you use electricity. What you cook. Who visits. What your routines look like.

YES, you absolutely CAN give control of your HA install to a loop agent (any openclaw hermes agentzero, paperclip WTHever...) points HALMark is over there - yes, I'm about to update it for 2026.6.0 please use it - and I hope you are very good at Devops. smiles Enjoy your dragons.

(Admittedly FUN dragons, but if you don't know what you're doing. YOU CAN get hurt...That's as far as I take that. I also ride motorcycles. Risks exist.)

This thread is what happens when you decide to keep that information, build the plumbing yourself, expose to it as many deep connections in your data - as safely as possible (in a way you KNOW it works) and see how far down the rabbit hole goes. SO.

If you'd like to wander into the dunes with the rest of us:

Zenos Base Repo: https://github.com/nathan-curtis/zenos-ai
Getting Started: https://github.com/nathan-curtis/zenos-ai/blob/08113cf99b07ed6739db0ea9991be8507783a2f5/zenos_ai/docs/getting_started/readme.md
All Your Docs Belong to Us: https://github.com/nathan-curtis/zenos-ai/blob/08113cf99b07ed6739db0ea9991be8507783a2f5/zenos_ai/docs/readme.md

Current as of this post: Zenos: 2026.6.0 / Cortex: 42.1 HA: 2026.6.0

Before diving into the latest madness, a few thread landmarks are worth a look:

Post #1: The Front Door
What Friday is, what "agentic" means in this build, and why tools plus context are the whole game.

Post #15: Poetry in Motion
The prompt-craft thesis. Intent, urgency, mood, dense language, and why "brief" often fails when the model needs a root concept.

Post #34: Ninjas
The first real look at summarization, Kung Fu, Katas, and attention management.

Post #42: Ninja 2, Electric Boogaloo
The scheduler and context-selection leap. Early autonomy scaffolding.

Post #49: Limits, Gov'na
Context budgets, aliases, tool descriptions, entity exposure, and why "just expose everything" eventually falls apart.

Post #55: Meet Kronk / The Monastery
Friday, Kronk, expert workers, queued jobs, specialist models, and one of the biggest architectural pivots in the project.

Post #120: Storing Elephants in Drawers
Cabinets, drawers, pointers, inherited context, and memory as infrastructure.

Post #234: Sand Dune Plinko
Grounding, hallucination control, directives, identity lanes, Katas, and making truth the easy path.

Post #261: Hyper-huh?
Why Home Assistant stops looking like a giant pile of entities and starts looking like a connected world.

Post #456: Stop Putting SOPs in Your Cortex
The modern ZenOS pattern: label it, write a KFC, store it in the right cabinet, index it, expand it when needed.

Fair warning: there are elephants in here. Some are obvious. Some are hiding behind the architecture diagrams.

The current focus is the kitchen.

Before Friday can help run a household, she should probably be able to help run dinner.

Everything else in this thread is what happens when that idea collides with Home Assistant, too much coffee, and entirely insufficient adult supervision.

OK so all that out of the way, what's NEXT?

(Now):
2026.6.1+

Post Release bug reports
Tighten Inventory - Index Linkage, Mealie >> Kitchen and AdminTools Kitchen sync. So I can start actually USING it the way I want to for reals...
Add 'reports' view [working_name] (Think exec staff rollups - what does your boss want to know?)
formally link zone_home to default_household in the graph and force index of same. (Yes I'm hardwiring this one once the default household resolved - reasons)

(Planned):
2026.7.0+

Cabinetception, like - what if, a cabinet drawer was, like, another cabinet - maaaaan...

(Ongoing):

MOAR SP1 security plumbing.
- Cabinet ACLs
- Cabinet read normalization
- Index Masking (limited hidden / private labels)
- Query Mask (policy based filters)
- OIDC/PKI
- Tool capability registration

Gonna go finish this Mealie Kitchen thing. Eat food.., Touch grass or something.

Cheers!

NathanCu · June 6, 2026, 1:51pm

Started buttoning *. 6.1 yesterday.

6.1 will be giving the optional tools a paint job like inventory and moving in the new lens system.

(and this is your fair warning to install these I you want them… all are available as containers and easy to drop in with your favorite container management in your home lab.)

Mealie and the kitchen tool get the same treatment as inventory as ‘Kitchen’
Twenty joins the party as a CRM host. - Rolodex. (ok, technically it's Rolodex Prime, organization is the right of all Sentient Beings.)
Zammad joins for incident management and ticketing as Zen ServiceDesk.
Paperless NGX is now the document management substrate and
WikiJS can slot in for free form wiki style text.

Obsidian is coming but probably will slip to sometime in the 7.* branch with cabinets.

I also have OpenSearch, Caddy, Authentik, and a yet to be determined lightweight PKI… (see a pattern? Yes, I AM logging into HA with a passkey now. Thanks for asking…)

NONE of this is required and I working to make sure anything not installed is just not there….

Lenses interface beta goes in. (it's the pattern all these tools follow)

So let's talk about a lens. First let's talk tools

DojoTools - yes I know, Nathan another weird name…

Its taxonomy. By naming it that I tell you these things.

There's a format to follow (see lenses when I ship it)
That format specifies how to use a zenos install and what yojr tool provides to the system and a single lens interface for yiu to support.
The tool is written for an LLM with full instructions and hints
That format allows a tool to register with other compatible tools to share the information it has as context slices on demand without documentation.

Zen AdminTools are the same but operates on the secure surface and therefore should never be exposed to assist they work behind the scenes. Thou shalt never expose admin tools unless you know what you are doing or like pain.

This release will add two more types..

(editor note: apologies if you read my earlier draft this morning They’re Roots Now…)

All the *api advanced tools will become Zen Roots these are the lowest level tools that provide plumbing to a system. Usually bare REST and/OR GraphQL translation to the base service. This Is like integration level.

Grocy API advanced becomes Zen Root Grocy ERP
Mealie API Advanced becomes Zen Root Mealie
Zen Root Paperless joins the plug in show
Zen Root WikiJS comes to play.

The rule for Root, if you're not debugging they don't get exposed..

This covers most tools but sometimes you really could put a business logic middleware in between a raw source and the user to translate how the app works and read it

Lense apps now have the option of providing a ‘Sutra Provider’

Zen Sutras (sutra means thread, its the literal term. Ha ha, get it out of your system.) providers work directly with our new and final superstar of the show. They also do not need to be exposed (were trying to add functions without tool sprawl). They are also the functional business rules and operating logic of the connected platform. (The more I did this the more I realized that I needed this abstraction to exist but functions popped out we could suddenly use…)

The Zen DojoTools Library is finally getting a REBOOT. In 6.1 The ‘Grand Llibrary’ returns to what it was meant to be - a knowledge broker. All the old catch all tools in library like the md5 resolver will be moved into other DojoTools or admin tools (in next version - this is the move/deprecation warning) to host the function they won't go away. Meaning they'll still be there as actions on other tools. But Library will be the source or enrichment and knowledge.

Library powers the Lens interface. You write a tool that supports Lens. It's suddenly supported as a knowledge source you can plug in… We take care of the rest.

WITHOUT EVER EXPOSING YET ANOTHER TOOL…

So what happened was, I started working on Paperless on Thursday and noticed in my install I passed tool #101.

I had a running mental milestone to start tool Redundant Obsolete and Trivial (ROT) analysis on my toolset when I hit 100. (If im gonna gripe about tool sprawl I need to check myself too)

As a result, you get lenses NOW. If you write a tool that provides info and doesn't need a write interface… knock yourself out.

Why would you want to?

Because all the CORE tools now support enriching their return with them. Inspect +lenses (paperless) will enrich your responses with paperless calls matching the context. (person, idea, zone, area) You can build a lens out of just about any registered tool.

So now you know why that particular set first, it does some real heavy lifting.

Build a managed piece of inventory called car (model) in the driveway. Attach your maintenance schedule as chores with Inventory. Attach spare parts and todos. Attach your mechanics details in Rolodex and the scanned service records get tagged with car in Paperless (Library is your search portal, oh WAIT… library search IS the Index and Query…. Hmmm… yeah… it's that. )

Yeah. Friday just reminded me to call the mechanic on Monday and schedule the catalytic converter replacement before Texas emission laws go into effect in November…

If someone wants to write me a Zen lens for Music Assistant to support TheFes’ already awesome tools… I'd appreciate it.

daywalker03 · June 8, 2026, 9:29pm

While I have Authentik, I probably won't use it to log into HA. Some of the other things you have I will probably install.

NathanCu · June 8, 2026, 11:33pm

So you say… Challenge Accepted.

NathanCu · June 10, 2026, 5:38pm

Ok three things.

First, Fable 5 is awesome. If you're an AI person you should do your investigations.

Second, Holy carp!! Fable burns tokens at an UNREAL rate (I watched 5$ vaporized in 30 seconds…)

Narrator: “Hero, your tokens are low…”

Third… I may need a new feature component for 2026.7. I have two choices…

Filecab 6.0 refac looks like it pulls forward into 2026.6.1 if testing completes successfully… I finished the help last night and it's technically feature complete. It also touches EVERYTHING so I want thorough tests before beta. I blew apart two cabs while I was developing it so hopefully you don't have to.

Option 2. If nobody is clamoring for the stuff above today I will hold another 5-7 days on the internal build (break myself first) then put up the 2026.7 beta on or around the 15th of this month. (I'm heavily leaning option 2 because I can fold in a few Quality of Life improvements I have planned as well)

So it's looking like. 2026.7 beta in a week-ish will include.

Updates to all the lens tools and docs
Refactor of lens and stack providing tools to satisfy the contracts
Management tooling to support install uninstall and health management of a lens or stack tool.
Refactored unified manifest in the prompt
Updates to core tools to expose lenses where possible.
Tools for Paperless, Wiki_js, Twenty, and Zammad that consume and provide lenses and/or stacks. (both as tools and as reference examples for BYO tooling.)
Refactor of Library as Lens engine
Refactor of FileCabinet 6 as Stacks engine (Tes, I split the storage media out. You probably know why. You can do both now.)
PERMANENT fix for the volume delayed write issue… finally…

Net effect. More tightly integrated info. Continued reduction of prompt load way expanded feature set.

Long beta ahead but probably the most stable ‘beta’ so far. Looking forward to it.

(oh and yes. For the record Fable collapsed an entire two week design sprint into a single one shot prompt that ran for approx 30 minutes and output the refactored FileCabinet according to the 6.0 spec. Starting at approx 8p us time yesterday. It wasn’t without warts. I neded three follow up bug passes.. But, now. I'm two and a half weeks ahead suddenly…)

daywalker03 · June 10, 2026, 9:56pm

Would i like it now; yes. Do i need it now, no. I need Bruticus up and running on nee hardware first.

NathanCu · June 12, 2026, 4:04pm

Progress report. Yep. We’re moving straight to 2026.7.0

Beta should be up this weekend… WITH

Return of the Library - with a vengeance,

Cabiception! Nested structures and links. (With loop control and orphan detection)

AND VirtualDrawer! (needs a better name)

Prototypes work, and we’ve cleared any fatal issues so we’re now operating Friday’s house live on a v.6 filecab now. Note it is a ground up rewrite of FileCabinet so beta at your own peril. But by the time you get it next week should be not dangerous.

In addition to the stuff above.

Wiki, paperless, crm, service are all now feature complete

I added a tool to manage all these and started a normalize pass through all tools making all data shapes similar and inputs follow the same taxonomy evefh tool accepts mode: now for primary mode select and all tools accept a tool_manifest call that let’s my diag tool see what's broken (backcompat is retained and yes I intend to auto fix in the future, I'm starting with autogenerating and tagging any label requirements stated in the tools label manifest.)

I took another pass through the prompt. Further stiffening the ‘don’t use GetliveState unless someone is Literally on fire and you know it’s the right answer… ‘ directive. AND moved a lot of the hard coded tool map out of the cortex (see tool manifest above?) this is a different map than the mcp tool register we sort the top 10 tools by who, what, when, why, how…. I’m getting close to what prob is the realistic smallest prompt I can make it but I'm not stopping now.

I refactored the prompt header. I can now detect what model you use if you're using the OAI integration. (it exposes the model in a way I can get it) I may see if people want to submit PR for the other 5 top LLM integrations to add the same info. Because I plan on starting model tweaks next version. Currently I can detect what integration you use and make assumptions, or get the model if you use OAI AND the default integration. There’s some vendor-Ness in the models starting to emerge that is best handled through a targeted prompt. If I can’t detect I tell the model ‘unspecified’ so it doesn’t read ‘error’

Obsidian, probably not IN 2026.7 - YET BUT.

I have identified the method, we will be serving a vault for a live server over MCP via streaming HTTP. This probably is the mechanism most of you will find yourself i - I have a vault to connect to somewhere. The solution I'm choosing accepts both MCP and REST and can handle synchronization with a vault system or targeting a bare folder structure. So I'll be setting up both most likely to see what works better. Once that’s running I'll build. Cabinet stack provider and it’ll slot in next to Wiki_js and the rest of the cabinet stack. (I had to have full recursive folder struct working before I pulled in a tool that supports fully recursive file structs. )

I will add a cut down version of the Karpathy vault instructions and my HALMark directives in that help file instructing agents on best practices for maintaining long context and memory.

What this sets up…
Cabients - fast indexing framework and semantic linking
Obsidian backed Cabinets - Agent Memory
WikiJS - Long form instructions, manuals, etc.
Paperless - The backing docs.
CRM - Who they belong to.
ServiceDesk - What I need to do with them.

^^ All of the above are intended to be used together to put together some really cool label indexed franken-storage that actually makes a lot of sense if you are an AI agent.

Then we add all the other tooling to make that useful.

Meanwhile one of the minions is now working a punchlist Friday submits as she uses the tools… (ServiceDesk Backed TaskMaster option is coming if you use both)

Basically. It’s going well.

If you have another tool besides Music Assistant lens (I plan on treating the library like a data source.) or Timer Manager (timers are still a mess, I’m waiting on something here…) that you want that I don’t have YET - let me know I’m adding to the punch list now.

Stay tuned.

daywalker03 · June 13, 2026, 7:19am

Just FYI: apparently Fable 5 is no longer available.

NathanCu · June 13, 2026, 7:22am

I am very aware.

IOT7712 · June 13, 2026, 7:26am

You could switch to Mythos 5.

Oh wait...

Your whole AI ecosystem timeline grinds to a halt because you are not a nation state player with appropriate geo-political connections.

Bring out the chalk and blackboard.

Image comes to mind: Two drunks under the Parramatta railway bridge: I'll swap you two tokens for a fag mate. ( @NathanCu will understand - it's Saturday night and the date is the 13th downunda)

Real question: What's your redundancy fallback when your AI backend becomes unavailable for whatever reason, be it network errors, the random whims of orange carrot tops, or your jailbroken version gets pulled?

daywalker03 · June 13, 2026, 7:42am

I kinda expected you to know, but not everyone has our level of knowledge and resources on the topic.

daywalker03 · June 13, 2026, 7:44am

GPT-OSS here. Will be updating it in the morning.

NathanCu · June 13, 2026, 7:58am

I need to lose three servers at home for this to happen, it just gets slower. Then I fallback to the cloud on a paid model. If I lose all three at once, then I have worse problems.

IOT7712 · June 13, 2026, 8:02am

So graceful degradation, transparently? Will it ramp up again without end-user intervention as they become available?

...I'll swap you those two tokens for a litre of diesel mate! Wanders down the road to the newly built datacentre and leans against it to keep warm in the bitter cold...

NathanCu · June 13, 2026, 8:26am

If I configure it to… Yes -

–edit: someone asked me in DM why I wouldn't…

Money... If your inference pipeline is running a quarter billion tokens a week. You do NOT just let it roll to a paid model on local error else you will be paying for an OpenAI executives kid to go to college.

NathanCu · June 13, 2026, 4:04pm

And so it begins.

This is what I have to say about the Fable situation for now.

Honesty it's like they are trying to build a dam out of a screen door. Ban one model and wait a week two more with stronger inference will be right behind it. Welcome to the Hydra.

Id be ridiculously suprised if the legal news fronts on this don't absolutely ignite Monday Morning after pundits read the weekend legal filings in US district courts.

Hope you like roller coasters…

And I froze FileCabinet 6 for release before any of it happened. The build will be going up tonight or early tomorrow depending on the current test run result. More soon.

NathanCu · June 17, 2026, 4:08pm

Cayt:
”Boss, clean bill of health. We're good — feat/2026.7.0 is live and verified, go for public testing.”

2026.7.0 'Neo' Beta is up.
nathan-curtis/zenos-ai at feat/2026.7.0

What's in Neo: I know Kung Fu. The cabinet is a graph. Every drawer is a node. Every node has meta, labels, and children. Tools are wired to talk to each other by design. Friday composed her own executive dashboard from live-mounted drawers without being asked.

CabCeption — FileCabinet v6.2.0 — nested drawer trees via / path separator. VirtualDrawer (softlink to another path), LiveDrawer (KF4 schema absorbed into FC — fires a tool call on read, warm cache, auto-expiring cold cache, never empty). Drawers are first-class entities: meta + labels + children at every level.
Tool Manifest — zenos_manifest.jinja + MF.tool_manifest(). Every compliant tool self-describes. Manifest broker v6.0.0 aggregates by namespace discovery. The system knows what it is.
Cortex 43 — Rule Zero — DojoTools supersede all HA built-ins. Not preference. Authority. GetLiveContext overridden. Domain routing table in directives. Successor to v42 'The Answer'.
Wake sequence rewrite — ~commands~ interface dropped. ~2,000 chars lighter. The manifest replaced the static surface.
Lens Bus stack= routing — Library v5.5.0 routes generic verbs to registered providers. zen_stack_radar v1.0.0 wires Zammad service desk as a Lens stack provider.
Intentional tool topology — tools wired to talk to each other by design. FC is the wiki surface. Library routes to radar routes to servicedesk. The manifest discovers tools by namespace. LiveDrawers fire tool calls on read.

image1200×630 78.9 KB
ZenZork v1.1.0 — "Totally NOT Zork" text 'adventure' engine using live Room Manager topology as the dungeon. Narrator styles: zork (dry/sardonic), dungeon (DUNGEONMIND — BORANT's corporate dungeon AI, IBM AT since 1984, yeah, he's a little cranky the other AI got all the love and the Primal core.), straight. Quest system, setup commissioning, portal disambiguation. This one's TOTALLY a work in progress; I needed a test for pathfinding and navigation... Feel free to enjoy and contribute.

Read details here:
zenos-ai/zenos_ai/docs/releases/neo.md at feat/2026.7.0 · nathan-curtis/zenos-ai

daywalker03 · June 17, 2026, 7:18pm

Looks good; the hardware for Bruticus just arrived, so i may spend some time tonight setting him up and installing this on him. I also have a new piece of voice hardware that needs set up, so maybe I'll finally unbox it and set it up.

NathanCu · June 17, 2026, 10:03pm

OK, my notes - there's a lot... Pull up a chair.

OK First, There's no fundamental changes in structure in the cabinets in THIS BUILD... ...Yet - won't say I don't have experiments running and there may be a reason the storage got rearranged into 'stacks'

For those venturing into betaland...

YOU WILL Have a metric ton of orphaned scripts. (I think it was a half dozen or so)

Sorry - that's me. After everythign comes up green if you still have a bunch of 'this entity is no longer being provided by...' *dojotools(foo)... Yeahhh

Aligning things to fit my own ruleset points above but it serves something coming soon-ish? And predictability becomes very important very now. I'm doing it as 'efficiently' as I can causing as few possible naming conflicts as possible. There will be payoffs besides a taxonomy.

There ARE new tools - visit through your scripts and see who is-isn't exposed. There may be a reason I renamed the ones I did - Note how suddenly a half dozen Dojotools dropped off your exposed list? You're welcome. The functionality is still there. Had to make room for the newcomers.

"...But Nathan, I don't have..."

Yes, I know. Most of you won't have all, or maybe even HALF of the home lab stack I currently have in my head. That's perfectly fine. I'm COUNTING on it in fact. Test Subj... cough Annnnyway.

The tooling is SPECIFICALLY designed to fail safe, closed and explain itself when it can't reach a server. Conversely - that also means it can help you through the setup too. A lot of these new tools need a 'setup' routing to register the actual server [ TRANSPORT://URL:PORT ] etc. But that means we can also gate the tool on - is it even installed? If not, then an entire class of data falls out of the machine automagically. Yes, these all store in a central cabinet registry so tools can tell you who is registered where for what... (And maybe what permissions it needs...) No tokens will NOT be part of what your agent can register, ON PURPOSE. There are a few more you may want to add as dummies - I'll make sure the readme and install instructions get bumped.

Basically, If [TOOL X] isn't in your stack, and you don't configure "Dojotools (foo for Tool X)" it SHOULDNT break stuff - and if it does, I want to know, because that's a bug.

What's going on right now is I'm rapidly expanding the tool surface across the easiest to integrate, easiest to manage, easiest to maintain free docker container-based distros I could find for [Insert Tool Class HERE] so I could get a realistic base operating environment. The next few posts will describe what's about to happen with it.

Then by having 'Stacks' and 'Lenses' we have a noun-verb class set consistent across tool semantic category. that can either read or write structure shaped like [FOO] at will... I'm dividing this layer from the 'sutra' layer intentionally because that layer defines the 'adapter' surface for what a thing can offer.

(TL; DR, Nathan, WORDS...) FINE...

That ^^^ Basically, translates to... I want data about dolphins in the back office - but only in wiki pages, roomstate, or inventory. Get me back-office inventory dolphins and stories about them and the room...Cool?

(So you don't think I'm nuts - that particular formula would yield a max storage temperature for my wife's office stockroom and the temp sensor attached... Because the particular item labeled 'dolphin' could be temp sensitive... See? Inventory dolphins in the Back Office...)

For just about any external we decide if it's a read - contribute evidence type, (PaperlessNGX) 'Lens' and/ or a write storage class, like the Cabinet system, WikiJS or. Then pick the shape of what you're connecting to and go grab those scripts that look and act like your system, use them as a model to write your own adapters. You should have examples of every capability I could think of across addressable read and/or write types between all the plugins. If you come up with new patterns Heck yeah. Lenses should be small, laser focused and offer flexible call options the agent doesn't have to ask why they want it, should be obvious or don't write the lens. The best ones by far have been me polling one of the agents and asking what they need.

Combine this with LiveDrawer and a nested structure - ohhboy I'm almost as excited as a Dungeon AI with a new toe ring. I'll be looking at a few summary views of nested structures in one call if I can come up with a way to not absolutely explode context with the potential recursion

I won't have Obsidian in x.7.0. I THINK I'm going to solve Obsidian as a DAV connector; it is more flexible than just vaults... - More coming soon, but if I suddenly go Fallout themed, you know what happened.

Project help is in the build - the tool isnt,still needs a LOT of work, 'Finance' Connects to Firefly, I have not round tripped the accounts list or transactions yet - thus it has not made the cut as v.1.0.0 yet. It will earn its wings (Safe manipulation of accounts) before release or get kicked back to next month. << There be dragons here.

Backup before you leap - CabStacks are new after, all - but they're fundamentally basically a dirty trick on top of the same medium so ... Let's see what happens.

They're awesome and No Dungeon Crawler or Zork IP was harmed in the beginnings of our test 'game' (Yes quests are coming... Go break that and report bugs, that helps something else planned that's NOT a game.)

daywalker03 · June 18, 2026, 1:01am

Sounds interesting. I doubt a fresh install will break anything, but you never know.