Friday's Party: Creating a Private, Agentic AI using Voice Assistant tools

Catch up time!

If you're new here, you may be wondering


what exactly the hell is going on?


That's fair. TOTALLY,

This started simple: could I build something that feels less like a chatbot and more like a member of the household? Also teaching tool - Story WAAAAY up there - feel free.

Most systems know facts. People live in context.

Now - If I tell you we're having friends over Saturday night, you immediately start thinking about food, music, food, shopping, guests, more food, schedules, "where's my wok?", and if you're my wife, whether the house is ready. Nobody explains how those things connect. They just... DO.

Current assistants don't really work that way. They answer questions and commands. In fairness, OK some of the commercial ones based on models can get a little farther than that if you're willing to hand over a frankly alarming amount of information about your life. Where you are. When you're home. What you buy. How you use electricity. What you cook. Who visits. What your routines look like.

YES, you absolutely CAN give control of your HA install to a loop agent (any openclaw hermes agentzero, paperclip WTHever...) points HALMark is over there - yes, I'm about to update it for 2026.6.0 please use it - and I hope you are very good at Devops. smiles Enjoy your dragons.

(Admittedly FUN dragons, but if you don't know what you're doing. YOU CAN get hurt...That's as far as I take that. I also ride motorcycles. Risks exist.)

This thread is what happens when you decide to keep that information, build the plumbing yourself, expose to it as many deep connections in your data - as safely as possible (in a way you KNOW it works) and see how far down the rabbit hole goes. SO.

If you'd like to wander into the dunes with the rest of us:

Current as of this post: Zenos: 2026.6.0 / Cortex: 42.1 HA: 2026.6.0

Before diving into the latest madness, a few thread landmarks are worth a look:

Post #1: The Front Door
What Friday is, what "agentic" means in this build, and why tools plus context are the whole game.

Post #15: Poetry in Motion
The prompt-craft thesis. Intent, urgency, mood, dense language, and why "brief" often fails when the model needs a root concept.

Post #34: Ninjas
The first real look at summarization, Kung Fu, Katas, and attention management.

Post #42: Ninja 2, Electric Boogaloo
The scheduler and context-selection leap. Early autonomy scaffolding.

Post #49: Limits, Gov'na
Context budgets, aliases, tool descriptions, entity exposure, and why "just expose everything" eventually falls apart.

Post #55: Meet Kronk / The Monastery
Friday, Kronk, expert workers, queued jobs, specialist models, and one of the biggest architectural pivots in the project.

Post #120: Storing Elephants in Drawers
Cabinets, drawers, pointers, inherited context, and memory as infrastructure.

Post #234: Sand Dune Plinko
Grounding, hallucination control, directives, identity lanes, Katas, and making truth the easy path.

Post #261: Hyper-huh?
Why Home Assistant stops looking like a giant pile of entities and starts looking like a connected world.

Post #456: Stop Putting SOPs in Your Cortex
The modern ZenOS pattern: label it, write a KFC, store it in the right cabinet, index it, expand it when needed.

Fair warning: there are elephants in here. Some are obvious. Some are hiding behind the architecture diagrams.

The current focus is the kitchen.

Before Friday can help run a household, she should probably be able to help run dinner.

Everything else in this thread is what happens when that idea collides with Home Assistant, too much coffee, and entirely insufficient adult supervision.

OK so all that out of the way, what's NEXT?

(Now):
2026.6.1+

  • Post Release bug reports
  • Tighten Inventory - Index Linkage, Mealie >> Kitchen and AdminTools Kitchen sync. So I can start actually USING it the way I want to for reals...
  • Add 'reports' view [working_name] (Think exec staff rollups - what does your boss want to know?)
  • formally link zone_home to default_household in the graph and force index of same. (Yes I'm hardwiring this one once the default household resolved - reasons)

(Planned):
2026.7.0+

  • Cabinetception, like - what if, a cabinet drawer was, like, another cabinet - maaaaan...

(Ongoing):

  • MOAR SP1 security plumbing.
    • Cabinet ACLs
    • Cabinet read normalization
    • Index Masking (limited hidden / private labels)
    • Query Mask (policy based filters)
    • OIDC/PKI
    • Tool capability registration

Gonna go finish this Mealie Kitchen thing. :wink: Eat food.., Touch grass or something.

Cheers!

4 Likes

Started buttoning *. 6.1 yesterday.

6.1 will be giving the optional tools a paint job like inventory and moving in the new lens system.

(and this is your fair warning to install these I you want them… all are available as containers and easy to drop in with your favorite container management in your home lab.)

  • Mealie and the kitchen tool get the same treatment as inventory as ‘Kitchen’
  • Twenty joins the party as a CRM host. - Rolodex. (ok, technically it's Rolodex Prime, organization is the right of all Sentient Beings.)
  • Zammad joins for incident management and ticketing as Zen ServiceDesk.
  • Paperless NGX is now the document management substrate and
  • WikiJS can slot in for free form wiki style text.

Obsidian is coming but probably will slip to sometime in the 7.* branch with cabinets.

I also have OpenSearch, Caddy, Authentik, and a yet to be determined lightweight PKI… (see a pattern? Yes, I AM logging into HA with a passkey now. Thanks for asking…)

NONE of this is required and I working to make sure anything not installed is just not there….

Lenses interface beta goes in. (it's the pattern all these tools follow)

So let's talk about a lens. First let's talk tools

DojoTools - yes I know, Nathan another weird name…

Its taxonomy. By naming it that I tell you these things.

  • There's a format to follow (see lenses when I ship it)
  • That format specifies how to use a zenos install and what yojr tool provides to the system and a single lens interface for yiu to support.
  • The tool is written for an LLM with full instructions and hints
  • That format allows a tool to register with other compatible tools to share the information it has as context slices on demand without documentation.

Zen AdminTools are the same but operates on the secure surface and therefore should never be exposed to assist they work behind the scenes. Thou shalt never expose admin tools unless you know what you are doing or like pain.

This release will add two more types..

(editor note: apologies if you read my earlier draft this morning :wink: They’re Roots Now…)

All the *api advanced tools will become Zen Roots these are the lowest level tools that provide plumbing to a system. Usually bare REST and/OR GraphQL translation to the base service. This Is like integration level.

  • Grocy API advanced becomes Zen Root Grocy ERP
  • Mealie API Advanced becomes Zen Root Mealie
  • Zen Root Paperless joins the plug in show
  • Zen Root WikiJS comes to play.

The rule for Root, if you're not debugging they don't get exposed..

This covers most tools but sometimes you really could put a business logic middleware in between a raw source and the user to translate how the app works and read it

Lense apps now have the option of providing a ‘Sutra Provider’

Zen Sutras (sutra means thread, its the literal term. Ha ha, get it out of your system.) providers work directly with our new and final superstar of the show. They also do not need to be exposed (were trying to add functions without tool sprawl). They are also the functional business rules and operating logic of the connected platform. (The more I did this the more I realized that I needed this abstraction to exist but functions popped out we could suddenly use…)

The Zen DojoTools Library is finally getting a REBOOT. In 6.1 The ‘Grand Llibrary’ returns to what it was meant to be - a knowledge broker. All the old catch all tools in library like the md5 resolver will be moved into other DojoTools or admin tools (in next version - this is the move/deprecation warning) to host the function they won't go away. Meaning they'll still be there as actions on other tools. But Library will be the source or enrichment and knowledge.

Library powers the Lens interface. You write a tool that supports Lens. It's suddenly supported as a knowledge source you can plug in… We take care of the rest.

WITHOUT EVER EXPOSING YET ANOTHER TOOL…

So what happened was, I started working on Paperless on Thursday and noticed in my install I passed tool #101.

I had a running mental milestone to start tool Redundant Obsolete and Trivial (ROT) analysis on my toolset when I hit 100. (If im gonna gripe about tool sprawl I need to check myself too)

As a result, you get lenses NOW. If you write a tool that provides info and doesn't need a write interface… knock yourself out.

Why would you want to?

Because all the CORE tools now support enriching their return with them. Inspect +lenses (paperless) will enrich your responses with paperless calls matching the context. (person, idea, zone, area) You can build a lens out of just about any registered tool.

So now you know why that particular set first, it does some real heavy lifting.

Build a managed piece of inventory called car (model) in the driveway. Attach your maintenance schedule as chores with Inventory. Attach spare parts and todos. Attach your mechanics details in Rolodex and the scanned service records get tagged with car in Paperless (Library is your search portal, oh WAIT… library search IS the Index and Query…. Hmmm… :smiling_face_with_sunglasses: yeah… it's that. )

Yeah. Friday just reminded me to call the mechanic on Monday and schedule the catalytic converter replacement before Texas emission laws go into effect in November…

If someone wants to write me a Zen lens for Music Assistant to support TheFes’ already awesome tools… I'd appreciate it.

1 Like

While I have Authentik, I probably won't use it to log into HA. Some of the other things you have I will probably install.

1 Like

So you say… :smiling_face_with_horns: Challenge Accepted.

1 Like