Friday's Party: Creating a Private, Agentic AI using Voice Assistant tools

NathanCu · April 23, 2026, 3:15am

Follow up to this one because I think there’s a sleeper hit coming in this one. Some of you noticed it was there.

Version 4.6.3+ …Recursive indexing.

I needed it for the camera KFC I’m writing. In the new camera tool you’ll be able to write an index key and store it with the camera and use that key to pump extra metadata into the summarizer when you send a camera to be reviewed. When I was working through it last night, I kept banging my head around the complex index call I needed. I had to do just one more prune to get the exact ent set I… Oh yeahhh the index technically was set up to do this originally so… Send a compound recursive index call (json dict) into index_command and the tool will unwind the parts (up to three levels deep) and make the calls and return it as one call.

I had to detect a recursion attempt and increase the timeout internally. So every loop in the recursion gets up to 5 seconds *3 for a total of 15 max. (if it doesn’t return in 15. Something is very wrong.)

You can ask for ((label_1 And label_2) NOT label_3) now and the results are returned as one index set. ZQ1 filters are applied after the unwind.

(I did this so you can tag all the cameras and get a status report with just a single index call.)

The second enhancement to the index… Inspect, rather. It can read the camera summary and staleness into an index call if the ent set includes a camera with a summary present AND expand is on. Well genuius what if you include that fancy camera trick and send it back into the camera for inference… Them you have old state corrupting a summary.

True. And why I’m adding a series of exception filters to ZQ1. Specifically a list of ents, or a domain, or…

And then I fold it in to your ‘extra context’ label call for the camera’s entity being reviewed as an explicit carveout… No more camera… As far as that summary is concerned. I’m sure you can think of creative uses for that filter…

So the index just learned two new superpowers… I am also applying this ability to any existing tool that calls to the index like this, specifically the ninja summarizer. The entire KFC chain can use the recursive index for context queue…

Don’t think I’ll get it out tonight, still have a doc pass and a few polish items (including another one found by Phil and one from Tes) but very likely tomorrow sometime.

Edit: this is up on a 2026.4.3 feature branch for test/escrow.

Sawtaytoes · April 24, 2026, 8:34pm

I read through the first post and a few on the way down here. I’m not really understanding what this is. Is it fully local?

It’s Agentic AI. Do I install something into Ollama or do I need something fundamentally different?

Does it use Home Assistant’s Voice Assistant or is there something else I need to install?

NathanCu · April 24, 2026, 8:42pm

It is roughly…

a research project that grew to be installable. Patterns and practices and what works what doesn’t. This is a year an a half of hard knocks in HA voice land. Yes there is a hard requirement on ha conversation integration. The hardware. You pick your favorite HA voice compatible… Have fun

That also means most of the answers to common llm in HA issues are in the previous *counts 500-something posts.

100% native tooling. Scripts, automations, and the builtin integrations. With one exception. A hard requirement on Spook.boo

…I do not believe that someone should be forced into a black box. All the scripts are open so you can see what I’m doing and how I’m doing it because it represents best practices in llm prompting. I also did it this way because you can choose not to use the llm prompt side and you’ll still have some pretty honestly badass tools that even work with mcp.

its what you want it to be. Full cloud your funeral but doable full local yup can do.

I am currently running the daily driver 100% full-time local (except experimentation) with inference on an Nvidia rtx5070ti (16g) on a 26000 entity HA install.

So have ha with 1 valid conversation agent integration working then go to the repo. Pull down the files and set the settings and voila.

NathanCu · April 25, 2026, 1:59am

2026.4.3 Lights, Camera, Action! is up

Lights! The Index learned a hella skill.

index_command:
  operator: OR
  index_1:
    label_1: hot_tub_manager
    label_2: hot_tub_deck
    operator: OR
  index_2:
    entities_1: [weather.home]

Yeah - that works now.

Camera! Camera can create summaries and cache them

Modes

Mode	LLM	Snapshot	FC Write	Description
`look`	yes	yes	yes	Analyze single camera, cache result
`read`	no	no	no	Return cached description instantly
`scan`	yes (per camera)	yes (per camera)	yes (per camera)	Sweep all labeled cameras
`info`	no	no	no	Entity attributes + configured _default_ctx
`set_default_ctx`	no	no	yes	Validate + store per-camera default context call
`help`	no	no	no	Return schema

Caching context for a scheduled run is where its at… Pick ents that tell the story of what should be real with that camera and watch your summarization quality JUMP. Pick those summaries up in your security manager… That IS what you did first with your KFCs? Right?

How do I get all those summaries at once? You ask…

I am now injecting the camera’s last LLM summary content and cache age into the attributes of a camera entity at the inspect level just like the hint off a drawer. So just index your camera(s) (yeah that works, just give them all one label - poof instant camera group!) with expand turned on.

You’re welcome. Camera scan anywhere that accepts an index command (you’ll see more of this coming where additional context makes sense.)

Action!
Postman replaces Notification Router and gets a BIG glowup:

Image Attachments

Use image_entity (preferred) or image_url for push notification images. image_entity takes precedence if both are set.

`image_entity` type	Resolved URL
`image.zen_image_<slot>`	`/local/zen_<slot>.jpg`
`image.<other>`	`state_attr(entity, 'entity_picture')`
`camera.<entity>`	`/api/camera_proxy/<entity_id>`

Of course it was the pictures… But theres more coming to this tool you can probably guess where.

So with 2026.4.3, take your favorite camera, add your favorite default context to it, summarize it, respond to the result with something that finds you and can send the picture to you in… *counts fingers. oh yeah - ONE call.

…Because every single back and forth we save by chaining tools is a couple toks, with prob a couple hundred tok payload. Yes - it matters.

NathanCu · April 30, 2026, 2:12pm

Morning team, Tooka few days to relax after x.4.3 dropped. It seems to be working well for people, so far very few issues reported with 4.3, AshaiRey found some sequencing issues in the docs and I’m updating those now and should go out in one of the upcoming 5.x releases.

(Great first Bug @AshaiRey !)

I’m going to do a guided setup of taskmaster and alert manager either today or tomorrow - mid write now.

Next release I’m continuing CTX trims and have pulled another 1K out of the overall CTX size. (Im now under 14K injected into the prompt post entities. No loss of capability. Current target is redundancy in the Dojo dump and improvement to the manifests.)

Last build added an alarm tool beta - I need any of you who use Alarmo to let me know if you need the Alarmo specific functions or if the built-ins are ok. If you say you need them you’re also signing up to test - I don’t use Alarmo.

I’m currently shoring up the experiences enabled by postman, enabling a few interesting options including actionable notifications by tool, and finalizing the docs. I’m also doing some housekeeping tasks like keeping recorder bloat down (you can exclude your cabinet files from recorder safely…) and silencing nuisance logs in the event emitter. (updated a few logs to align with HA defaults so now logger can control log emissions from zen_event, I’m going to advise a log level warn for default run conditions)

At the end of the day, what I’m trying to enable with the latest few releases:

Given a set of cameras lableled with a specific set of labels, The system can auto scan over time, cache the results, and use them in a security assessment component.

After a camera scans it events, the security manager sees the cameral summary event and then triggers a summarization the security system.
(Don’t try to summarize a camera and it’s rolein overall security in one pass. Instead one to get the camera info and another to collate the context. Simpler faster context rich passes work best here.)
The re-summarry of security fires an event and informs its done.
Alert Manager gets the result, runs its inference rules, and decides to raise an alert or no. Assume in our chain we have a person seen on camera
The identification gets noted and passes the notify threshold so its tossed to postman.
Postman knows how to reach me (because I previously saved a home and personal notification profile in the cabinet… Making the tools tell that error now if missing) so
It (Postman) can fire (unprompted) a notification - to my phone including the cached image from the camera with a question about how I want to handle it.

(With 4.3, Missing link, coming in 5.x - final escalation to alarm tool to throw the alarm into ‘ALARM’ mode, that missing link is whitelisting the alarm tool to Alert Manager as a valid endpoint.)

Watch for that how-to soon.

After we get through that - I’m going hardcore into SP1 security stuff and porting my Room Manager solution from NR into HA Core automations and integrating them with Zen. (You probably saw the incomplete package named RM?). You may have seen code references to the Reflex system… Room manager. I’m working on room level summary caching driven by an occupancy and state driven room condition so Friday can ‘Read the Room.’ It looks like the new trigger system enables the system to work the way I need. More on that soon.

Also, a good reminder for all of us working with LLMs when they start - they are dumb and inside a black box. Use that as your starting frame when designing your tools.

daywalker03 · May 2, 2026, 1:02am

Well, I still have a couple of minor issues here, mainly me not being sure of exactly how to update the schema on my older installation. But otherwise, the system is playing nice.

NathanCu · May 2, 2026, 2:09am

Ping me in DM. Let me know what it’s doing.

NathanCu · May 3, 2026, 5:34pm

Good news, everyone! 2026.5.0 is in prerelease escrow.

Five open loops. All closed.

Priority inject is the flagship. Cameras classify motion. Alertmanager routes error-severity alerts into _zen_priority_inject. The prompt now renders a NOTIFICATIONS block at the top of every context frame — active alerts, urgency, providers, time since. The AI enters every conversation already knowing what’s wrong, without being told. The loop from “something happened” to “the AI carries that forward” is now closed end-to-end.

Camera alert policy closes the classification gap. Each camera can now hold a _alert_policy drawer that defines how its motion events get classified and where they route. set_alert_policy mode writes it; look and scan honour it; info surfaces it. Private keys (_*) are now preserved across all look/scan cycles — alert policy no longer gets clobbered on the next camera pass.

provision_member closes the onboarding loop. Before this release, adding a family member who had no HA person.* entity and no pre-provisioned cabinet required a manual four-step sequence that left the person as an orphan after restart (#135). One call now does the full job: find the first available expansion slot, provision it, wire it into the family, rebuild the manifest. OOBE routes external family members through it automatically.

Profile editor write closes the loop that was never actually closing. FC returns confirmed on success, not success — the check was wrong and every write was silently failing its own gate. Second writes also failed because _u_cur didn’t parse the JSON string FC returns for existing drawers. Both fixed.

ZQ-1 v4.6.0 closes the regex loop. is regex() is not a valid Jinja2 test in HA — regex filter was silently matching nothing. Now regex_search(). Also: entity_id_regex filter for matching against the entity ID string directly, and stats_eligible for filtering recorder-eligible entities. (Suprised this one lasted as long as it did, moral on this one, periodically do a code scrub for known bad patterns kids.)

The bootstrap paradox at the center of this release: the alert system feeds context into the AI that manages the alerts that feed the context. It caused itself!

Good news, everyone!

2026.5.0 - Fry’s Grandaddy has joined the chat… Bender not included…

daywalker03 · May 3, 2026, 10:14pm

Sounds good, including the tool to upgrade the cabinets without having to invoke the Cabinet Admin tool directly.

AshaiRey · May 4, 2026, 1:04pm

Doing some clicking around with the camera tool but no success yet

Failed to perform the action script.zen_dojotools_camera. Entity ID unknown is an invalid entity ID for dictionary value @ data['entity_id']. Got None

In the description I see this
The ZenOS Agent’s visual surface: analyze any camera with LLM vision,

I know there is an integration called LLM vision too. Most this be installed for this to work?

NathanCu · May 4, 2026, 1:20pm

No it’s purely using Ai tasks. I do have that integration but I’m intentionally avoiding it. Give me a dm if you continue having issues. Also if you’re using the. 5.* version posted yesterday there were some revisions and it could be a bug

EDIT: It IS a bug ish.

That error can throw if you have not setup your AI Tasks Image entity. I just pushed a fix to the camera tool in 2026.5.x detect the condition and offer a better help message.

NathanCu · May 5, 2026, 2:53am

Still no shiny metal aaaaaanyway…

See post /532 a few above. (you don’t need a link)

Good News!
2026.5.0 became his own grandfather a few minutes ago. (is promoted to main)

I would have done a May 4th pun but then I would have been trying to hit a day not a ready line. Doc lift and tightening the attention system ate in this time plus some ready Ness for compound tools.

Coming next, more prompt trimming and a couple REALLY COOL tools…

(pushed a few post release hot fixes this am. If you pulled last night pull again. If your loader has Cortex 38, it’s the latest)

daywalker03 · May 7, 2026, 12:34am

To continue the discussion started in the 2026.5 release thread, I personally have no need for RBAC because I am the only user of my system; the only other person in the house has zero interest in actually using any part of what I've done directly. I do get that others want it, and even the developers agree that they want to add it, but it introduces a level of complexity that many users may not like unless the developers can find a way to make it opt-in.

NathanCu · May 7, 2026, 12:50am

I totally disagree - we'll have to agree to leave it there.

It has ZERO to do with single user or no - you have to realize now with an LLM agent onboard, it's now a new user. One not named you that cannot be allowed unfettered access from the system. (It really is a safety issue.)

Well I disagree so much - here's what I'm doing about it.

Caller Token. (Later will be derived from user session)

Eventually that will be where the agent will pass its identity. Not authorized for tool, no tool for you.

Why. (I had this discussion with someone else this am.)

THIS:

The event emitter / dispatcher. Will be very useful for the interactive agent and allow for certain shortcut tools - BUT. There is no way you should ever ever ever allow the Agent unfettered access to this tool in an automated unsupervised fashion - because it can technically fire ANY event in the system. So if I'm writing the stuff to define whether you should have access to the tool and let it work? Its only one extra step to allow /deny any given single tool given situation at that point... and Boom - RBAC.

(based on Kerberos Constrained Delegation, if anyone who cares is paying attention. The agent will be delegated it's base access from the system and gets a TGT at 'login')

So step 1 is ALREADY baked in...

Basically, if you're not in this list you CANNOT be called as an automatic response action at the end of the Summarizer pipeline. (Phil, this is what you kept slammign against)

If I could not do this it would be entirely too dangerous to allow an automated agent to run on your system.

^^^ THIS is RBAC well proto style...

I cant do it for the UI, but I can sure as hell do it for the LLM tools

SP1 will make this a situational/acl'd list instead of a basic pass/fail.

daywalker03 · May 7, 2026, 8:27pm

I fully agree that UI Agents should be heavily restricted. I also agree that OIDC/RBAC are worth adding for those that feel they are important features; I just don't personally see the value for me based on what I'm doing with Home Assistant.

NathanCu · May 7, 2026, 9:17pm

See that's where I really want to change your mind I haven't even started the cool stuff yet...

On deck.

Testing a patch for action at end of summary pipe.
RoomManager (remember Zach? What IF, a tool knows where it is and can remember the playbook)
Light Manager (how it looks and how the stuff here spills over)
MediaManager (how it sounds here, and in an adjacent room...)
Climate Manager (or how the zones impact each other)
Shades (where one tool impacts the outcome of multiple domains and may be impacted by summary context?)

Yeah I think the base tools are solid enough to pile these on.

daywalker03 · May 9, 2026, 9:53pm

Good luck changing my mind because unless you eventually plan to use Built in RBAC for this, my information only approach to Home Assistant dashboards doesn’t require that I use it. And these features look good but won't be usefright away.

zsahs84 · May 10, 2026, 7:01pm

To any cloud kids out there still stuck on cloud because you’re trying to prove to your wife how an expensive GPU is a family investment and not a new toy…

Googles Gemini integration using Gemini Flash 3.1 Lite preview for the conversation and OpenRouters integration using Mistral Nemo for the tasks (ninjas and supers) seems to so far be a capable and cheap effective combination.
I’ve been using this set up about a week. Dramatic drop in costs and house seems smart as ever.
I’d love to hear any other cloud users set ups and any of your notes. This is a combo I found through trial and error that works for me but I’m still learning and always open to other ideas.

philip.schmid1142 · May 12, 2026, 6:46am

Do you have the same experience as me that you basically need to specify whats nominal and whats not thorughout every edge case because otherwise it gets escalated in the KFCs?

philip.schmid1142 · May 12, 2026, 12:55pm

After lots of trying and failing and even more trying and failing:

What im trying to achieve: KFC Component that:

Realizes, hey someone just came home and the house it set to unoccupied (Master switch if someone actually living in it, holiday home) --> Says Hi, would you like to set house to occupied (script execution if yes with a pin safety)
Ask whos staying in the house (Indexed 1-6)
Checks preferences if default room is already inside a drawer with preference regarding blinds, tempearture etc and sets it
Asks for unclear things like: multiple default rooms, room if no prefernces are recorded for that person etc etc.
Deletes person in active_guests subkey of travel_manager drawer midnight of the "stays_until" date

Currently I have:

 travel_manager:
    value:
      component_group:
        component_key: travel_manager
        parent_component_key: ""
        primary_index_key: travel_manager
        shared_delay_seconds: 0
        shared_labelset: []
        shared_schedule_group: ""
        shared_trigger_subscriptions: []
      component_instructions: |-
        guest_occupancy_rules:
          priority: high
          core_safety_rules:
            - Never toggle input_boolean.house_occupied directly.
            - To change house occupancy, use only script.zenos_secure_toggle_house_occupied.
            - Require user-provided authorization before calling the secure toggle script.
            - Never expose, repeat, store, log, or summarize authorization codes.

          source_of_truth:
            active_guests:
              path: household_drawer.active_guests.value
              purpose: Only valid source for current guests.
              format: JSON array
              required_fields:
                - name
                - room
                - departure_date
                - notes
            guest_preferences:
              path: household_drawer.guest_preferences
              purpose: Per-person room, temperature, and notes preferences.
            rule:
              - Do not infer current guests from person entities or device trackers.
              - If a person is not listed in household_drawer.active_guests.value, assume they are not staying on the island.
              - People not listed in household_drawer.active_guests.value are likely in Switzerland and should not affect guest logic.

          guest_handling:
            chat_detection:
              - Treat guest mentions in normal chat as travel or guest input.
            missing_details:
              - Consult household_drawer.guest_preferences first.
              - Then ask only focused follow-up questions for missing required fields.
            workflow_order:
              - First record the guest in household_drawer.active_guests.value.
              - Then prepare the assigned room silently.
              - Use separate service calls per domain when preparing the room.

          presence_filtering:
            purpose: Prevent stale or irrelevant person states from creating false alerts.
            rules:
              - Only consider person entities that match someone in household_drawer.active_guests.value.
              - Ignore all person entities that are not in household_drawer.active_guests.value.
              - Ignore unknown and unavailable states.
              - Ignore stale not_home states older than 24 hours.
              - Ignore any person that has not been home within the last 24 hours.
              - Only consider home or not_home states when last_changed is less than 24 hours ago.
              - After filtering, compare valid person states against household_drawer.active_guests.value.
              - If filtered person states and household_drawer.active_guests.value match expected occupancy, status is nominal.

          house_occupied_reminder:
            purpose: Remind user to index guest stays shortly after the house becomes occupied.
            suggested_act:
              tool: postman
              action_required: true
              urgency: 6
            send_only_if_all_true:
              - A person listed in household_drawer.active_guests.value changed from not_home to home.
              - input_boolean.house_occupied changed from off to on.
              - input_boolean.house_occupied last_changed is less than 30 minutes ago.
              - Guest stays have not already been indexed.
              - Reminder has not already been sent for this occupancy event.
            do_not_send_if_any_true:
              - input_boolean.house_occupied last_changed is 30 minutes ago or older.
              - Reminder was already sent for this occupancy event.
              - Guest stays are already indexed.
              - Person is not listed in household_drawer.active_guests.value.
              - Person state is unknown or unavailable.
              - Person state is stale.
              - Person has not been home within the last 24 hours.
            reminder_message_intent: >
              Remind the user to index guest stays if this was not already done.

          nominal_condition:
            - household_drawer.active_guests.value exists and is readable.
            - Recent valid person states match the expected active_guests list.
            - No stale, unknown, unavailable, or unrelated person entities are treated as problems.
            - No reminder is sent unless the house_occupied transition is newer than 30 minutes.
              custom_trigger_override: []
      delay_seconds: 0
      friendly_name: Travel Manager
      index_call:
        expand_entities: false
        filter_json: {}
        label_1: travel_manager
        label_2: ""
        operator: AND
        output_fields: ""
      kata_key: travel_manager
      label: travel_manager
      meta:
        enabled: true
        published_from: travel_manager
        requires:
          cert: ""
          level: 0
      more_info: ""
      parent_component_key: ""
      primary_index_key: travel_manager
      tool: ""
      trigger_subscriptions:
        - force_summary
        - daily_midnight
        - house_occupied_arrival
      version: 1.2.0
    timestamp: "2026-04-04T00:00:00+02:00"