Friday's Party: Creating a Private, Agentic AI using Voice Assistant tools

Read this thread start at the top. I know it’s long but yes you’re running into a wall. You can’t keep adding instruction upon instruction it eventually gets… Messy.

All of this is a mathematically constructed prompt to put ad much functionality I front of the agent as possible.

Short version yes this is why Friday is wicked smart and punches WAY above her weight class.

AI isn’t all about the model. It’s a combination of model, harness (home assistant assist, and in my case the Zenos core prompt) and tools.

Weakness in any of the three =idiot agent

The difference with mine. I DO NOT install an integration. I work wholly within what HA brings to the table. No Javascript or python code was added to get here.

I know…
I am just asking if i can connect some AI model to this friday or zenos or how do you call it.

Yes.

Step 1.

Homeassistant.

Step 2

Choose your preferred model.

Step 3 install zenos scripts.

And also:

  1. Is this powered by AI, a database of commands, or??
  2. What are the pros and cons of this system (without reading the whole thread)
  3. Can I use it with H.A. Docker?

Yep 330. Damn I was going for 400…And it’s going to be 500 minutes by next Wednesday. You won’t win fans by complaining about me giving info… Like I’ve said before if you want a summary notebook lm does a great job. Point it at the thread.

I just gave you the link and if you’re not going to try I’m not spoonfeeding it to you. Read the links I gave you and them let me know if you have further questions.

But short version yeah it’s probably gonna be what you want but it takes a LOT OF reading.

W.H.Y. I CANNOT INSTALL IT VIA HACS?
Whoever made it that complicated?
BTW: how can you start replying 2 seconds after i post something?

There’s no Javascript. It doesn’t work that way. :smiling_face_with_sunglasses:

If… and when HACS is appropriate I’ll do that, for right now it is not.

It takes 5 minutes to copy files. Because I’m staring at my phone and I type fast.

There Is an entire install document gives every step I excruciating detail.

But if you want an ai that’s more than a show pony its going to take some work on your part. No excuses. Right now there’s a lot of fly by night install my agent it do… And you’re limited to what that person thinks an agent should do. My preference is provide tools and a general framework for the agent to understand it’s environment and see what happens next. Unfortunately there’s real work that’s required to make that happen.

This isn’t for someone who wants a big hands free easy button. It was essentially a research project at first.

That said there’s already plenty of folk who have taken the dive in and can help now if you do take the plunge.

TRON Update!

Ok 2026.9.1 posted yesterday. Testers are reporting good things. It’ll move to main tomorrow if I do t hear anything else.

For TRON (2026.10. 0)
I setup Nyx in a loop to

  • Read tickets identify next work.
  • Read existing tool.
  • Analyze tool,
  • Identify gated actions and assign a cert level
  • Identify path to ‘Caddilac’ it
    • apply single entry single exit
    • refac error messages on all branches.
    • Apply gates
    • apply common resolver
    • apply the common output shape envelope
    • remediate callers using old pattern
  • Write plan.
  • Implement changes on T (test HA server, vanilla)
  • Test tool manifest on T including functional deny
  • Port to H (live prod)
  • UAT test on H (including the cert assignment and functional pass)
  • Update Docs
  • Send note to release management with disposition
  • Update and/or close ticket…

Loop. :smiling_face_with_horns:

Read: Milestone - functional automatic CI/CD pipeline.

Then release management is a pile of tickets ‘Cayt’ (lead dev and release management) reads and for each.

  • pii check
  • reformat comments.
  • validation that tool desc and input fields meet spec
  • validation that the tool is - correctly shaped or holds an exemption with an end date and that date has not passed.
  • doc check (consistent, does this doc describe that version)
  • build check (indices, tocs xref, etc.)

Then Cayt commits local and waits for a sync session to push to origin.

Nyx was halfway through when I got home from work yesterday and done with everything except file cabinet, Inspect, query, and Index this morning. And I am currently running the builds in prod successfully.

I still want to hand migrate the indexer set (index, query, Inspect, zq1) and filecab and may still hold full implementation of the pattern in filecab for 11. (it’s… Special.)

Let’s just say she tried once already…

Either way that means first betas of 10 will probably go live Saturday after I promote 9.1

Moved 2026.9.1 to main as stable.

I’m starting to build some archetecture diagrams for the docs. I know it’s dense. Here’s a first stab at the whole Pic (envisioned as of 2026.12.*)

I know it needs a simpler version for docs. Working on it, and I figured some of you (who’ve asked if I have pictures) might want a preview.

Looks good. Will have to take better look at it on PC

Posted 2026.10.0 Beta

If you’re adventurous or REALLY want to see how this cert system is gonna work - jump in. I’ve got it right-shaped now and only still need to do the validity / revocaton mechanism. That should be in place before RC, a giant chunk of the tools already have thier cert gates in.

I added a cert bundle mechanism so you can specify a bundle of certs or… maybe we just call it a Role, allowing for the Basis of Access Control… for your Agent. :slight_smile: Yeah I know what I said :wink:

The cert storage drawer is still in the clear so it’s not real security or policy control YET. Let me be VERY clear about that - we dont want you to have a false sense of security. There is ALSO a plan to move the cert drawer to a secure enclave cabinet coming in the near future. (Probably *.11.x) I have to make sure it actually works first.

RM got a SIGNIFICANT performance and streamline pass. Now that it works RIGHT it’s going to work right FAST. It also got a watchdog in the dispatcher. Any single RM pass hangs for longer than 3 min (read: more than double max expected), I’m pretty sure it’s hung and im restarting RM. It should finish in seconds. If you’re running 9 and it feels a bit sluggish, this should help.

Lots o bugs - like LOTS. “Jamal has so many sorries to give…”

Please dont read the changelog. :slight_smile: Who was that guy who wrote that crap - I ought to murder him. :slight_smile:

Prompt is going on a diet and so are the tool descriptions. Expect a SIGNIFICANT reduction in CTX - still working on the final numbers.

This one’s for Flynn, let’s Shore up the Grid.

The Tron upgrade is scheduled for when I have finalized the two system cluster, as I will have to reinstall what’s on the Bruticus hardware first. And yes, I hope to get to that some time within the coming week; I think I actually will have all of next weekend off this time around.

It’s been a while, I just wanted to stop back by and say hi.
You all may really be surprised by how modular Zen can be.
Here is just one of many many examples.
I have a Song Writing / Studio Notes app I’m building that is a PWA. Used local storage and backs up to Drive. cool.
but. local cache is only so big and what if you can’t get to drive?
I converted that part of my app over to Home Assistant and with some labels and a chat, stuff like this is now possible.

I wish I had more time to write more up about this and my experience. Nathan has built some pretty kick ass stuff that has given some pretty incredible abilities to home assistant.
If you asked me even a couple months ago if I ever thought I’d be able to lay on my couch and ask home assistant to change the lyrics to one of my songs I’d have laughed at you.

Nice work.

Pushed nightlies for Tron.

nathan-curtis/zenos-ai at feat/2026.10.0

Most of the scripts are right-shaped now and follow the SESE format with the common envelope and funnel. (Intentionally omitted a few this batch - filecab saved for v.next - I have plans for it..)

Certification Pass - all the certs and gates should exist now… Added a local at the action console no live ack bypass if you are a logged in user and hold admin privs in HA. (this is the first module that starts reading real HA user class info. It will be tagged to your person object by way of the ID tool… It’s also the same mechanism I test to see if we can make ZWave changes. Oh BTW there’s a ZWave tool now.)

Fixed some things in the cert chain and some rando bugs - and you now have dusk-dawn lighting in the security tool. Tag fixtures - let the system do the rest.

Put it on the grid.

Expose the Ring 0 tools automatically on boot or no?

  • Yes I want Flynn to identify and fix missing exposure of ring 0 tools automatically…
  • Stay out of my config.
0 voters

I think I can do it and it would be an easier setup for new kids. But… It’s messing with real settings. It’s going in the tools but the question is should we autofix it. No labels no filecab no zenos. So… Should I detect not exposed label, filecab or other security class tools and re expose them?

There is also a 3th option.
Show what is missing and ask yes or no.
Maybe include also what the result will from a yes or no

Possibly.

The structure may or may not be there to support it yet at the point if the label tool itself is not active. (which would be what I’m fixing same catch 22 I have securing the filecab tool)

I agree ask first… If I can. But at this point I may not be able to yet. No cabinet or no index…

While the third option is ideal, I’m okay with Flynn changing things on my system. However, other folks may not be.

Okay, it’s 23:25 here, bedtime for me already overdue and I am thinking out loud.
Maybe it is feasable to do it in 2 passes. The first pass detects the problem area(s) but does nothing with them but just reporting them first.
Just as you would do an automatc repair but without the actual repair.
This will show users with their own config where they are in danger.