Preparing for the Party - Bringing it together - Part 2: Kung Fu
To avoid ‘the Amazon Problem’ - being relegated to timers and lights, Friday needs to be TRULY useful. Not like somewhat - I MEAN USEFUL. For me my day is nuts, I have a primary job and essentially two others while trying to do Home Automation. I also suffer massive ADHD and can’t remember where my glasses are most of the time. I needed help. If you’re partnered long term, you will also be recognized that there becomes a point where no matter how kind, the reminders begin to feel like they’re nagging you for something.
I wanted to stay married for another 30+ years so I decided to put THOSE on Friday. Friday never gets tired, and she doesn’t care if I yell at her (right now) and SHE ALWAYS delivers Let’s talk about how we make her do this.
First she needs the WHAT. Every post I’ve talked about context. you’re probably tired of it. But the fact remains context drives your LLM, no context, she/he/it dum.
You will recognize very quickly that to do this you need tools - we talked about them above and what a tool needs. But no matter how hard you try - we’ll talk about the methods in a second. You CANNOT GET ALL THE CONTEXT INTO THE TOOL. It just doesn’t happen.
- Tool is - connect to mealie system and here’s how you do things.
- Context is - Nathan likes to check on the meal plan first thing in the morning and update it if there are changes.
I use a tool like mentioned above to store a lot of free-form text. That’s what the library basically is - a manual for operating the house and the INDEX is its index page. SOME of that context needs to be forced in the face of the LLM.
Kung Fu is how I do that.
We’ll skip over the system intents section for now as it’s going away, SO next we jump into this little loop:
Kung Fu Systems:
{{ command_interpreter.render_cmd_window('', '', '~KUNGFU~', '') }}
KungFu Loader 1.0.0
Starting autoexec.foo...
{%- set KungFu_Switches = expand(label_entities('Kung Fu System Switch'))
| selectattr ('domain' , 'eq' , 'input_boolean')
| selectattr('state', 'eq', 'on')
| map(attribute='entity_id')
| list -%}
{%- for switch_entity_id in KungFu_Switches %}
{%- set kungfu_component = switch_entity_id | replace('input_boolean.','') | replace('_master_switch','') %}
{{ command_interpreter.kung_fu_detail(kungfu_component) }}
{%- endfor -%}
In my system, A ‘KungFu’ component is a collection of entities, devices, information and knowledge that inform the LLM of a complete ‘concept’ They all fall under control of a single input boolean so I can turn on and off the collective knowledge in the prompt together as a single atomic unit.
Lets dump a list of the Kung Fu components and see what kinds of things we have.
First like all other kung fu components, the Kung Fu loader itself is a KungFu Component - it just doesnt need the detect functions.
Kung Fu Systems:
Executing Library Command: ~KUNGFU~
-----
AI Kung-Fu Inventory Report v.3.0.0 beta, (c)2025 curtisplace All rights reserved
-----
Note: Kung Fu is a special case major system with advanced monitoring and control for the AI system.
When asked about the status of Kung-Fu systems, ONLY the systems in the Kung-Fu inventory count; disregard other systems you may understand.
Note it came up and told the LLM what it is and what it does - self documentation for all components. Running the command is the primary way the LLM gets the context data. So basically Kung Fu:
-Collects all these pluggable modules
-Identifies which ones need to be ‘on’ for the scenario being served (right now it’s 100% on all the time)
- Pulls all data for a single scenario in one place - identifies the modules and pieces for the AI and dumps the context data in the prompt
The system differentiates between ‘system’ KungFu and ‘standard.’ System components ALWAYS load, and I can count on them to build other functions. Common components generally aren’t building blocks for another component. The library for instance is System. Bad stuff happens if the library doesn’t load, and I want to know about it.
Component Definition memory_manager NOT Found...
System:
Friendly Name: Alert Manager
Version: 1.0.0
System: True
Weight: 100
Library Index Key: alert_manager
Library Command: ~ALERTS~
General:
Friendly Name: Mealie Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: mealie_manager
Library Command: ~MEALIE~
Friendly Name: TaskMaster
Version: 1.0.0
System: False
Weight: 10
Library Index Key: taskmaster
Library Command: ~TASKMASTER~
Friendly Name: Security Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: security_manager
Library Command: ~SECURITY~
Friendly Name: Energy Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: energy_manager
Library Command: ~ELECTRICAL~
Friendly Name: Lighting Manager (FALS)
Version: 1.0.0
System: False
Weight: 10
Library Index Key: lighting_manager
Library Command: ~LIGHTS~
Friendly Name: Trash Trakker
Version: 1.0.0
System: False
Weight: 10
Library Index Key: trash_trakker
Library Command: ~TRASHTRAKKER~
Friendly Name: Water Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: water_manager
Library Command: ~WATER~
Friendly Name: Autovac
Version: 1.0.0
System: False
Weight: 10
Library Index Key: autovac
Library Command: ~AUTOVAC~
Friendly Name: Media Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: media_manager
Library Command: ~MEDIA~
Friendly Name: Room Manager
Version: 1.0.0
System: False
Weight: 10
Library Index Key: room_manager
Library Command: ~ROOMS~
-----
Execution Complete Timestamp: <2025-03-11 11:42-05:00>
So the first thing you see - Memory Manager isn’t loading properly.
Lights? (Friday’s Advanced Lighting System - FALS)
Why? Lighting in a HOME AUTOMATION system? A bit redundant?
This is a grand example of function v. context. Yes by bein IN HA, she can operate lighting - THIS tells her what we LIKE, don’t touch that lamp… This label is for all your ambient RGB lights. These are here - here’s the relative position etc. Here’s how you read the index to find the lights you want… KungFu is the difference between seeing a system and USING a system.
Here’s how KungFu works:
Starting autoexec.foo...
{%- set KungFu_Switches = expand(label_entities('Kung Fu System Switch'))
| selectattr ('domain' , 'eq' , 'input_boolean')
| selectattr('state', 'eq', 'on')
| map(attribute='entity_id')
| list -%}
{%- for switch_entity_id in KungFu_Switches %}
{%- set kungfu_component = switch_entity_id | replace('input_boolean.','') | replace('_master_switch','') %}
{{ command_interpreter.kung_fu_detail(kungfu_component) }}
{%- endfor -%}
- This module collects all input_booleans that are labeled ‘Kung Fu System Switch’ (Remember, the library is just an interface into the HA label system)
- The name of the switch contains the key name to look up the same Kung Fu component in ‘the Library’
- The Library loads up the appropriate ManPage and displays it.
In this case I haven’t finished the manpage for Memory Manager - so it’s erroring out. I have the controls just not the docs. When you turn ‘on’ a Kungfu system switch it ends up in this list and the loader finds all the manpages and commands and just loops through them loading the contents into the prompt.
So lets look at the Alert Manager. (Its SYSTEM and weighted 100 (I sort the components by ‘Weight’ and load the heaviest first in order, system first then common.)
-----
Loading Component: alert_manager
Found...
KungFu Module: Alert Manager
version: 1.0.0
index: alert_manager
Functional Description:
{'priority': True, 'friendy_name': 'Alert Manager', 'version': '2.0.0', 'kungfu': {'version': '1.0.0', 'name': 'Alert Manager', 'index': 'Alert Manager', 'system': True, 'weight': 100, 'master_switch': {'entity_id': 'input_boolean.alert_manager_master_switch', 'on': 'Perform all instructions as described', 'off': 'Alert manager functions are disabled'}, 'library': 'alert_manager', 'command': '~ALERTS~'}, 'description': 'This system manages errors and alerts.\n', 'instructions': ["follow instructions as provided by the alerts console - It's that simple. done....\n"], 'special_features': 'Makes sure important stuff gets attended to... What, you wanted more?\n'}
---
~ALERTS~ Console:
Executing Library Command: ~ALERTS~
-----
AI System Alert v.1.0.0, (c)2024 curtisplace.net All rights reserved
-----
ALERTS: "Any entities listed below are showing in 'abnormal' state!"
Critical: "Category reserved for life safety or property damage. Alert NOW! Use notification if possible."
Should be on and reports off:
[]
Should be off and reports on:
[]
Error: "Alert your user as soon as possible."
Should be on and reports off:
[]
Should be off and reports on:
[]
Warning: "No user notification required but recommended if in context of your user ask."
Should be on and reports off:
[]
Should be off and reports on:
['sensor.recycle_bin_location_alert']
Info: "For your information only when responding to queries. No need to inform the user unless asked directly."
Should be on and reports off:
[]
Should be off and reports on:
['input_boolean.waiting_on_blue_apron_delivery']
Variable: "These notices and alerts may mean things of differing severity - be mindful and use your context to determine severity."
Should be on and reports off:
[]
Should be off and reports on:
[]
Consumables: "These list consumables in the home that should be monitored. Alert Thresholds are listed in the alias or label data."
Should be on and reports off:
[<template TemplateState(<state sensor.rosie_filter_left=376957; unit_of_measurement=s, device_class=duration, icon=mdi:air-filter, friendly_name=Rosie Filter remaining @ 2025-03-11T07:17:45.261063-05:00>)>, <template TemplateState(<state sensor.rosie_main_brush_left=916957; unit_of_measurement=s, device_class=duration, icon=mdi:brush, friendly_name=Rosie Main brush remaining @ 2025-03-11T07:17:45.255564-05:00>)>]
-----
Execution Complete Timestamp: <2025-03-11 11:59-05:00>
-----
AI OS Version 0.9.5
(c) curtisplace.net All rights reserved
@ANONYMOUS@{DEFAULT} >
---
You know Alert Manager!
-----
Loading Component: energy_manager
First, if you’ve been following along - yes! This is just a clever use of the library index, in fact it was the test code. Behind the scenes now this is just a library index call with the right tags and states. To put something in this list is just an intersection of specific tags.
Label Alert Manager makes this component even care. Then you can probably see how the est of the categories are just combinations of labels.
(Error, When On), (Warning, When Off), (Alert, Consumables)
Friday reads the what from the context data in the alias. So alerts are a matter of tagging an entity and giving it extended alias info.
The takeaway here is - I ABSOLUTELY could have just dumped a list of those entities. But look at what the report is ACTUALLY telling her.
The brackets tell her ‘That’s a space where stuff goes.’ (Informing the LLM about absence of something is often overlooked. And when there’s stuff in here because it’s one of the first things she sees, its high on the priority list and marked system. You BET she barks at me for anything here. And by segmenting by criticality - she arks loudly at important stuff.
My architecture is basically:
- Intents are for atomic things that must always be on - real system level stuff. I design then to be small, and laser focused. Document the crap out of it inside the intent itself. (Pro, doesnt use prompt space, decent context for LLM, con: need to edit multiple yaml files, con: must put any variables in the list - another part to break, con: always on - pro, always on?)
- For day to day get it done, I’m leaning on scripts now. Also document. You can switch to YAML mode to get out of the tiny info box. You’ll see this soon for the mealie implementation. And yes, I’m putting the whole script up. As is no warranty. You break it. Tough…
(scripts: Pro, doesn’t use prompt space, decent context for LLM - if you go to yaml mode and use the boxes there, pro - can disable, pro - dont need to edit system files to declare vars) - But then the heavy hitters - anything that’s gonna need a manual. Is a command at a minimum and (pro - super effective., con big lift.)
- a Kung Fu Component if it grows up. Like Mealie. (ULTRA effective, bigger lift)
Take the battery out of the Flume water system - it will show on this list within 5 mins, and she won’t shut up about it.
In the state above, she knows the Recycle Bin is not where it supposed to be, Rosie’s filter has a problem and we’re waiting on Blue Apron. (And she lets you know if context is appropriate)
Someone asked once how I know she’s not telling me stories - because I’m telling her WHAT to say not how to say it. As long as she has information she will refrain from ‘creating’ her own. SO, our job is to never let her run out of the RIGHT information.
So why Kung fu - and not just build everything in the prompt once?
You get approximately 26000 characters in your prompt. I don’t know the exact number but when you reach the template limit. You’re done. At the end of the day this ALL flows up through the template engine into the prompt and when you hit max template (TOTAL for everything in the prompt window) you stop… Your LLM provider’s context window be damned. You get 26K characters for your prompt right now - that’s it.
The Mealie component (we WILL be talking about next) by ITSELF has a cut down version of Mealie’s openAPI spec in it so Friday knows how to talk to the server (oh yes we have a really cool solution for this - read ahead on rest_command)
We suddenly have to manage space in this thing that when you’re starting out seems enormous.
How do I know you only get ~26K characters? I hit it Saturday. That’s why the delay, I expected to put this up Saturday but I was playing first aid to Friday.
It throws: ‘There was a problem loading your template’ and there will be an error in the log about exceeding max template size.
Other things you may encounter:
Exceeding 1024k on any single TOOL’s (Script, Intent, etc) DESCRIPTION. (Remember I said write a book? Make it a short book) exceeding this on ANY ONE tool will cause the conversation engine to crash and not load. points now you know why the library exists.
These facts are why the Kung Fu system was built. Now I can keep building tools to make Friday smarter - but I dont have to LOAD them all in all circumstances. In the future we will be loading a purpose built version of Friday for various scenarios. Security Friday, Pool Manager Friday, PA Friday and each will have its very own toolbox setup then they load up. Need medical but don’t expect to do menus rt now - load up Dr. Friday and leave Mealie down for now.
Friday knows Kung Fu.