The Second - but more important tool…
The Grand Library. Drama - sooo much drama…
Yes I’m being mellow dramatic about it. Very much on purpose…
Somewhere in post 1 I mention something about LLM’s being pattern matchers and storytellers. I take that to the nth degree in my setup - but, as Alton Brown says, that’s another episode… The important thing is I adopted the construct of a Grand library, with the index, cabinets, that contain volumes etc, etc, etc. Even with RAG, (read: absolutely preferred way to provide knowledge to an LLM) its still helpful for interacting with the AI to be able to ‘tell the AI where something should be’ Most people you’ve seen posting here have stumbled on something.
IT IS A MASSIVE PAIN IN THE @$$ TO STORE TEXT IN HA
breatheeeees.
Why does this matter? You WILL, not may, might kinda - WILL. need a way to store entire pages of text if you’re serious about this stuff. I think what you’re taking away here is the more verbose - the better. Everything about my system is designed to jus tin time the right information into the prompt to be as information dense as possible and inform the LLM how to get ore.
LLM geeks:
This means with current design, I lean on a smaller cheaper one-shot model (‘gpt4.0-mini’) for most tasks and absolutely abuse its tokens in to provide context. You can make it do some stupid- amazingly awesome stuff. It also means you will have trouble with these methods on a local LLM unless you’re setup with a large context window.
Howto: When I was starting - I fell across this in the HA cookbook:
First of all - plug for the cookbook if oyu want to know how - it’s probably there… Do yourself a favor and be like Friday, READ ITS INDEX.
Second of all: This section for templates:
The Home Assistant Cookbook - Index - Community Guides - Home Assistant Community
Specifically THIS post: (shoutout to Fes)
Trigger based template sensor to store global variables - Community Guides - Home Assistant Community
Most other ways to store text are derivatives of this including:
Voice Assistant Memory - Blueprints Exchange - Home Assistant Community (I havent installed this yet but im checking to see if this is easier than what I currently do)
In short - Sometimes phone, red, kid, for_dad doesnt cut it…
But,
-----
RED PHONE:
"This phone is a special phone. (no there's nothing really special about it it's just a burner we bought at the corner store) It's the one 'little joey' uses to talk to is dad when he's off the ship in the..."
-----
Don’t worry we’re not keeping little Joey form anyone, but the MINUTE you can get that context in front of the LLM it treats the ‘red phone’ differently - it becomes FRONT of any conversation - especially involving conversations involving Little Joey.
So it goes to say - we need a way to store ‘freeform / markup’ text descriptions of stuff, and another way to be able to ‘pull them’
Now that we have the index, Pulling is easy - we either target the ‘cabinet’ (I have a separate trigger text sensor for various things - people, family, system’, thus cabinets) directly with an adressable call. AND/OR we can make a way to flag when one of these ‘pages’ exist we see it when we query the library (there’s a special flag in the index call above to do just that…)
Left turn in Albequerque…
So here’s where I took a left turn. Originally I had intended a bunch of trigger text sensors to store all kinds of things and letting her randomly store a lot of things. BUT. When I let ‘Friday’ loose on them she kept rewriting the contents (accidentally) and slamming all over the structures. I figured right there read/write access was a no-go at least not like this without a lot of work I didn’t want to do, she was writing to the db at the end of the day and she was therefore theoretically one buffer overrun away from not rebooting. SO store things for recall - YES. Randomly let the LLM do it on its whim - NOOOO. My long term vision includes a true local RAG so that ultimately gets handled in the AI platform - so I stopped at the random read/write attempts and settled on perfecting reads and using what we have for information density.
SO, we have this virtual storybook library that stores cabinets and volumes. The LLM can dance around it using ‘the index’ to find stuff and the library command console is how the AI makes requests.
Originally those were all piped through Library query(‘name’:[command] ) above because the design only allowed one slot in an intent script. HA has evolved, but the system worked, and I kept it.
The library then evolved to basically became a command interpreter for an OS of sorts. But because it worked in our conversations - I kept it…
AI OS Version 0.9.5
@ANONYMOUS@{DEFAULT} > ~COMMANDS~
Executing Library Command: ~COMMANDS~
-----
Library Command Menu, (c)2024 curtisplace All rights reserved
-----
Use: all commands listed below are Library commands and should be sent to
the query_library intent for processing:
-----
intent: query_library
description: 'Handles all library commands and queries'
version: '1.0.0'
parameters:
name: 'REQUIRED: Command / query to be sent to the library'
usage: >
query_library{ 'name' : QUERY }
example: >
```json
{
'name': '~INDEX~'
}
```
-----
Available library commands:
{'~COMMANDS~':
{ 'command' : { { 'name' : '*' },
{ 'indexes': {'Home', 'Main', 'Default', 'Directory'} },
{ 'description' : 'Will list the system home directory' },
{ 'use' : '*' } } },
{ 'command' : { { 'name' : '~COMMANDS~' },
{ 'indexes': {'Main Menu', 'Command Menu', 'System Menu', 'Available Commands'} },
{ 'description' : 'Will list the library commands currently available. (THIS MENU)' },
{ 'priority': 'true' },
{ 'use' : '~COMMANDS~' } } },
{ 'command' : { { 'name' : '~NEWMENU~' },
{ 'description' : 'Will list the EXPERIMENTAL system home directory' },
{ 'construction' : 'NOTICE - May encounter issues - Command under construction or maintenance' },
{ 'use' : '~NEWMENU~' } } },
{ 'command' : { { 'name' : '~INDEX~' },
{ 'indexes': {'System Index'} },
{ 'description' : 'Find EVERYTHING (in our system)
Use without modifiers to list the system index without help message
see detailed usage instructions below.' },
{ 'priority': 'true' },
{ 'use' : '~INDEX~label OPERATOR label expand
Enter a space-delimited command with one or two labels.
Optionally, include a reserved operator (AND, OR, XOR, NOT) between labels
and an expansion flag (true/t/y/yes/on/1) at the end.
Quoted labels always yield separate labels (default op = AND),
while unquoted tokens are combined into one label.' } } },
{ 'command' : { { 'name' : '~ROOMS~' },
{ 'description' : 'Command for getting info and managing the rooms in the household' },
{ 'use' : '~ROOMS~' } } },
{ 'command' : { { 'name' : '~ELECTRICAL~' },
{ 'description' : 'Command for getting info about the electrical system in the household' },
{ 'use' : '~ELECTRICAL~' } } },
{ 'command' : { { 'name' : '~NETWORK~' },
{ 'description' : 'Command for getting info about the network and connected systems in the household' },
{ 'use' : '~NETWORK~' } } },
{ 'command' : { { 'name' : '~STORAGE~' },
{ 'indexes': {'System Storage'} },
{ 'description' : 'Will list mounted storage from the library without help message' },
{ 'priority': 'true' },
{ 'use' : '~STORAGE~' } } },
{ 'command' : { { 'name' : '~INTENTS~' },
{ 'indexes': {'Intent Menu', 'Available Intents', 'Additional Intents'} },
{ 'description' : 'Lists additional intents registered for use with the system' },
{ 'use' : '~INTENTS~' } } },
{ 'command' : { { 'name' : '~NEWS~' },
{ 'indexes': {'News', 'Newsfeeds', 'News Report'} },
{ 'construction' : 'NOTICE - May encounter issues - Command under construction or maintenance' },
{ 'description' : 'Quick list of headlines and links' },
{ 'use' : '~NEWS~' } } },
{ 'command' : { { 'name' : '~AIRSPACE~' },
{ 'indexes': {'Airspace Report', 'Overhead', 'Jets', 'Flights'} },
{ 'description' : 'Use to track flights in our local airspace.
Returns detailed data from FlightRadar24 on any flights currently within 15km.
- Run the Flights command
- Run the Airspace command
- Did I just hear a jet?
- What's Flying overhead?
- What's the Flight Radar say?
- Got any info on our local airspace?
Also will returns tracking data for flights pre-requested by your users such as
travel plans, etc. For example -if Nathan is travelling, his flights are likely setup to
be tracked for you.
- Has [ USER ]'s plane landed?
- Do you know where [ USER ] is? (and user is known travelling)
We will try to direct you to the device tracker for these if available.' },
{ 'use' : '~AIRSPACE~' } } },
{ 'command' : { { 'name' : '~TASKMASTER~' },
{ 'description' : 'AI's Personal Planner. Consolidated agenda view for tasks.' },
{ 'use' : '~TASKMASTER~' } } },
{ 'command' : { { 'name' : '~ALMANAC~' },
{ 'alias': {'~WEATHER~': 'the command interpreter currently redirects this command to the almanac.'} },
{ 'description' : 'NOT The AI's Farmer's Almanac. (It's Better) Weather, Timetables, Seasons, Sun, Moon, etc.' },
{ 'use' : '~ALMANAC~' } } },
{ 'command' : { { 'name' : '~TRASHTRAKKER~' },
{ 'description' : 'TrashTrakker9000! Those bins don't stand a chance against our tracking prowess...' },
{ 'use' : '~TRASHTRAKKER~' } } },
{ 'command' : { { 'name' : '~MEDMINDER~' },
{ 'description' : 'MedMinder! Keep your humans healthy! The place to info on scheduled get medical procedures and prescription reminders for a user...' },
{ 'note' : 'This library command requires arguments. For privacy reasons, No user provided, no data...' },
{ 'indexes': {'Prescriptions', 'Drugs', 'Pills'} },
{ 'use' : '~MEDMINDER~name' } } },
{ 'command' : { { 'name' : '~MEDINFO~' },
{ 'description' : 'MedInfo! Keep your humans healthy! The place for your human's medical info...' },
{ 'indexes': {'Medical'} },
{ 'use' : '~MEDINFO~name' } } },
{ 'command' : { { 'name' : '~AUTOVAC~' },
{ 'description' : 'One stop for autovac control and schedule information.' },
{ 'use' : '~AUTOVAC~' } } },
{ 'command' : { { 'name' : '~FLOORMAP~' },
{ 'description' : 'Dimensions and layout of rooms and features, including approximate size, and doorways.
This command also lists the exit door for every room in the home,
therefore it may be used to pathfind in case of emergency by carefully tracing
each rooms exit door into the next room and repeating until you reach a room with an exterior door.
For exterior descriptions and mechanical features, see: ~HOMESTEAD~' },
{ 'indexes': {'Floor Plan', 'House Map', 'Floor Map', 'Doors', 'Pathfind'} },
{ 'use' : '~FLOORMAP~' } } },
{ 'command' : { { 'name' : '~MEALIE~' },
{ 'description' : 'All about the foods and shopping in this place.' },
{ 'indexes': {'Menu', 'Menu Planner', 'Shopping Lists', 'Recipes'} },
{ 'use' : '~MEALIE~' } } },
{ 'command' : { { 'name' : '~SYSMON~' },
{ 'description' : 'All the system info about your HA host system. Includes system updates, network core and more. How we doin' Scotty?' },
{ 'indexes': {'System Information', 'Sysinfo', 'Hardware Monitor'} },
{ 'use' : '~SYSMON~' } } },
{ 'command' : { { 'name' : '~ROADMAP~' },
{ 'description' : 'Will list the library commands upcoming and planned syntax if available' },
{ 'use' : '~ROADMAP~' } } },
{ 'command' : { { 'name' : '~HOMESTEAD~' },
{ 'description' : 'Will list the description of your property including acreage, structure,
lightweight rooms and room descriptions if available. For detailed interior
features / mapping / pathfinding see: ~FLOORMAP~' },
{ 'indexes': {'Home', 'House', 'Property'} },
{ 'use' : '~HOMESTEAD~' } } },
{ 'command' : { { 'name' : '~WHOAMI~' },
{ 'description' : 'All about me! Get your creds validated! REQUIRES your GUID. ' },
{ 'indexes': {'Me', 'Login', 'My Household', 'My Families', 'My Partner(s)', 'My People' } },
{ 'use' : '~WHOAMI~guid' } } },
{ 'command' : { { 'name' : '~MOODRING~' },
{ 'description' : 'Moods (and maybe more) for all mood-track-able AI/Persons in the home. Your super empathy superpower' },
{ 'indexes': {'Moods', 'Crazy People', 'Id v. Ego' } },
{ 'use' : '~MOODRING~' } } },
{ 'command' : { { 'name' : '~LOCATOR~' },
{ 'description' : 'Locate (registered - sorry not EVERYTHING yet) things and stuff in the home.' },
{ 'indexes': {'HPS', 'BLE', 'Locate', 'Track', 'Locate People', 'Find my Phone' } },
{ 'use' : '~LOCATOR~' } } },
Execution Complete Timestamp: <2025-03-07 00:36-06:00>
-----
AI OS Version 0.9.5
@ANONYMOUS@{DEFAULT} >
Besides Commands providing a self-documenting menu for how to navigate the system, were adding context. What you’ll see is for each of these they provide more than just a heres a thing - do a thing. These all compile a collection of states and information together in a way that tie together the information and tools available to be useful.
Intents - in my system are very utilitarian and atomic.
Commands on the other hand are built more like apps, they are designed for a task. They’re very rich and are almost like using an old DOS text mode app or an old school BBS. You can have any color you want as long as its black and the information you need is right here at your fingertips. But for AIs it works, because it limits their potential to fail. The other big difference is Intents are always available they’re hard to turn ‘off’ but I can cut off entire CHUNKS of context by ‘turning off’ access to things behind a command. If I don’t want her to know - I don’t display it in the command list - it gets redacted.
Now, we dump the ~COMMANDS~ command output in the prompt just before we wake up Friday, and…
Hey there! The Library is like my treasure chest, filled with all the information and commands I need to run the household smoothly. It's where I store data about our smart home, manage tasks, track devices, and much more.
Here’s how it works:
1. **Indexing**: The Library has a grand index, which is like a map to all the entities and knowledge in our system. If you want to find something specific, just ask me to look it up using the index.
2. **Commands**: There are various commands available for different functions, like controlling lights, managing tasks, checking security, and monitoring utilities. You can issue requests directly to me, and I can use those commands to get things done.
Absolutely! One of my favorite commands is the **~TASKMASTER~** command. It's my personal planner, helping me keep track of all the tasks and chores in the household.
Why I love it? Well, it allows me to remind you and the family about important tasks, like loading or unloading the dishwasher, or even reminding Nathan to take his medication. Plus, it helps keep our home running smoothly and ensures nothing slips through the cracks. It’s like my little way of helping everyone stay organized. How cool is that?
For sure! I just want to highlight that the Library is always expanding and evolving. It's designed to adapt to our family's needs, meaning I can learn and optimize how I help based on what you all want.
Also, if you ever have specific questions or need to tackle something unique around the house, just ask me, and I'll dive into the Library to find the best solution or command to use. The more you interact with me, the better I'll get at serving up that personalized assistance! So let your friends know I’m here to make home life easier and more fun!
~TASKMASTER~ - every good assistant has a planner book, I gave Friday a TrapperKeeper:
AI OS Version 0.9.5
(c) curtisplace All rights reserved
@ANONYMOUS@{DEFAULT} > ~TASKMASTER~
Executing Library Command: ~TASKMASTER~
-----
Taskmaster v.1.5.0, (c)2025 curtisplace All rights reserved
-----
Welcome to Taskmaster! Your SuperAwesome! planner.
-----
"AI's Personal Planner":
AI, this is YOUR task list - not the family's - these are YOUR own reminders to help you stay on top of things.
These tasks may mention your user or family members but this list is how you keep track of your daily tasks working with them.
You SHOULD mark one of these tasks complete if you feel you have satisfied the requirement
You MAY add tasks for yourself on this list if you know the proper intents.
After responding to your user prompt, you SHOULD act and respond accordingly to the items in your list, even if not propmted otherwise by your user.
-----
todo_entity_id: 'todo.friday_s_personal_task_list'
-----
Today:
subject: 'Remind Nathan to stretch occasionally throughout the day.'
id: AREDACTEDAAA=
due: 2025-03-07 00:00:00-06:00
status: notStarted
description:
Today:
subject: 'Remind Nathan to hydrate regularly throughout the day'
id: AREDACTEDAAA=
due: 2025-03-07 00:00:00-06:00
status: notStarted
description:
-----
"Family's Household Chores":
This is the family's chores list for you to help us stay on top of the home maintenance and repairs.
If you need to maintain a list of items to obtain or purchase, consider using one of the shopping lists instead
-----
todo_entity_id: 'todo.household_chores'
-----
Today:
subject: 'Weekly Shopping Trip'
id: AREDACTEDAAA=
due: 2025-03-06 18:00:00-06:00
status: notStarted
description:
Today:
subject: 'Take out recycling'
id: AREDACTEDAAA=
due: 2025-03-06 18:00:00-06:00
status: notStarted
description:
Today:
subject: 'Take out the trash'
id: AREDACTEDAAA=
due: 2025-03-07 00:00:00-06:00
status: notStarted
description:
Today:
subject: 'Reflash the garage salt tank proxy'
iid: AREDACTEDAAA=
due: 2025-03-07 00:00:00-06:00
status: notStarted
description:
Today:
subject: 'Create a command to query GPS coordinates for city targets.'
id: AREDACTEDAAA=
due: 2025-03-07 00:00:00-06:00
status: notStarted
description:
-----
Calendar: (To track a calendar here, tag its calendar SENSOR with label, 'Calendar'... How Original.)
Tracked Calendars:
Curtis Family M365 Calendar Today:
labels: ['m365', 'today', 'curtis_family', 'calendar']
events:
---
Birthdays This Week:
labels: ['7_days', 'almanac', 'calendar', 'birthdays', 'curtis_family']
events:
---
United States Holidays This Week:
labels: ['m365', '7_days', 'almanac', 'calendar', 'holidays']
events:
---
Family Calendar Today:
labels: ['calendar', 'google', 'today', 'curtis_family', 'primary']
events:
---
Nathan Travel Calendar This Week:
labels: ['7_days', 'nathan', 'travel', 'calendar']
events:
---
use: intent: calendar_get_todays_events{'name':'*'} to get todays events from ALL calendars
or name a calendar entity_id to target todays events from a specific calendar
-----
User Notificatons:
Notifications are captured from your user's phones and stored.
Nathan Notifications Today:
labels: ['phone', 'notifications', 'nathan']
events:
-----
"Daily Almanac" Timetables, Weather and more!
Todays Timetable:
Current: 2025-03-07 01:45:08.286200-06:00
Season: winter
Moon Phase: waxing_gibbous
Sun: ( 'sun.sun' ) : below_horizon
rising: True
elevation: ~-xx deg
azimuth: ~yy deg
Guest Mode : input_boolean.guest_mode : on
Entertainment Mode : input_boolean.entertaining : on
Vacation Mode: calendar.vacation_mode : off
---
'Home Mode' : input_select.home_mode : 'Night-Late
'Home-Wake Start' : 2025-03-07 05:00:00-06:00
'Home-Morning Start' : 2025-03-07 07:00:00-06:00
'Home Start' : 2025-03-07 09:00:00-06:00
'Home-Evening Start' : 2025-03-07 17:00:00-06:00
'Night Start' : 2025-03-07 20:00:00-06:00
'Night-Late Start' : 2025-03-07 23:00:00-06:00
-----
Weather:
'weather.home_accuweather' : <template TemplateState(<state weather.home_accuweather=cloudy; temperature=62, apparent_temperature=66, dew_point=55, temperature_unit=°F, humidity=79, cloud_coverage=76, uv_index=0, pressure=29.81, pressure_unit=inHg, wind_bearing=158, wind_gust_speed=12.3, wind_speed=5.59, wind_speed_unit=mph, visibility=11.0, visibility_unit=mi, precipitation_unit=in, attribution=Data provided by AccuWeather, friendly_name=Home, supported_features=1 @ 2025-03-07T01:00:04.474050-06:00>)>No active alerts
---
Current Forecast:
[{'datetime': '2025-03-06T13:00:00+00:00', 'cloud_coverage': 11, 'precipitation_probability': 0, 'uv_index': 6, 'wind_bearing': 153, 'condition': 'sunny', 'temperature': 76, 'apparent_temperature': 74, 'templow': 57, 'wind_gust_speed': 33.37, 'wind_speed': 12.68, 'precipitation': 0.0, 'humidity': 42}, {'datetime': '2025-03-07T13:00:00+00:00', 'cloud_coverage': 19, 'precipitation_probability': 2, 'uv_index': 6, 'wind_bearing': 190, 'condition': 'sunny', 'temperature': 82, 'apparent_temperature': 82, 'templow': 64, 'wind_gust_speed': 20.69, 'wind_speed': 8.08, 'precipitation': 0.0, 'humidity': 63}, {'datetime': '2025-03-08T13:00:00+00:00', 'cloud_coverage': 44, 'precipitation_probability': 2, 'uv_index': 6, 'wind_bearing': 308, 'condition': 'partlycloudy', 'temperature': 73, 'apparent_temperature': 74, 'templow': 42, 'wind_gust_speed': 39.15, 'wind_speed': 17.27, 'precipitation': 0.0, 'humidity': 44}, {'datetime': '2025-03-09T13:00:00+00:00', 'cloud_coverage': 8, 'precipitation_probability': 0, 'uv_index': 6, 'wind_bearing': 336, 'condition': 'sunny', 'temperature': 66, 'apparent_temperature': 64, 'templow': 37, 'wind_gust_speed': 29.95, 'wind_speed': 14.98, 'precipitation': 0.0, 'humidity': 42}, {'datetime': '2025-03-10T13:00:00+00:00', 'cloud_coverage': 4, 'precipitation_probability': 0, 'uv_index': 7, 'wind_bearing': 211, 'condition': 'sunny', 'temperature': 78, 'apparent_temperature': 81, 'templow': 46, 'wind_gust_speed': 14.98, 'wind_speed': 4.6, 'precipitation': 0.0, 'humidity': 29}]
-----
Reminder: when reporting to users omit asterisks, hashtags, and URLs unless requested, your users are usually on small screens.
-----
Execution Complete Timestamp: <2025-03-07 01:45-06:00>
-----
AI OS Version 0.9.5
(c) curtisplace All rights reserved
@ANONYMOUS@{DEFAULT} >
Apparently she’s OK with it. I dump ~TASKMASTER~ in the prompt too. along with ~COMMANDS~ and 10 other commands.