Lets reduce number of LLM tools

Feature Requests
,
1

Problem

All the built-in tools passed to LLM are costing tokens and response time. We have so many tools and many of them could be simplified and merged.

We currently have:


    HassTurnOn
    HassTurnOff
    HassSetPosition
    HassStartTimer
    HassCancelTimer
    HassCancelAllTimers
    HassIncreaseTimer
    HassDecreaseTimer
    HassPauseTimer
    HassUnpauseTimer
    HassTimerStatus
    HassBroadcast
    HassLightSet
    HassClimateSetTemperature
    HassMediaUnpause
    HassMediaPause
    HassMediaNext
    HassMediaPrevious
    HassSetVolume
    HassListAddItem
    HassListCompleteItem

Each one has its parameters defined and passed to LLM like:

{
  "type": "object",
  "properties": {
    "hours": {
      "type": "integer",
      "minimum": 0
    },
    "minutes": {
      "type": "integer",
      "minimum": 0
    },
    "seconds": {
      "type": "integer",
      "minimum": 0
    },
    "start_hours": {
      "type": "integer",
      "minimum": 0
    },
    "start_minutes": {
      "type": "integer",
      "minimum": 0
    },
    "start_seconds": {
      "type": "integer",
      "minimum": 0
    },
    "name": {
      "type": "string"
    },
    "area": {
      "type": "string"
    }
  },
  "required": []
}

Well its a LOT of tokens. 8904 charactes to be exact.

Proposal

Lets merge some of the tools.

E.g.:

HassTurnOn & HassTurnOff

  • Proposed New Tool: HassSetDeviceState (or HassControlDevice)
  • Key Parameter: state (required, enum: "on", "off", possibly "toggle")
  • Rationale: These are fundamentally the same action (changing state) on the same types of entities, just with opposite target states. Combining them is very logical. Adding "toggle" could cover even more use cases.

We could do similarly:

  • HassMediaUnpause, HassMediaPause, HassMediaNext, HassMediaPrevious, HassSetVolume → HassControlMediaPlayer
  • HassListAddItem, HassListCompleteItem → HassManageList (or HassControlList)

and possibly even timers, even though maybe in more than just one tool:

  • HassStartTimer, HassCancelTimer, HassIncreaseTimer, HassDecreaseTimer, HassPauseTimer, HassUnpauseTimer, HassTimerStatus (and potentially HassCancelAllTimers) → HassManageTimer

WDYT?

2

100% all built-in tools should be multi tools wherever possible.

So additional background - In my Friday’s Party experiments I’ve isolated the limits. It’s all in that thread.

But yes 100% build multi use tools when necessary. This includes the core Hass intents. (and had planned this very FR do the exact same reason)

ok one minor edit - entity state not device state, you’ll be manipulating entities. One could also pass special flags to perform specific functions. Entirely possible and part of why I built the library - This is how the library works in Friday

1 tool for Task manipulation rather than 12. Uses 1 slot not 10% of my available tool slots.
You only get 128, and each must have less than 1024k in it’s description.

3

I dont think there is a way to disable the built-in tools and provide our own though, is there?

Btw, I use TheFes’ voice blueprints and it gets even worse there. That’s why I’ve opened the issue: Feature request - make calendar script that takes calendar name as tool parameter · Issue #33 · TheFes/ha-blueprints · GitHub

Just to illustrate what an effect it has - I have 3 calendars with that script and they each take 2300 characters. It does add up!

1 Like
4

I belive we can edit the files they live in directly but I haven’t considered it yet. There’s better ways.

The worst BY far are calendar and tasks. I know exactly why they’re done the way they are. - it’s easy (and the alternate means dynamically handling which vars get passed for the action correctly so there’s a lot of defensive code to write. (if case x and missing blah throw errors)

But we’re gonna have to. Each base domain should have One MAAAAYBE two base intents that are Swiss army knives of that domain type. Because each script you expose is a tool ad when you want fez’s calendar and music scripts. My api scripts for openwebui etc etc. It adds up. Faster than you expect 128 isn’t a lot.

Im getting very seriously close to not having room for a hot tub script and that’s simply not OK. What’s the use of having an advanced AI if you cant drive a hot tub with it?

(and no not kidding. Concrete starts in about 6 hours from now)

5

We could definitely use this, though I don’t have nearly the need for it that others might at this time

1 Like