Voice Assistant, Ollama, prompt - how to control?

Configuration Voice Assistant
1

Is there a way to adjust the ollama integration’s prompt in areas not exposed?

I noticed in the system part under debug, a few pages of prompt not in my prompt, and some portions are a bit at odds with mine. Is there any way to get to that? Change it?

I’ve tried the Extended Conversation OpenAI integration and had all sorts of issues getting it to work at all (though maybe I’ll try again).

I’ve tried the OpenAI and it apparently is only for cloud based use.

Is there a supported way to take control over the prompt, I think HA is trying to “help” me and the conflicts are giving the LLM a headache.

2

Define ‘take over the prompt’ you’re using some… Unique vocabulary…

What do you want success to look like

You can control.

  • Exposed entities and/or thier aliases
  • Exposed Tools
  • The Integration prompt (let’s call it user mode prompt)

You cannot control (usually, at least in core Integrations)

  • HA system prompt (in most integrations there will be some things the integration sends. HA core definitely does.)
  • Which intents to expose (again in most these two would DEFINITELY need a custom integration. Tool descriptions here are the source of most conflicts)
3

That’s because I have almost no idea what I’m doing. :wink:

I had been working on a prompt over a few days using the Ollama integration thinking that prompt was complete (see, that “no idea”). I finally noticed in the big green bar in debug that there’s an addendum.

I’m not sure how much it matters, but some things seem to conflict. I had things in mine trying to limit it to either acting on an explicit area (and domain), an explicit entity, or all. I did not want it to presume the area in any case, but the added prompt includes:

You are in area Office and all generic commands like 'turn on the lights' should target this area.

I suspect that’s why occasionally “all of the lights” becomes just my office, but not sure, especially if it doesn’t hear the word “all”.

I also had to work on a case where Qwen3.5:9b was giving answers like this, where it would turn on one light then say:

I would need to continue this action for each of the 39 individual lights...

I suspect but am not sure that it’s the static content that might have hinted at that sort of response. Maybe. Though providing the static content rather than continually querying for it seems like a good idea. It took a few iterations of explicit instructions before it stopped doing that.

I guess I was just surprised that there were prompt like things hidden (well, let’s say not obvious). I would not have been surprised to see it feeding the static content in alone.

But… I’m curious why Ollama integration provides this stuff and (sort of) hides it, or at least seems to discourage you from changing it?

4

Because most people want set and forget.

There are some basics that must be in to be functional.

Also I cover most of this in Fridays party if you stick with the read. Yes it’s long. If you want to skip to end you can use any or all of that work.

Most of your success comes from building good tools and clarity in naming. Trying to override behaviors in the main prompt is a losing battle. The user prompt should be more rules of the road.

Targeting comes from clarity in naming and aliases.

Most of us that do this find some kind of index/lookup trumps just stuff it all in the prompt.

5

Thanks.

For the record it passed “Long” three longs ago. :rofl:

I’ve made it through some of it, found your (I think your?) prompt creater, which gave me a start (since pretty changed).

Your point is fair enough. I think what really bothered me is I didn’t know it was in there. I’ve never appreciated software that protected me from myself silently.

6

Skip to the end. Link to the github. Read that version if you’re trying to reverse engineer. I changed a LOT between last March and September.

7

Have you seen GitHub - skye-harris/llm_intents: Exposes internet search tools for use by LLM-backed Assist in Home Assistant · GitHub?

It lets you set different tools which have the ability for custom prompts.

8

I’m a fan. I’m evaluating thier integration set in my build. (using Nik’s voice recipes.)

9

I had to laugh at this comment coming from you Nathan…

10

Truth, and. You understand every word.. Eventually. Tell me you don’t know sand dune plinko. :slight_smile:

11

Thanks for that. I spent some time looking over it, and it hits a lot of places I think I want to go (including on point for this topic). Going to try to get stable on this setup first and understand it. I put Extended OpenAI in which (I think maybe) lets me see the whole prompt, and lost some tool use for reasons a bit unclear.

Got a lot to learn. Also, as an old-school software guy, I really want to understand it all, but I guess a lot of the LLM process is by definition not understanding what it does, just guessing why. But I also raised a child, and finding the right prompt to get the behavior I want at least some of the time is much more familiar than from my programming experience.

12

Think of teaching a rather well behaved 9yo. But they need impeccable instructions.

The rip is. Yoo don’t have enough room for all the instructions your 9yo needs so you need to get creative on how you send information. You need to get as much accurate information and instructions in and disambiguate. In as few words as possible. At the end that’s the whole game.

But you’re right there’s structure upon structure like Russian nesting dolls. And the inner ones you don’t see matter most.

13

One of the things I was noticing today is that Ollama (the integration) does not set a temperature, or at least it appears not to. And the model I was using is 1.0. I changed it in the model file to 0.1, and in some brief tests it appears more consistent. (It went a bit nuts at one point using setbrightness to turn on a light, or calling hasturnon an array instead of specific values for … I think it was… area (despite one value in the array). Then like magic it started working BEFORE I changed to 0.1, but it seemed like less randomness was a goodness.

It does beg the question though… the only way I felt I could tell where the 1.0 came from (i.e. hard coded in the integration or from the model) for sure was to change it and see if it stuck. And it did change so it wasn’t from the integration, but there are a fair number of other parameters.

Is there a specific place to look to tell what comes from here? The log just showed the change, did not appear to indicate a source. Or should I interpret the lack of settable parameters to mean all come from the model file?

slot get_availabl: id  0 | task -1 | selected slot by LCP similarity, sim_best = 0.997 (> 0.100 thold), f_keep = 0.998
slot launch_slot_: id  0 | task -1 | sampler chain: logits -> penalties -> ?dry -> ?top-n-sigma -> top-k -> ?typical -> top-p -> ?min-p -> ?xtc -> temp-ext -> dist
slot launch_slot_: id  0 | task -1 | sampler params:
    repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 1.500
    dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 15104
    top_k = 20, top_p = 0.950, min_p = 0.000, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, top_n_sigma = -1.000, temp = 0.100
    mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000, adaptive_target = -1.000, adaptive_decay = 0.900
14

I wanted to return to this and say thank you. I wanted to experiment further before adding yet another variable, but this (notably the very last feature in the doc) appears to be exactly what I need if I want to override the OIlama’s prompt addendum. I haven’t done so yet, but put it in vanilla and saw that things still worked.

But yes, this seems to be the real answer to my original question. (Begs the question if I really NEED to modify it, or if it’s something else, but now I CAN modify it).

Incidentally and sort of related: the Extended Conversation OpenAI seems to also allow this, but I could not get it to work calling scripts with parameters, it just seems to have a built in presumption that a script can simply be activated, and no parameterization of the tool. But the llm_intents seems to solve the problem without creating others (at least so far). And I can implement other features now if I choose (one thing at a time though).

15

Extended OAI solved a problem with things the builtin OAI could not solve in the beginning. Specifically, the ability to change the endpoint. It has its own tool calling surface that hasn’t aged well. Once the real oai started doing tool calling I dropped extended - too much trouble.

I use local LLM for connecting to my local server. And am considering ^^^ for my zenos builds in the future.

Things I want to personally change

The default prompt should never call out tools to use just basic grounding. Some of the integrations don’t do it the same way. This leads to tool call conflicts when you have something different and in conflict than the integration system prompt. Thier words win.

If i want to override /takeover the builtin intents for things like HassTurnOn and add an RBAC envelope I can’t do it with most LLM . That one does.

The combo would allow me to replace HassTurnOn (disable all the tools I don’t want and turn off assist first) then provide my own. Without the prompt telling the agent to just look everything up with get live state and turn it on with Hass turn on. (default is setup as a wide net sledgehammer. I want a scalpel)

16

You’re welcome! I’m very glad that it seems to do what you want.

17

Apologies but a terminology question.

I know of the Extended Open AI (mentioned above).

I know of Ollama (the integration).

I know of the llm_intents that seems to intercept some of but otherwise use the Ollama integration.

I know of the OpenAI Integration which seems restricted to the cloud.

Is the “real oai” you mentioned the cloud integration, and what is the “local LLM” mentioned, is that yet a different one that replaces Ollama (the integration)?

18

‘real’ in your question is the core provided one

Skye also… Custom integration for connecting to local LLM providers.

You would use this instead of the core oai provider

You’ve notices lots of puzzle pieces and they’re not all the same shape.

19

Ah, well, jigsaw puzzles I’m good at. I had one that said “2-6 years” on the cover, and it only took me a few months. :wink:

And sometimes I think it’s bugs somewhere I can’t see. I just spent hours chasing a tool call formatting error in XML, then restarted ollama.cpp (with no changes to it or models or anything else, just turned on debug) and the problem completely vanished. Sigh…

Jigsaw pieces where the final result is a photo of a field during a snow storm whiteout. :rofl:

20

In 2026.8 there’s a new llama.cpp integration. https://www.home-assistant.io/integrations/llama_cpp

I haven’t tried it (why is another story though I’m going to snapshot and test at some point), but…

Is this sort of a replacement Ollama integration + llm_intents? It sounds like it’s meant as a way to expose more control over prompt, temperature, etc., but in particular prompt which is why I was using llm_intents.

Except it jumped from non-existent (?) to core integration, so at least in theory the under-the-covers integration with HA relative to tools exposure, etc. may be more maintained now?

If not, anyone offer a translation of what it’s purpose in life is relative to Ollama (the integration)?