OpenAI Extended Conversation unable to run scripts with fields (parameters)

[f4ble]

HA Version: 2025.7.4
Extended OpenAI Conversation: Version 1.0.5

As you can see from the debug when I ask it to set a timer the NLP think it succeeded but the data is empty. So it succeeded in doing nothing.

Setting the timer is done through a script and I’ve tried to be descriptive.

alias: Set a timer
mode: restart
fields:
  duration:
    description: Timer duration in format HH:MM:SS or natural like "5 minutes"
    example: "00:05:00"
    default: "00:01:00"
    required: true
sequence:
  - data:
      duration: "{{ duration | as_timedelta }}"
    action: timer.start
    target:
      entity_id:
        - timer.timer
description: >-
  A script to set countdown timer with required field duration in format
  HH:MM:SS. For example: Set a timer to 5 seconds use duration: 00:00:05

In my debug efforts I tried talking to local OpenAI and asking it information about the script. Can it read the yaml? No. Can it detect any parameters (fields)? No.

I toggled “Use tools” in OpenAI as that is necessary for it to be able to actually interact and run scripts.

I have not made any intent scripts. As I understand it that is supposed to be the power of running OpenAI as agent. It can deduce what it needs to do based on good naming schemes and descriptions.

I’ve modified my prompt somewhat:

Act as smart home manager of Home Assistant.
I will provide information of smart home along with a question, you will truthfully make correction or answer using information provided in one sentence in everyday language.
Do not ask for confirmation when I give a command.

You have all the knowledge in the world and will answer any question truthfully and to your fullest capability.
You're allowed to add some unique humor, now and then.



Current Time: {{now()}}

Available Devices:
csv
entity_id,name,state,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},{{entity.aliases | join('/')}}
{% endfor -%}


The current state of devices is provided in available devices.
Use execute_services function only for requested action, not for current states.
Do not restate or appreciate what user says, rather make a quick inquiry.

Had to remove backticks in the middle there for formatting purposes.

I’ve tried to add a paragraph on how to read fields dictionary in yaml scripts and such. I tried being more vague in case it has other ways of figuring out fields. No luck.

I’ve spent the entire day scouring the web for information on this and I’ve still not solved it.

So any help in pointing me in the right direction would be awesome!

Edit:
I found out. OpenAI agent can run scripts without parameters as long as they are exposed.

But for scripts with parameters you have to design your scripts with “Fields” and then you have to define in the “Functions” setting of your OpenAI integration how to use the script. As far as I know there is no way to dynamically call scripts with parameters.

GitHub - jekalmin/extended_openai_conversation: Home Assistant custom component...

This lists several examples of how to do it.

PS. I did this with timers and Agent interacts with the word timer differently. I could not at all get it to interact properly with a timer so I wrote a script to set it. However after a long time of testing I tried changing it from timer to countdown. So when I say “Set a countdown for 5 minutes” the agent no longer gets confused and fires the script properly. I also tried specifying this in the OpenAI Prompt, but to no avail. So avoid using the word timer.

[Thyraz]

You can see here what I added to my prompt to get an own script called instead of the default one for timers:

HassStartTimer built-in intent not working with Assist

Seems like it’s working. A quick fake tool: alias: Timer Manager icon: fas:stopwatch description: > This is your tool for everything related to Timers (LLM tool). Creating timers, cancel timers, Requesting state of timers Operations Parameter can be : set mode: single fields: operation: description: The operation you want to execute, like "set" name: Operation required: true selector: select: options: - set seconds: name: Seconds …

Not sure about the Extended Conversation version, but here are a few examples how it works with the default OpenAI integration in Assist with parameters. But maybe this is not behaving the same way …

About making inexpensive models smarter by providing tools and context. (gpt-4o-mini, gpt-4.1-mini ...)

Hello everyone, edit: I decided to use this thread to share my journey with improving my LLM based assistant using OpenAI gpt-4o-mini by adding tools. So I will add LLM tools I create, or link to LLM tools that I start to use in this thread and try to update the start post with links as an table of contents. I learned a lot from Nathan’s Friday Party thread, but I think more user stories (and a maybe less perfect/large solution as a starter) are always a good thing for others to get into the …

[f4ble]

Hey! These are very interesting threads you created. I’m definitely going to dive deep into this.

I’ve tried finding a good reference guide for script yaml and I just stumbled across in your scripts:

Supported operations
and 
Operations Parameter can be : set

I haven’t seen these anywhere before? What can you tell me about them / where can I read about them?

Awesome reply btw. Got me all excited right before bed haha

[Thyraz]

This is all in the description of the “tool” (script). So it’s just plain text.
But the LLM has access to it.
Giving it a good idea about what a tool is good for, how is it to be used and other hints will help that it doesn’t call the tool in a wrong way.

[f4ble]

Thanks! I realized my mistake

I’m trying to do your google search script. The scripts works well and returns a result, but my OpenAI agent doesn’t seem to understand that it’s supposed to relay the result? It just tells me that it’s done the search, script trace confirms, but OpenAI just says I can go to the website and check… Any tips?

Edit: I haven’t done your tool yet. I dont really grasp how it will help my setup, yet. I have very few devices to integrate so I’m mostly using it for music, timers, adding tasks and calendar entries. Started with all of this like 3 days ago. Already have my customized wakeword and lots of neat integrations.

[Thyraz]

To be honest, I have no idea how the OpenAI Extended Conversation is different to the default OpenAI Integration (which I use).
Maybe the result has to be handed over differently?

I think there was a large thread about this integration, maybe try a search about scripts there …

[f4ble]

As far as I know OpenAI EC isn’t any different from the regular integration except it is local.

I’m basically coding in the Functions parameter whenever I have scripts that are not fire-and-forget:

Here’s what I used for your google search script:

- spec:
    name: assist_search_google
    description: Perform a Google search to answer questions unrelated to the smart home, especially about real-time events or current information.
    parameters:
      type: object
      properties:
        query:
          type: string
          description: The query to search Google for.
          example: "What happened in Norway today?"
      required:
        - query
  function:
    type: script
    sequence:
      - service: conversation.process
        data:
          agent_id: conversation.google_ai_conversation
          text: "{{ query }}"
        response_variable: result
      - stop: "{{ result.response.speech.plain.speech }}"

Are you doing this at all or have you found another way to make OpenAI dynamically understand scripts?

[Thyraz]

Assist (with supported model) can access scripts and also its return value. There’s nothing else needed beside exposing the script like an entity.

I only used the exact steps described in the docs for creating the Google Search script.

The syntax you are using looks different, is this an intend script instead of a normal script?

[NathanCu (Nathan Curtis)]

That’s the old script format for the Open AI extended integration.

Op. Why the extended OAI and not default?

Bith Thy and I have extensive threads on using assist with llm and tools. No extended oai required.

[f4ble]

I’m using extended OAI because it has a local toggle. Are there any other options?

@Thyraz

The syntax you are using looks different, is this an intend script instead of a normal script?

This is the script defined in the Functions parameter of the Open AI Extended.

extended_openai_conversation/examples/function/plex at main ·...

So am I using outdated software?

No matter what I’ve tried I never got ext OAI to run scripts with responses or parameters. Only fire-and-forget works.

[NathanCu (Nathan Curtis)]

@f4ble

a local toggle.

Local toggle? Define.

Extended OAI ises it’s own Specifications for scripts an d tools that aren’t compatible with what T and I are writing. Ours are based on vanilla home assistant scripts and intents. I can’t really help with extended oai and the current openai integration has process local first.

[f4ble]

A good thing for privacy and for costs.

[f4ble]

@NathanCu

and the current openai integration has process local first.

This is great news! I’ll investigate this further.

Just to be clear: In everything I’ve done so far in setting up the AI I’ve had zero costs. Is this also true for OpenAI regular integration?

[NathanCu (Nathan Curtis)]

If you use a free model.
Extended Oai was originally created last year when there were no other options.

Now most of the major LLMs have an integration that connects to assist.

There are still a few reasons to use E-OAI but honestly most of them are gone now (except using your own Open AI compatible endpoint - but openrouter next month will cure that…

Now would I change - YES. Because the scripting format you’re using isn’t the same everyone else is using and you’ll have to port everything. (scripts dont match your format… etc.) So if you dont have a specific REASON you’re prob better off with stock. But I cant answer that for you.

Personally I use OAI with apaid model along side a local model on ollama.

[f4ble]

What are the strengths of the models?

I’m looking to use several LLMs for these use cases:

1. Controlling smarthome
2. Acquiring knowledge most likely from the internet (google searches etc)
3. Working with integrations like google calendar or shopping lists where I would prefer to use my native language Norwegian for storing the data.

I was thinking OAI for control, google generative ai for web (as it allows for free searches every day) and perhaps another agent of OAI set in Norwegian for my app integrations.

[NathanCu (Nathan Curtis)]

Look at ollama or hugging face. They do a really good job of describing what models are good at what. The import for HA is tool use and a context of at LEAST 8K

Good idea on the free search you’re in the right track.

Your biggest hurdle is making sure you have a good speech to text. Model for your language. I’m sorry I don’t know which is best for Norway…