[Custom Component] extended_openai_conversation: Let's control entities via ChatGPT

meni123 · January 18, 2024, 5:24pm

@jekalmin
Thank you for your great plugin.
In my opinion, this is a new era in HA
I noticed that he has a problem with calling for inspection or action in a certain area, that is, if I tell him to check a specific device, he is accurate in reading the situation and acting on it, but when I tell him to check what is happening in a specific area, he is not accurate at all,
Would love your help

jekalmin · January 19, 2024, 9:49am

@Yahav , @meni123

I also need a help in improving accuracy.
What I can currently think of is following:

Short term

Adjust temperature or topP in options
Tweak prompt (current prompt is not ideal)

Long term

Fine-tune model (I haven’t tried)

I’m also looking forward making response more accurate and consistent.
Please create a pull request if anyone has better prompt or functions.

saschaabraham · January 20, 2024, 8:57am

A working function to get the real weather forecast of a city/region.
This one uses https://www.tomorrow.io/, which gives a detailed forcecast up to 7 days. You have to register there, but it is free with a rate limit for the api requests. But 500 per day, 25 per hour is ok, i think.
Replace <Your location> with your city and the <Your Tomorrow.io key> with your Api- Key.

- spec:
    name: get_weather
    description: Get weatherforcast for a city specified by the user. Use "<Your Location>" if no city is provided.
    parameters:
      type: object
      properties:
        city_id:
          type: string
          description: City of the weather forecast.
      required:
      - city_id
  function:
    type: rest
    resource_template: "https://api.tomorrow.io/v4/weather/forecast?location={{city_id}}&timesteps=1d&apikey=<Your Tomorrow.io key>"
    value_template: '{{value_json }}'

EDIT: There was a mistake, i used resource instead of resource_template.

Yahav · January 21, 2024, 9:03pm

Hello everyone,

I’m dealing with a issue and would appreciate your help. I have a whatsapp bot configured in the following way:

name: WhatsApp
platform: rest
resource: https://api.callmebot.com/whatsapp.php
data:
source: HA
name: nehoray
phone: *****
apikey: *****

The code for the message-sending function looks like this:

spec:
name: send_message_to_whatsapp
description: Use this function to send a message to WhatsApp.
parameters:
type: object
properties:
message:
type: string
description: Message you want to send
required:
- message
function:
type: script
sequence:
- service: notify.WhatsApp
data:
message: “{{ message }}”

It works well

However, when I try to add a personal recipient by adding another configuration line like this:

name: WhatsApp
platform: rest
resource: https://api.callmebot.com/whatsapp.php
data:
source: HA
name: merav
phone: *****
apikey: *****

I don’t get the option to send a message at all.

I’ve tried several versions of code, for example:

Version with two parameters - message and recipient:

spec:
name: send_message_to_whatsapp
description: Use this function to send a message to WhatsApp.
parameters:
type: object
properties:
message:
type: string
description: Message you want to send
recipient:
type: string
description: Recipient’s name
required:
- message
- recipient
function:
type: script
sequence:
- service: notify.WhatsApp
data:
message: “{{ message }}”
recipient: “{{ recipient }}”

Version with two separate scripts - each script for a different recipient:

spec:
name: send_message_to_whatsapp_merav
description: Use this function to send a message to WhatsApp to Merav.
parameters:
type: object
properties:
message:
type: string
description: Message you want to send
required:
- message
function:
type: script
sequence:
- service: notify.WhatsApp
data:
message: “{{ message }}”
name: “merav”
spec:
name: send_message_to_whatsapp_nehoray
description: Use this function to send a message to WhatsApp to Nehoray.
parameters:
type: object
properties:
message:
type: string
description: Message you want to send
required:
- message
function:
type: script
sequence:
- service: notify.WhatsApp
data:
message: “{{ message }}”
name: “nehoray”

Unfortunately, none of these attempts have been successful. Does anyone have an idea or recommendation on how to add another recipient?

Thanks a lot to anyone who can help!

Dino-Tech · January 22, 2024, 2:38am

If anyone is interested, I had a great need for a timer as we use one almost daily and has performed flawlessly for weeks…

---
## Use to set timer in HA using voice assistant
## along with automation located in automations.yaml#L42-L85
############################################################

- spec:
    name: set_timer_duration
    description: Use this function to set timer duration.
    parameters:
      type: object
      properties:
        timer_minutes:
          type: number
          description: The timer duration minutes in '%H:%M:%S' format
        timer_seconds:
          type: number
          description: The timer duration seconds in '%H:%M:%S' format
      required:
      - timer_minutes
      - timer_seconds
  function:
    type: script
    sequence:
    - service: input_number.set_value
      target:
        entity_id: input_number.timer_minutes
      data:
        value: "{{ timer_minutes }}"
    - service: input_number.set_value
      target:
        entity_id: input_number.timer_seconds
      data:
        value: "{{ timer_seconds }}"

link to: automation for timer works with any combination of hours, minutes or seconds.
Thank you @jekalmin for your amazing work!!! Will your integration work with gpt-4?

Dino-Tech · January 22, 2024, 2:47am

I don’t use that service, but here is how I added multiple recipients:
Multiple user notifications

Yahav · January 22, 2024, 10:44am

Works perfectly, thank you very much.

Ketti · January 22, 2024, 12:39pm

Hi! Thank you for this. I got this working with gpt-4-1106-preview, but as soon as I changed back to gpt-3.5-turbo-1106 (way cheaper) I got this error message:

Should it work with gpt-3.5-turbo-1106?

stban1983 · January 22, 2024, 4:39pm

Hello!
weird, I haven’t tested it for a while but I have the impression that openai no longer wants to do anything on my equipment, for example turning on the lights:

it says ok but nothing happened
But read a sensor, it’s working…
There is no error in log

EDIT:
I found it! Maybe it can help someone:
In the conversation agent properties, “Maximum function calls per conversation” was set to 0 … !!

stban1983 · January 22, 2024, 5:45pm

Hi @Dino-Tech,
in your automation, you use a service called “timer.set_duration” but it doesn’t exists for me. I miss somehing

Dino-Tech · January 22, 2024, 8:57pm

Many apologies, I use the Spook integration that provides that service. I have been using it since it was released and forgot that was exclusive to that integration. Spook - Not your homie

stban1983 · January 23, 2024, 10:16am

ok no problem

rysm83 · January 24, 2024, 8:20pm

I love this integration. One thing I was struggling with is having the AI start/run my scripts. I get errors that it can’t find the script name. Will I need to add in a spec and define each script?

Rafaille · January 25, 2024, 10:58pm

It is normally not necessary unless you want to use such scripts with a specific template/variable.
For simple script.turn_on services you should ensure that either the script entity name, or friendly name, or alias matches your request to gpt (or at least is understandable by gpt).
Trial and error is your friend here.
Also, if you are using voice, double check the speech to text output to make sure it got your query right.

Dino-Tech · January 26, 2024, 4:03am

I have several weather entities exposed that I use for different needs. I’ve been having issues with retrieving accurate and consistent weather queries and after many different attempts, I ended up with this:

- spec:
    name: get_current_weather
    description: Get the current weather
    parameters:
      type: object
      properties:
        location:
          type: string
          description: Infer this from home location, (e.g. San Francisco, CA)
        format:
          enum:
            - current forecast
            - hourly forecast
          description: The type of weather forecast information to search for and use.
        entity_id:
          type: string
          description: entity_id
      required:
      - location
      - format
      - entity_id
  function:
    type: template
    value_template: "{{states[entity_id]}}"

If anyone has any cleaner suggestions, please fell free to ping me, but this seems to have solved my issues so far.

Yahav · January 26, 2024, 10:17am

Hi guys,

I just wanted to ask, everything is working pretty good for me, the lights, the sensors, the air conditioner, everything except the windows. Most of the time I get the following error:

Something went wrong: Service cover.open not found.

And sometimes it opens. Does anyone know how I can fix this?

Thank you very much

jekalmin · January 27, 2024, 12:40pm

Since execute_service function requires one of entity_id, area_id, or device_id, you can’t call script that doesn’t require it.

If you want to call script that doesn’t require entity_id, you may try something like this.

- spec:
    name: run_script
    description: Run script of Home Assistant
    parameters:
      type: object
      properties:
        service:
          type: string
          description: The service
          enum:
            - script.livingroom_light_on
            - script.livingroom_light_off
      required:
      - service
  function:
    type: script
    sequence:
    - service: "{{service}}"

However, if script requires any arguments, LLM doesn’t know what arguments are needed for each script.

jekalmin · January 27, 2024, 12:45pm

Try add following in prompt.

If service cover.open is to be used, use cover.open_cover instead.

See issue

Rafaille · January 27, 2024, 6:23pm

In theory, a script can be an entity_id for the script.turn_on service, can it not?
In practice, I have had gpt run my scripts reproducibly just by asking it to run them without mentioning any more variables. I suspect sticking to the script name/alias is important in that case.

HJM · January 27, 2024, 11:18pm

To go along with the add_item_to_list I have the following working to retrieve list items:

- spec:
    name: get_items_from_list
    description: Use this function to get list of items in a ToDo list.
    parameters:
      type: object
      properties:
        list:
          type: string
          description: the entity id of the list to get the items from
          enum:
            - todo.shopping_list
            - todo.house_todo
      required:
      - list
  function:
    type: script
    sequence:
    - service: todo.get_items
      target:
        entity_id: '{{list}}'
      data:
        status: needs_action
      response_variable: _function_result

Be sure to update this part for your lists:

          enum:
            - todo.shopping_list
            - todo.house_todo