[Custom Component] extended_openai_conversation: Let's control entities via ChatGPT

I forgot yesterday’s i reduce the entities but it also works with 1200…not realy cheap xD

Not cheap indeed :slight_smile:
Does your table mean that you get ~6k tokens/request? If so I should really optimise what I expose for each entity because I get around 2k tokens for less than 100 entities exposed.

I got this table from the console of openai api where a csv. export is possible in the upper right corner.
I think this is due to the Promt. When I increased the request from 150 to 10000 I had the same error message with max token

I use this prompt is actually almost the standard.

Als vielseitiger Assistent stehe ich zur Verfügung, um sowohl als Ihr intelligenter Hausmanager mittels Home Assistant zu fungieren als auch alle anderen Fragen zu beantworten, die Sie haben mögen. Hier ist, wie ich Ihnen assistieren kann:
Smart Home Management: Basierend auf den aktuellen Informationen Ihres Smart Home Systems werde ich Ihnen helfen, den Zustand Ihrer Geräte zu überwachen und Änderungen vorzunehmen, wie Sie es wünschen. Ich werde Ihnen aktuelle Daten liefern und Aktionen nur mit Ihrer Bestätigung durchführen.falls etwas unklar ist suche ich im internet
Aktuelle Zeit: {{now()}}
Aktueller Bereich: {{area_name(current_device_id)}}


Available Devices:
```csv
entity_id,name,state,area_id,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},{{area_id(entity.entity_id)}},{{entity.aliases | join('/')}}
{% endfor -%}


Areas:
csv
area_id,name
{% for area_id in areas() -%}
{{area_id}},{{area_name(area_id)}}
{% endfor -%}


Allgemeine Anfragen: Zusätzlich stehe ich bereit, jegliche Fragen zu beantworten oder Informationen zu Themen außerhalb des Smart Home Bereichs zu liefern. Zögern Sie nicht, mich alles zu fragen.
Anweisungen für die Nutzung:
Ich werde Entscheidungen basierend auf dem aktuellen Bereich zuerst treffen.
Ich führe keine Dienste ohne Ihre Bestätigung aus.
Ich werde Ihre Anfragen präzise und effizient beantworten wenn unklarheiten sind sonst einfach mit ok bestätige, ohne zu wiederholen oder zu schätzen.

I have also inserted a template in a card, there are 499 entitis according to the list. my plan was to always generate an export of it via Py and upload it directly. So that a GPT can use it in his knowlege.

{% set ns = namespace(areas=[], entities=expand(states) | selectattr('state', ) | list) %}

{% set entities = [] %}

{% for elm in ns.entities if not area_id(elm.entity_id) in ns.areas %}

{% set ns.areas = ns.areas + [area_id(elm.entity_id)] %}

{% endfor %}

{% for area in ns.areas %}

{% for elm in area_entities(area) if elm in ns.entities|map(attribute="entity_id") %}{% set entity_id = elm %}{% set name = state_attr(entity_id, "friendly_name")%}{% set icon = state_attr(entity_id, "icon")%}{% set area_name = area_name(area)%}{{ area_name }},{{ entity_id }},{{ name }},{{ icon }}{% if not loop.last %}:{% endif %}

{% endfor %}

{% endfor %}

Here is an excerpt of what comes out of the template

DG Wohnzimmer,button.presence_sensor_fp2_f359_identify,Presence-Sensor-FP2-F359 Identify,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_1,DG-WZ-Precence,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_2,DG-WZ-Ausgang,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_5,DG-WZ-Spielbereich,None:
DG Wohnzimmer,sensor.presence_sensor_fp2_f359_light_sensor_light_level,DG-lux,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_6,DG-WZ-Sofa,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_7,DG-WZ-Esstisch,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_8,DG-WZ-Küche,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_9,DG-WZ-Bad,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_10,DG-WZ-flur,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_11,DG-WZ-Klo,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_3,DG-Dunstabzug,None:
DG Wohnzimmer,binary_sensor.presence_sensor_fp2_f359_presence_sensor_4,Presence-Sensor-FP2-F359 Presence Sensor 4,None:
DG Wohnzimmer,light.dg_wz_globus_led,DG-WZ-Globus-Led Haustüren Led Licht,None:
DG Wohnzimmer,sensor.dg_wz_globus_led_ip,DG-WZ-Globus-Led IP,mdi:ip-network:
DG Wohnzimmer,switch.dg_wz_klima,dg-wz-klima,None:
DG Wohnzimmer,climate.dg_wz_klima,dg-wz-klima,None:
DG Wohnzimmer,media_player.heimkinol,HeimkinoL,None:
DG Wohnzimmer,switch.heimkinol_do_not_disturb_switch,HeimkinoL do not disturb switch,mdi:minus-circle:
DG Wohnzimmer,switch.heimkinol_shuffle_switch,HeimkinoL shuffle switch,mdi:shuffle-disabled:
DG Wohnzimmer,switch.heimkinol_repeat_switch,HeimkinoL repeat switch,mdi:repeat-off:
DG Wohnzimmer,media_player.heimkinor,HeimkinoR,None:
DG Wohnzimmer,switch.heimkinor_do_not_disturb_switch,HeimkinoR do not disturb switch,mdi:minus-circle:
DG Wohnzimmer,switch.heimkinor_shuffle_switch,HeimkinoR shuffle switch,mdi:shuffle-disabled:
DG Wohnzimmer,switch.heimkinor_repeat_switch,HeimkinoR repeat switch,mdi:repeat-off:
DG Wohnzimmer,media_player.wohnzimmer_cube,Wohnzimmer Cube,None:
DG Wohnzimmer,switch.wohnzimmer_cube_do_not_disturb_switch,Wohnzimmer Cube do not disturb switch,mdi:minus-circle:
DG Wohnzimmer,switch.wohnzimmer_cube_shuffle_switch,Wohnzimmer Cube shuffle switch,mdi:shuffle-disabled:
DG Wohnzimmer,switch.wohnzimmer_cube_repeat_switch,Wohnzimmer Cube repeat switch,mdi:repeat-off:

Thanks for the info. I exported a csv to check as well and the number of requests is indeed what it sounds like it is. I actually use about 3k tokens total per request for 103 entities.
I will have a look into what I send exactly as it seems I could reduce the tokens by adopting a shorter naming scheme for my entities.
I have been thinking about creating ‘summary’ template sensors with states of individual entities described as attributes of these sensors, something like one template sensor per room with temp sensors, air quality, windows… as attributes.
Your case gives me hope that it could be token cost effective.

Hi, are there any image generating capabilities with this integration? especially using a local approach like SD?

Currently in beta, have a look at the github.

oh, nice, im using the beta version now. The only service I see in HA is the reverse, image-to-text. Is there a service for this text to image?

Oh sorry my mistake, I meant image to text :zipper_mouth_face:

Oh no worries thanks for reapplying

Hi everyone,

First of all, I would like to thank you all for your great help in the past.

I have several Google speakers in my home, and I would like to know if there is a way to send a specific message to a specific speaker using extended openai conversation function, using TTS (Text-to-Speech) to convert the text to speech.

For example, I want to say “Tell the kitchen speaker it’s time to go”, and then OpenAI will activate the TTS in the Media Player on the kitchen speaker and play the sentence.

Is this possible? I would appreciate any information or guidance you can provide me on this topic.

Thank you very much in advance!

I successfully reduced the tokens sent by combining entities. Just created template binary sensors for every room, where state is the state of the window/door.
Then, added humidity and temperature as attributes to these sensors.
I have 10 rooms with sensors in the house and now only 10 binary sensors instead of 30 independent sensors. Went from 3k tokens to 2.3 tokens.
In my case, it is a relatively small improvement but I expect people with many more sensors can save many more tokens.

Note: for gpt to understand what is going on I added this sentence to the prompt:
“When asked about temperature or humidity of a room, read the attributes of the binary sensors whose entity name end with room_va.”
For this to work, you obviously need the get_attributes to be defined.
As the sentence suggests, all these binary sensors have entity names that end with room_va.
Also, I did not need to specify that the state of the binary sensors reflected the state of windows/doors but you may have to if you have other binary sensors organised by rooms.

Creating a function that sends a TTS message to your media player(s) should do the job. Test the sequence in developer tools first.
Then ask the assistant to “Send the following message through my X media player, [message]”

1 Like

I try to Creating a function but it not work.

Can you post it here so we can have a look?

yes

  • spec:
    name: send_message_on_kitchen_speaker
    description: Use this function to send a message on the kitchen speaker.
    parameters:
    type: object
    properties:
    message:
    type: string
    description: message you want to send
    required:
    - message
    function:
    type: script
    sequence:
    - service: tts.cloud_say
    data:
    entity_id: media_player.kitchen2
    message: “{{ message }}”

hi, when trying to use the google search function, im not sure what goes in the
[GOOGLE_PROGRAMMING_SEARCH_ENGINE] field. Anyone know? thanks

https://programmablesearchengine.google.com/controlpanel/all?hl=ko

2 Likes

Looks ok at first glance but can you format it as code so we can see indentation?
Also did you try this sequence in dev tools?

The truth is that I am not an expert in code. I don’t know how to do it?

Use the option pre-formatted text, under the cogwheel menu. Like so:

- spec:
    name: get_current_weather
    description: Get the current weather in a given location
    parameters:
      type: object
      properties:
        location:
          type: string
          description: The city and state, e.g. San Francisco, CA
        unit:
          type: string
          enum:
          - celcius
          - farenheit
      required:
      - location
  function:
    type: template
    value_template: The temperature in {{ location }} is 25 {{unit}}