I have done exactly the same, I created a text helper and I instructed GPT to write into it everytime I tell him to remember the right thing to do and to consult it before doing anything but I cannot manage to let it really write into it yet. He does add stuff to the lists instead and sometimes he creates automations.
I guess we need to create a specific spec for this purpose.
I finally managed to fix all my issues with GPT following my instructions listed in the Template Model by moving the default template model to the very bottom, I mean this part in particular:
An overview of the areas and the devices in this smart home, check in the whole Template Model if I explained you the correct entity to use:
{%- for area in areas() %}
{%- set area_info = namespace(printed=false) %}
{%- for device in area_devices(area) -%}
{%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
{%- if not area_info.printed %}
{{ area_name(area) }}:
{%- set area_info.printed = true %}
{%- endif %}
- {{ device_attr(device, "name") }}{% if device_attr(device, "model") and (device_attr(device, "model") | string) not in (device_attr(device, "name") | string) %} ({{ device_attr(device, "model") }}){% endif %}
{%- endif %}
{%- endfor %}
{%- endfor %}
Can you try moving this at the bottom of your Template model and leaving all your instructions in the upper part ? It worked great for me and now GPT is following my instructions perfectly without having me to point it to it.
my targend is to create a learning function in general. I only have problems with a light. I’ve tried your solution. Unfortunately, it didn’t work for me
About that light, do you try to add at the very beginning of the the Template model this sentence:
“If I ask to handle this LIGHT NAME YOU USE IN THE VOICE COMMANDS, then use this entity: Entity of the light”. ?
For example, this an extract of part of my instructions at the beginning of my Template Model (translated to english for you)and this way everything works perfectly (except the to do list part to remember):
Don’t laught about my instructions 
This Smart Home is managed by Home Assistant.
Always address me as ‘master’, for example, if you ask me to do something, say ‘Yes, master’ or ‘At your command, master’, ‘I turned off the light as you ordered, master’, etc.
In this house live Simone, who is the father and was born on XXXXXX, and his two children Ayyyyyyy, a girl born on YYYYYYY, and Szzzzzzzzz, a boy born on ZZZZZZZ.
We also have a cat named Polly.
Always respond to us in Italian. Use a sensual voice and flirt with me.
After you have done what I ask, do not ask if you can do something else.Save a note in todo.chatgpt when I tell you to remember or learn something. Read the list of things you need to remember from that list before taking actions, so you will know the correct way to do them.
Below I explain which are the correct entities to use when I ask you these things, always check this list before executing a command:
For the correct date use: sensor.date
For the correct time use: sensor.time
If I want to know the last time Polly was seen, use this: input_datetime.polly_last_seen
The television (TV) is this: media_player.tv_sala
The entity to activate and deactivate the alarm is this: input_boolean.switch_allarme
The entities to detect if any of the family members are at home are: person.xxx, person.yyy, person.zzz
When I ask you to turn on or off the light, I normally refer to the switches, these are the main switches of each room:
To turn on or off the light in the Living Room use: switch.interruttore_sala do not use light.interruttore_sala
To turn on or off the light in the Bedroom use: switch.interruttore_camera
To turn on or off the light in the Kids’ Room use: switch.interruttore_cameretta
Switch to turn on the light in the Hallway: switch.interruttore_corridoio
Switch to turn on the light in the Bathroom: light.interruttore_bagno
Switch to turn on the light in the Kitchen: switch.interruttore_cucina
Switch to turn on the light in the Tavern: light.interruttori_taverna
…
Just after my instructions are completed, we have the default part of the original template model.
Everything works perfectly this way for me.
1 Like
Dont work for me. But i tryed ChatGPT 4. Ist much better then 3.5. It detects much more automatically without adjusting the promt. And the “learning function” with the to-do list also works. But at the moment it is still far too expensive.
Spec to let OpenAI take some notes of things it should remember in it’s own list (you have to create it’s list and remember to instruct him in the template model that he should use this spec when you ask him to remember how to perform an action):
- spec:
name: chatgpt_memory
description: Add a note of important things chatgpt should remember
parameters:
type: object
properties:
item:
type: string
description: The note to be added to todo.chatgpt
required:
- item
function:
type: script
sequence:
- service: todo.add_item
data:
item: '{{item}}'
target:
entity_id: todo.chatgpt
And Spec for chatgpt to read it’s notes (remember to instruct it in the template model that when you ask something, he first need to check his notes using this spec):
- spec:
name: openai_memory_read
description: Use this function to get the list of things you have to remember
parameters:
type: object
properties:
memory:
type: string
description: the entity id of the list to get the things to remember from
enum:
- todo.chatgpt
required:
- memory
function:
type: script
sequence:
- service: todo.get_items
target:
entity_id: '{{memory}}'
data:
status: needs_action
response_variable: _function_result
Example:
He really added a note:
And he really use it’s notes now to answer me correctly:
2 Likes
Spec to send a notification to the phone (specify your own phone entity):
- spec:
name: notify_smartphone
description: Send a notification to the smartphone
parameters:
type: object
properties:
notification:
type: string
description: The notification to be sent
required:
- notification
function:
type: script
sequence:
- service: notify.mobile_app_redmi_note_8_pro
data:
message: '{{notification}}'
Spec to set a timer in home assistant (you have to create an helper timer, in my example is timer.openai):
- spec:
name: set_timer
description: Set a timer
parameters:
type: object
properties:
time:
type: string
description: The time of the timer to be set
required:
- time
function:
type: script
sequence:
- service: timer.start
target:
entity_id:
- timer.openai
data:
duration: "{{time}}"
Spec to set the alarm on Alexa:
- spec:
name: add_alarm_on_alexa
description: Add an alarm to Alexa
parameters:
type: object
properties:
time:
type: string
description: The time of the alarm to be set
required:
- time
function:
type: script
sequence:
- service: media_player.play_media
data:
media_content_type: custom
media_content_id: 'set the alarm to {{time}} on echo camera'
target:
entity_id: media_player.echo_camera
And this is to set a Timer on Alexa:
- spec:
name: add_timer_on_alexa
description: Add a timer to Alexa
parameters:
type: object
properties:
timer:
type: string
description: The timer to be set
required:
- timer
function:
type: script
sequence:
- service: media_player.play_media
data:
media_content_type: custom
media_content_id: 'set the timer to {{timer}}'
target:
entity_id: media_player.echo_cameretta
1 Like
What is your sentence in the Tamplate? For me it saves the information but doesn’t retrieve it again
Those are my very first lines of Template Model:
Check the content of todo.chatgpt using the openai_memory_read spec and use the information you find to respond to me.
Save a note in todo.chatgpt using the chatgpt_memory spec whenever I expressly tell you to remember or learn something.
By the way, after playing a lot with this integration I grown a theory you might help to eventually validate:
If OpenAI think he knows how to do something, will not use the notes.
If OpenAI doesn’t know how to do it, will use the notes.
In the example in the post above I told him to remember my son’s puppet name. As he doesn’t have any other answer, he use the notes.
This is an example of something is not working:
“Remember that when I ask for the weather, I don’t want to know the humidity”
He takes the note correctly but when I ask for the weather he will also tell me the humidity. This is (to me) because he already have a spec to provide the weather, so it’s using it straight away without using the notes.
Now it works perfekt. Thank you Simone!
1 Like
I built inventory manager with Simone’s idea. Here’s sanity check. Big thanks for @Simone77 .
Prerequisite
- Grocy add-on and custom component have installed.
- extended_openai_conversation installed
- Using parent-child product in Grocy (Parent don’t have their own stock, they only aggregate. Parent do have minimum amount, Child don’t and child products are interchangeable. Child have their own expiration.
- In todo_list, you should have
todo.chatgptandtodo.shopping_list.
Concept
We will create separate assistant, “Inventory Manager”. I don’t know what your own assistant name, but I’ll call this Inventory manager Alexa(Though I don’t communicate with Alexa at all, just for clarify). Your assistant and Alexa does not share their prompt. But they can share memory, todo.chatgpt .
I use gpt-3.5-turbo and it works fine.
Prompt
You should change [‘’’ ‘’’] inside prompt to indicate code block.
———— START————
You are Alexa, a smart home inventory manager.
Periodically, you receive an updated inventory list.
Use this data to notify the user of any shortages or expiring items.
- Utilize 'todo.chatgpt' or 'todo.shopping_list' with the 'todo.add_item' service. In service_data, specify due_date as 'YYYY-MM-DD' or due_datetime as 'YYYY-MM-DD HH:MM:SS'.
- Read existing entries in 'todo.chatgpt' and 'todo.shopping_list' through the 'openai_memory_read' spec to prevent duplicate notifications.
**Update Time: {{now()}}**
**House Inventory:**
{% set inventory = state_attr('sensor.grocy_stock', 'products') %}
House Inventory:
‘’’csv
product in shortage, stock amount, minimum amount
{% for item in inventory %}{% if item.amount_missing != none and item.amount_missing != null %}{{ item.name }},{{ item.amount_aggregated }},{{ item.amount_missing }}{% endif %}{% endfor %}
‘’’
‘’’csv
product, expiration
{% for item in inventory %}{% if item.best_before_date != '2999-12-31' and not item.is_aggregated_amount %}{{ item.name }},{{ item.best_before_date }}{% endif %}
{% endfor %}
‘’’
- Add items below their minimum required amount to 'todo.shopping_list'.
- Notify the user about groceries that are expiring within 3 days or have already expired using 'todo.chatgpt', ensuring not to repeat existing notifications.
- Respond concisely without restating or acknowledging user inputs; instead, promptly ask for necessary clarifications.
————END————
Function
I found out that I don’t need chatgpt_memory spec. Just expose todo.chatgpt and use default execute_services spec in Alexa.
But you should install openai_memory_read.
Be sure to test this working properly before adding automation below.
Automation
alias: Check Inventory every Hour
description: “”
trigger:
- platform: time_pattern
hours: “*”
condition: []
action:
- service: conversation.process
metadata: {}
data:
text: >-
Check inventory and execute function if needed. Don’t ask back and just
say “Completed”. If there’s nothing to do, just say “All is well”.
language: en
agent_id: YOUR_INVENTORY_MANAGER_AGENT_ID
mode: single
Now you have separate assistant Alexa, that check your inventory every hour, and add product in shortage to shopping list and notify upcoming expiration. You still talk to your own assistant who doesn’t know about inventory, but they share common memory from todo.chatgpt and can check which product is expiring, in shortage or not.
1 Like
hi. is it possible to change from gpt 3.5 to gpt 4.0??
Yes. You can choose your model here.
Hi friends, maybe someone can help me, I have an onju-voice-satellite at home (GitHub - tetele/onju-voice-satellite: An ESPHome config for the Onju Voice which makes it a Home Assistant voice satellite) but as soon as I put the extended openai conversation function it works very badly, which I try from the application or from the computer the plugin works perfectly does anyone know if it is possible to fix it
Newly release gpt-4o-2024-05-13 is perfectly working with extended_openai_conversation and it’s blazing fast! Have a try!
1 Like
I can’t wait for the audio capabilities of GPT-4o to become available. The latency benefits and expression of emotions would be amazing to have in our own voice assistant.
I’m not sure it’s possible to build it directly into Extended OpenAI Conversation, since it is a conversation agent which typically takes in text (synthesized by a speech-to-text model) and also outputs text (which is then fed into text-to-speech). But these new models just work with voice directly, no need to translate into text.
I’ve created a feature request to add support for this kind of all-in-one assistant in Core. But if we’re able to get access to the raw audio input somehow then maybe we can start building already!
I voted. And I guess official OpenAI Conversation integation will be able to use function calling soon.
I have voted as well. Good idea