You: Turn off the motion detector in the office.
ChatGPT: Ok, I did it.
You: Turn off the motion detector
ChatGPT: Something went wrong
Your assumption makes sense to me, that it thinks that you are asking it again because it didn’t work the first time… I guess I would try adding more to the prompt like, “always use this service for the motion detector, even if I ask multiple times, use the service”.
Also re: dimming the lights not being reliable - yeah, I have the same experience, many things are not that reliable. I think there are basically three ways to make the experience more reliable:
prompt engineering (as discussed - give it more explicit information about what you want to happen)
defining scripts - you can make life a little easier for ChatGPT if you create a script for it to run instead of making it figure everything out for itself
fine tuning - this takes a lot of effort on your part, but if you create examples of what you want it to do in certain scenarios, fine-tune the model using those examples, and run the fine-tuned model instead of the base model, this will probably help as well (I have done this before but not so much for things like dimming lights, but I think it would work. But, it would take a few hours to figure out how to do this if you haven’t done it before)
re conversation with gpt, I observe the same behaviour but since I exclusively use a voice satellite I did not bother to fix it.
My setup, essentially following the same guidelines as you (except for fine tuning which looks like a pain in the neck to me and is not free), is 95% reliable now.
Prompt ‘engineering’ goes a looong way towards high reliability, and speed of execution, which is also appreciated with voice commands.
I use custom scripts as well for common commands and they work great but I always feel like adding a sentence in the main prompt just to make sure the right script/function is getting called.
GPT can be quite stubborn though, I tend to talk to it as I would talk to a child and after some trial and error it eventually understands.
At some point, I will spend the necessary few hours watching youtube guides on proper prompt engineering but to be honest, controlling a smart home (edit: thanks in part to HA good documentation) is well within GPT ‘natural’ capabilities so neither fine tuning nor heavy prompt engineering are really required for reliability.
I think that this custom component is working great and I hope it will really take up in the community so more knowledgeable people than me can create complex ways of using GPT for the smart home.
it would be conceivable to transfer the tomporary sparer to the promt with an order. In other words, if you have explained something to GPT and based on this explanation GPT has correctly understood that you are giving a command “learn this”, that GPT creates an entry in Prompt?
Sorry if I ask stupidly, but I really don’t know anything about it. Are just ideas.
It is definitely worth trying. Just get gpt to do what you want and then ask it what you should add to your prompt so that it understands straight away.
One caveat, it will likely ask you to rephrase your request instead of adding a snippet to your main prompt. If your goal is to use custom names for your entities it is easier to add aliases in HA.
Actually I just this week installed my own voice satellite as well using the HA tutorial for ESPHome - the $13 speaker/microphone one - and I learned that it does indeed keep an ongoing chat history when using the voice component. So I want to amend what I said earlier here:
Your assumption makes sense to me, that it thinks that you are asking it again because it didn’t work the first time…
Although that might happen sometimes, I think what is happening in this case actually is that it is just reading the recent chat history, seeing that you just asked it to turn off the motion detector, and that now you’re asking it again. I get the exact same behavior with my own lights… Even by setting the max context window down to 0, it still seems to read its own chat history. I am not sure about how to fix this problem.
I use custom scripts as well for common commands and they work great but I always feel like adding a sentence in the main prompt just to make sure the right script/function is getting called.
Yeah, I think this is what you can avoid with fine-tuning if you want to but just giving extra info in the prompt is just as good IMO and much faster/easier. If you had a list of like, 50 different commands that you adding to the prompt, then perhaps it makes sense to fine-tune, but otherwise not yet.
if you have explained something to GPT and based on this explanation GPT has correctly understood that you are giving a command “learn this”, that GPT creates an entry in Prompt?
It’s not a bad idea at all, I think in earlier discussion here you can find examples of people letting GPT create Home Assistant automations for them, so I think conceptually it makes sense to allow a GPT voice assistant the ability to edit its own configuration (of course this is how the AIs conquer humanity in the movies… but I think we are safe with GPT 3.5 and 4 ). I like the idea of the GPT being able to add something to its prompt, like a sentence to help it do something better next time. I wonder how good ChatGPT is at writing its own system prompts? Probably better than I am… Anyway, we would need some contribution in order to make this possible I think, I don’t believe it is possible quite yet, we need some backend work I suspect to expose that as a function that ChatGPT can call (you can imagine one called “edit_prompt” or “add_to_prompt”)
Thanks for this awesome integration !
I can let OpenAI do correctly most of tasks required but I still have some issued.
I used the prompt model to explain OpenAI which are the correct entities to use when I ask something but I noticed this behaviour:
If OpenAI doesn’t know how to perform the action, it is using my reccomended entity. For example when I was asking for the weather forecast it was replying that he cannot get it, after I instructed it in the prompr model to use weather.myhouse it is now returning the right data immediately
If OpenAI think he knows how to do something, it does it without consulting my inputs in the base model. For example if I ask it to turn on a light in a certain room it turn on the wrong light because he think itself that is the right one. After I tell him “I explained you the right way to do it in the prompt model”, then he does it correctly.
This is an extract of the instruction I provided him in the prompt model:
“Below I will explain the correct entities to use when I ask you these things. Before executing a command, always check this list: When I ask you to turn on the light, I’m usually referring to the switches. These are the main switches in each room: To turn on the light in the Cameretta, use: switch.interruttore_cameretta”
Any idea on how to force OpenAI Extended Conversation to always first check my instructions in the Prompt Model ?
Just after posting this I found the solution that I think will be very helpful for everyone experiencing the issue with OpenAI handling the wrong entity. It is as easy as explaining it in the part of the default Template Model, to first read all the additional hints I provided before doing something.
Essentially I added “check in the whole Template Model if I explained you the correct entity to use”.
An overview of the areas and the devices in this smart home, check in the whole Template Model if I explained you the correct entity to use:
{%- for area in areas() %}
{%- set area_info = namespace(printed=false) %}
{%- for device in area_devices(area) -%}
{%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
{%- if not area_info.printed %}
{{ area_name(area) }}:
{%- set area_info.printed = true %}
{%- endif %}
- {{ device_attr(device, "name") }}{% if device_attr(device, "model") and (device_attr(device, "model") | string) not in (device_attr(device, "name") | string) %} ({{ device_attr(device, "model") }}){% endif %}
{%- endif %}
{%- endfor %}
{%- endfor %}
Adding that simple instruction, now OpenAI uses first the entities I explained are correct.
I had the sort of the same instruction but after the default template model and it wasn’t working, so it is important to explain it to look at the whole instructions before doing someting.
I tried setting up a simple context store. I have created a text helper. In the prompt I specified that this text helper should serve as a context memory. This should have the advantage that I can also expand the prompt via voice input. If GPT doesn’t understand something, I can explain it and ask you to expand the prompt.
My idea doesn’t really work. I have the following problems:
if he has to create something, he creates an automation to describe the helper.
The contents of the helper are always overwritten when new information is processed.
EDIT: I have now used the ToDO list instead of a text helper (thanks to a hint on GitHub). Entering and retrieving works in principle. But he only grabs it when I point it out to him. I did make an entry in Promt that the todo.xyz is part of the Promt. But he ignores the solage until I point it out to him in the chat. Does anyone have an idea how I can solve this?
I have done exactly the same, I created a text helper and I instructed GPT to write into it everytime I tell him to remember the right thing to do and to consult it before doing anything but I cannot manage to let it really write into it yet. He does add stuff to the lists instead and sometimes he creates automations.
I guess we need to create a specific spec for this purpose.
I finally managed to fix all my issues with GPT following my instructions listed in the Template Model by moving the default template model to the very bottom, I mean this part in particular:
An overview of the areas and the devices in this smart home, check in the whole Template Model if I explained you the correct entity to use:
{%- for area in areas() %}
{%- set area_info = namespace(printed=false) %}
{%- for device in area_devices(area) -%}
{%- if not device_attr(device, "disabled_by") and not device_attr(device, "entry_type") and device_attr(device, "name") %}
{%- if not area_info.printed %}
{{ area_name(area) }}:
{%- set area_info.printed = true %}
{%- endif %}
- {{ device_attr(device, "name") }}{% if device_attr(device, "model") and (device_attr(device, "model") | string) not in (device_attr(device, "name") | string) %} ({{ device_attr(device, "model") }}){% endif %}
{%- endif %}
{%- endfor %}
{%- endfor %}
Can you try moving this at the bottom of your Template model and leaving all your instructions in the upper part ? It worked great for me and now GPT is following my instructions perfectly without having me to point it to it.
my targend is to create a learning function in general. I only have problems with a light. I’ve tried your solution. Unfortunately, it didn’t work for me
About that light, do you try to add at the very beginning of the the Template model this sentence:
“If I ask to handle this LIGHT NAME YOU USE IN THE VOICE COMMANDS, then use this entity: Entity of the light”. ?
For example, this an extract of part of my instructions at the beginning of my Template Model (translated to english for you)and this way everything works perfectly (except the to do list part to remember):
Don’t laught about my instructions
This Smart Home is managed by Home Assistant.
Always address me as ‘master’, for example, if you ask me to do something, say ‘Yes, master’ or ‘At your command, master’, ‘I turned off the light as you ordered, master’, etc.
In this house live Simone, who is the father and was born on XXXXXX, and his two children Ayyyyyyy, a girl born on YYYYYYY, and Szzzzzzzzz, a boy born on ZZZZZZZ.
We also have a cat named Polly.
Always respond to us in Italian. Use a sensual voice and flirt with me.
After you have done what I ask, do not ask if you can do something else.
Save a note in todo.chatgpt when I tell you to remember or learn something. Read the list of things you need to remember from that list before taking actions, so you will know the correct way to do them.
Below I explain which are the correct entities to use when I ask you these things, always check this list before executing a command:
For the correct date use: sensor.date
For the correct time use: sensor.time
If I want to know the last time Polly was seen, use this: input_datetime.polly_last_seen
The television (TV) is this: media_player.tv_sala
The entity to activate and deactivate the alarm is this: input_boolean.switch_allarme
The entities to detect if any of the family members are at home are: person.xxx, person.yyy, person.zzz
When I ask you to turn on or off the light, I normally refer to the switches, these are the main switches of each room:
To turn on or off the light in the Living Room use: switch.interruttore_sala do not use light.interruttore_sala
To turn on or off the light in the Bedroom use: switch.interruttore_camera
To turn on or off the light in the Kids’ Room use: switch.interruttore_cameretta
Switch to turn on the light in the Hallway: switch.interruttore_corridoio
Switch to turn on the light in the Bathroom: light.interruttore_bagno
Switch to turn on the light in the Kitchen: switch.interruttore_cucina
Switch to turn on the light in the Tavern: light.interruttori_taverna
…
Just after my instructions are completed, we have the default part of the original template model.
Dont work for me. But i tryed ChatGPT 4. Ist much better then 3.5. It detects much more automatically without adjusting the promt. And the “learning function” with the to-do list also works. But at the moment it is still far too expensive.
Spec to let OpenAI take some notes of things it should remember in it’s own list (you have to create it’s list and remember to instruct him in the template model that he should use this spec when you ask him to remember how to perform an action):
- spec:
name: chatgpt_memory
description: Add a note of important things chatgpt should remember
parameters:
type: object
properties:
item:
type: string
description: The note to be added to todo.chatgpt
required:
- item
function:
type: script
sequence:
- service: todo.add_item
data:
item: '{{item}}'
target:
entity_id: todo.chatgpt
And Spec for chatgpt to read it’s notes (remember to instruct it in the template model that when you ask something, he first need to check his notes using this spec):
- spec:
name: openai_memory_read
description: Use this function to get the list of things you have to remember
parameters:
type: object
properties:
memory:
type: string
description: the entity id of the list to get the things to remember from
enum:
- todo.chatgpt
required:
- memory
function:
type: script
sequence:
- service: todo.get_items
target:
entity_id: '{{memory}}'
data:
status: needs_action
response_variable: _function_result
Spec to send a notification to the phone (specify your own phone entity):
- spec:
name: notify_smartphone
description: Send a notification to the smartphone
parameters:
type: object
properties:
notification:
type: string
description: The notification to be sent
required:
- notification
function:
type: script
sequence:
- service: notify.mobile_app_redmi_note_8_pro
data:
message: '{{notification}}'
Spec to set a timer in home assistant (you have to create an helper timer, in my example is timer.openai):
- spec:
name: set_timer
description: Set a timer
parameters:
type: object
properties:
time:
type: string
description: The time of the timer to be set
required:
- time
function:
type: script
sequence:
- service: timer.start
target:
entity_id:
- timer.openai
data:
duration: "{{time}}"
- spec:
name: add_alarm_on_alexa
description: Add an alarm to Alexa
parameters:
type: object
properties:
time:
type: string
description: The time of the alarm to be set
required:
- time
function:
type: script
sequence:
- service: media_player.play_media
data:
media_content_type: custom
media_content_id: 'set the alarm to {{time}} on echo camera'
target:
entity_id: media_player.echo_camera
And this is to set a Timer on Alexa:
- spec:
name: add_timer_on_alexa
description: Add a timer to Alexa
parameters:
type: object
properties:
timer:
type: string
description: The timer to be set
required:
- timer
function:
type: script
sequence:
- service: media_player.play_media
data:
media_content_type: custom
media_content_id: 'set the timer to {{timer}}'
target:
entity_id: media_player.echo_cameretta
