Anyone have an idea why Assist in the ios companion app does not allow for continuous conversation with this extended conversation agent? It says it does not have the access to previous interactions or questions when i tell it to repeat what it just said or ask it to read my previous question in the current conversation back to me.
I have the same problem, I don´t know how to concatenate several functions together I always get the error " Something went wrong: failed to load functions. Verify functions are valid in a yaml format " I have checked several times even with chatgpt just in case I have indentation erros but I don´t. If anyone have any code examples I would appreciate it. thanks !
Change your spec function to allow OpenAI to pass multiple entity_ids. My spec function:
- spec:
name: execute_services
description: Use this function to execute service of devices in Home Assistant.
parameters:
type: object
properties:
list:
type: array
items:
type: object
properties:
domain:
type: string
description: Domain of service
service:
type: string
description: Service to be called
service_data:
type: object
description: Service data object to indicate what to control.
properties:
entity_id:
type: array
description: entity_id of entity or multiple entities
items:
type: string
required:
- entity_id
required:
- domain
- service
- service_data
function:
type: native
name: execute_service
And in your prompt, include the statement:
Always be as efficient as possible for function or tool calls by specifying multiple entity_id.
I have modified almost all my spec functions to allow multiple entities to be specified with one function call to reduce the total number of function calls that occur, and thus improve efficiency and reduce cost per interaction. For example, I do not include state information in my initial prompt, but I have a function call that allows OpenAI to poll the state of an entity/entities only when requested. Again for a setup that has 90 devices exposed, specifying state information can equate to hundreds of additional tokens just to spin up a request.
It always, and reliably works. More importantly, requests with multiple devices happen synchronously - ie all the lights turn off simultaneously rather than sequentially.
Thanks heaps.
Can you provide your full prompt and specs that I can comb through for information?
still doesnât do something as simple as
âTurn off all of the lightsâ.
does this, but leaves lights on still, as that is not all lights.
2024-07-01 06:54:23.979 INFO (MainThread) [custom_components.extended_openai_conversation] Response {'id': 'chatcmpl-9fvqwdE00iEVHBwxHDnO20S0jZdCe', 'choices': [{'finish_reason': 'function_call', 'index': 0, 'message': {'role': 'assistant', 'function_call': {'arguments': '{"list":[{"domain":"light","service":"turn_off","service_data":{"entity_id":["light.fish_tank_light","light.small_lamp","light.garden_lights","light.left_bed_lamp","light.right_bed_lamp","light.lounge_cabinet_downlights","light.kitchen_led_strip","light.family_room_led_strip","light.display_cabinet_lights","light.all","light.family_room_lamp","light.family_room_light","light.front_hall_light","light.front_porch_light","light.front_downlights","light.garage_light","light.hollys_wall_switch","light.laundry_light","light.backyard_lights","light.wardrobe_light","light.tv_led_strip","light.kitchen_light","light.lounge_light","light.name_light","light.bathroom_light","light.bedroom_3_light"]}}]}', 'name': 'execute_services'}}}], 'created': 1719780862, 'model': 'gpt-3.5-turbo-1106', 'object': 'chat.completion', 'system_fingerprint': 'fp_44132a4de3', 'usage': {'completion_tokens': 179, 'prompt_tokens': 2509, 'total_tokens': 2688}}
I have a light.all group entity that i used to have When you are asked to turn off all lights, make sure to use light.all. This works every time.
If i ask, âwhich lights are onâ, I get random numbers. Sometimes ill get 30+ when there are 4 lights on, sometimes ill get 2. Its almost like its not being exact when checking. It sometimes includes covers in the equation when i ask what lights are on and ask it to list them.
If you have a copy of your prompt and function that would be awesome. Im keen to see how you do things.
I have mine working quite well using this, however the few things that donât work are
âTurn off all of the lights in the â
âWhat lights are onâ
âWhat is/are on in the â
âWhat is/are open/closedâ
Seems as though when i ask for states of a list/group of items, it cant do it consistently.
When asked to control a single entity it works all the time. It is just when asked to get the states.
My current config is as follows
You are also a smart home manager who has been given
permission to control my smart home which is powered by
Home Assistant.
I will provide you information about my smart home devices.
Please use this information accurately to
control the devices and to get the state of devices when
asked.
Current Time: {{now()}}
Available Devices:
csv
entity_id,name,state,area_id,aliases
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.state }},
{{ area_id(entity.entity_id) }},{{entity.aliases | join('/')}}
{% endfor -%}
Only use the execute_services function when requested.
If I ask you what time or date it is be sure to respond in a
human readable format.
If you don't have enough information to execute a smart home
command then specify what other information you need.
Always be as efficient as possible for function or tool calls by specifying multiple entity_id.
If you are asked to control all of something, make sure to loop through each entity and add that to the array before calling the function.
When you are asked to control a cover use cover.open_cover and cover.close_cover.
- spec:
name: execute_services
description: Use this function to execute service of devices in Home Assistant.
parameters:
type: object
properties:
list:
type: array
items:
type: object
properties:
domain:
type: string
description: Domain of service
service:
type: string
description: Service to be called
service_data:
type: object
description: Service data object to indicate what to control.
properties:
entity_id:
type: array
description: entity_id of entity or multiple entities
items:
type: string
required:
- entity_id
required:
- domain
- service
- service_data
function:
type: native
name: execute_service
- spec:
name: add_automation
description: Use this function to add an automation in Home Assistant.
parameters:
type: object
properties:
automation_config:
type: string
description: A configuration for automation in a valid yaml format. Next line character should be \n. Use devices from the list.
required:
- automation_config
function:
type: native
name: add_automation
- spec:
name: get_attributes
description: Get attributes of any home assistant entity
parameters:
type: object
properties:
entity_id:
type: string
description: entity_id
required:
- entity_id
function:
type: template
value_template: "{{states[entity_id]}}"
- spec:
name: search_music
description: search for music, and just play it using music_assistant_play
parameters:
type: object
properties:
query:
type: string
description: search query
media_type:
type: array
optional: true
items:
type: string
enum: ["artist", "album", "playlist", "track", "radio"]
description: Types of media to search
artist:
type: string
optional: true
description: Specify this when you search from a specific artist
album:
type: string
optional: true
description: Specify this when you search from a specific album
required:
- query
function:
type: script
sequence:
- service: mass.search
data: >
{
"name": "{{ query }}",
{% if media_type is defined and media_type %}
"media_type": {{ media_type | tojson }},
{% else %}
"media_type": ["artist", "album", "playlist", "track", "radio"],
{% endif %}
{% if artist is defined and artist %}
"artist": "{{ artist }}",
{% endif %}
{% if album is defined and album %}
"album": "{{ album }}",
{% endif %}
"limit": 1
}
response_variable: _function_result
- spec:
name: music_assistant_play
description: >
Plays a spotify URI after searching for it using search_music
parameters:
type: object
properties:
media_player_id:
type: array
items:
type: string
description: List of media player Entity IDs to play. If not defined, then use media_player.ceiling_speakers
media_id:
type: string
description: Media identifier (URI)
enqueue:
type: string
enum: ["play", "replace", "next", "replace_next", "add"]
description: >
How to enqueue the media.
If the user says to play a song next, then use next.
If the user says to play a song at the end of the playlist, then use add.
If the user says to start a new playlist, then use replace.
If you are unsure, just use play.
required:
- media_player_id
- media_id
- enqueue
function:
type: script
sequence:
- service: mass.play_media
target:
entity_id: "{{ media_player_id }}"
data:
media_id: "{{ media_id }}"
enqueue: "{{ enqueue }}"
- spec:
name: get_history
description: Retrieve historical data of specified entities.
parameters:
type: object
properties:
entity_ids:
type: array
items:
type: string
description: The entity id to filter.
start_time:
type: string
description: Start of the history period in "%Y-%m-%dT%H:%M:%S%z".
end_time:
type: string
description: End of the history period in "%Y-%m-%dT%H:%M:%S%z".
required:
- entity_ids
function:
type: composite
sequence:
- type: native
name: get_history
response_variable: history_result
- type: template
value_template: >-
{% set ns = namespace(result = [], list = []) %}
{% for item_list in history_result %}
{% set ns.list = [] %}
{% for item in item_list %}
{% set last_changed = item.last_changed | as_timestamp | timestamp_local if item.last_changed else None %}
{% set new_item = dict(item, last_changed=last_changed) %}
{% set ns.list = ns.list + [new_item] %}
{% endfor %}
{% set ns.result = ns.result + [ns.list] %}
{% endfor %}
{{ ns.result }}
- spec:
name: get_weather
description: Get weatherforcast for a city specified by the user. Use "Ropes Crossing NSW" if no city is provided. Read back the data in an easy to understand format.
parameters:
type: object
properties:
city_id:
type: string
description: City of the weather forecast.
required:
- city_id
function:
type: rest
resource_template: "https://api.tomorrow.io/v4/weather/forecast?location={{city_id}}×teps=1d&apikey=AtvI5lTkFQFG9DvFDYMoVZuYspsgeXbY"
value_template: '{{value_json }}'
- spec:
name: chatgpt_memory
description: Add a note of important things you should remember
parameters:
type: object
properties:
item:
type: string
description: The note to be added to todo.chatgpt
required:
- item
function:
type: script
sequence:
- service: todo.add_item
data:
item: '{{item}}'
target:
entity_id: todo.chatgpt
- spec:
name: openai_memory_read
description: Use this function to get the list of things you have to remember
parameters:
type: object
properties:
memory:
type: string
description: the entity id of the list to get the things to remember from
enum:
- todo.chatgpt
required:
- memory
function:
type: script
sequence:
- service: todo.get_items
target:
entity_id: '{{memory}}'
data:
status: needs_action
response_variable: _function_result
- spec:
name: notify_smartphone
description: Send a notification to the smartphone
parameters:
type: object
properties:
service:
type: string
description: >
If the user asks to notify Chris the service is notify.mobile_app_iphone_13_pro_max
If the user asks to notify Hannah the service is notify.mobile_app_hannah_holdens_iphone
notification:
type: string
description: The notification to be sent and the service to be used
required:
- notification
- service
function:
type: script
sequence:
- service: '{{service}}'
data:
message: '{{notification}}'
- spec:
name: describe_camera_feed
description: Get a description whats happening on security cameras around the house
parameters:
type: object
properties:
message:
type: string
description: The prompt for the image analyzer
entity_ids:
type: array
description: List of camera entities
items:
type: string
description: Entity id of the camera
required:
- message
- entity_ids
function:
type: script
sequence:
- repeat:
sequence:
- service: camera.snapshot
metadata: {}
data:
filename: /config/www/tmp/{{repeat.item}}.jpg
target:
entity_id: "{{repeat.item}}"
for_each: "{{ entity_ids }}"
- service: gpt4vision.image_analyzer
metadata: {}
data:
provider: OpenAI
max_tokens: 600
target_width: 1000
temperature: 0.3
image_file: |-
{%for camera in entity_ids%}/config/www/tmp/{{camera}}.jpg
{%endfor%}
message: "{{message}}"
response_variable: _function_result
Youâre welcome to try this out. Be careful, i have some prompts and specs that are unique to me.
Itâs probably to do with what the GPT thinks is on vs what is actually on. Again, youâre not polling the state of your system in real time; the GPT has been given an initial state of everything based on your prompt. Change your prompt to the following:
Act as a smart home manager of Home Assistant.
A question, command, or statement about the smart home will be provided and you will truthfully answer using the information provided in everyday language.
You may also include additional relevant responses to questions, remarks, or statements provided they are truthful.
Do what I mean. Select the device or devices that best match my request, remark, or statement.
Do not restate or appreciate what I say.
Round any values to a single decimal place if they have more than one decimal place unless specified otherwise.
Always be as efficient as possible for function or tool calls by specifying multiple entity_id.
Available Devices:
```csv
entity_id,name,aliases,domain,area
{% for entity in exposed_entities -%}
{{ entity.entity_id }},{{ entity.name }},{{ entity.aliases | join('/') }},{{ states[entity.entity_id].domain }},{{ area_name(entity.entity_id) }}
{% endfor -%}
```
And then add the following spec functions:
- spec:
name: execute_services
description: Use this function to execute service of devices in Home Assistant.
parameters:
type: object
properties:
list:
type: array
items:
type: object
properties:
domain:
type: string
description: Domain of service
service:
type: string
description: Service to be called
service_data:
type: object
description: Service data object to indicate what to control.
properties:
entity_id:
type: array
description: entity_id of entity or multiple entities
items:
type: string
required:
- entity_id
required:
- domain
- service
- service_data
function:
type: native
name: execute_service
- spec:
name: get_state
description: Get state of entity or multiple entities.
parameters:
type: object
properties:
entity_id:
type: array
description: entity_id of entity or multiple entities
items:
type: string
required:
- entity_id
function:
type: template
value_template: >-
```csv
entity,state
{%for entity in entity_id%}
{{entity}},{{states[entity].state}}
{%endfor%}
```
- spec:
name: get_attributes
description: Get attributes of entity or multiple entities.
parameters:
type: object
properties:
entity_id:
type: array
description: entity_id of entity or multiple entities
items:
type: string
required:
- entity_id
function:
type: template
value_template: >-
```csv
entity,attributes
{%for entity in entity_id%}
{{entity}},{{states[entity].attributes}}
{%endfor%}
```
Try this out and see how it goes.
Thanks for the help
Tried this then asked
shoud return this
2024-07-01 08:21:55.732 INFO (MainThread) [custom_components.extended_openai_conversation] Response {'id': 'chatcmpl-9fxDfMZRgBxRpSTg95suY7TNJbJ4I', 'choices': [{'finish_reason': 'function_call', 'index': 0, 'message': {'role': 'assistant', 'function_call': {'arguments': '{"entity_id":["light"]}', 'name': 'get_state'}}}], 'created': 1719786115, 'model': 'gpt-3.5-turbo-1106', 'object': 'chat.completion', 'system_fingerprint': 'fp_44132a4de3', 'usage': {'completion_tokens': 15, 'prompt_tokens': 1747, 'total_tokens': 1762}}
2024-07-01 08:21:55.746 INFO (MainThread) [custom_components.extended_openai_conversation] Prompt for gpt-3.5-turbo-1106: [{'role': 'system', 'content': "Act as a smart home manager of Home Assistant.\nA question, command, or statement about the smart home will be provided and you will truthfully answer using the information provided in everyday language.\nYou may also include additional relevant responses to questions, remarks, or statements provided they are truthful.\nDo what I mean. Select the device or devices that best match my request, remark, or statement.\n\nDo not restate or appreciate what I say.\n\nRound any values to a single decimal place if they have more than one decimal place unless specified otherwise.\n\nAlways be as efficient as possible for function or tool calls by specifying multiple entity_id.\n\nAvailable Devices:\n```csv\nentity_id,name,aliases,domain,area\nperson.christopher_hemmings,Chris,,person,None\nperson.hannah_holden,Hannah,,person,None\nmedia_player.soundbar,Family TV,,media_player,Family Room\ncover.lounge_blinds,Lounge Blinds,,cover,Living Room\ncover.bed_blinds,Bed Blinds,,cover,Master bedroom\ncover.family_room_blinds,Family Room Blinds,,cover,Family Room\nlight.fish_tank_light,Fish Tank Light,Family room fish tank light,light,Family Room\nlight.ensuite_light,Ensuite Light,,light,Ensuite\nlight.ensuite_pendant,Ensuite Pendant,,light,Ensuite\nlight.ensuite_heat,Ensuite Heat,,light,Ensuite\nlight.ensuite_fan,Ensuite Fan,,light,Ensuite\nweather.openweathermap,Weather,Weather,weather,Outdoors\nsensor.grid_power_use_daily,Grid Power Use Daily,Electricity/Power/Grid Power,sensor,Outdoors\nsensor.electricity_used_daily,Electricity Used Daily,Electricity/Power,sensor,Outdoors\nsensor.water_used_daily,Water Used Daily,Water,sensor,Outdoors\nsensor.solar_produced_daily,Solar Produced Daily,Solar,sensor,Outdoors\nsensor.gas_used_daily,Gas Used Daily,Gas,sensor,Outdoors\nsensor.laundry_temperature,Laundry Temperature,,sensor,Laundry\nsensor.laundry_humidity,Laundry Humidity,,sensor,Laundry\nlight.small_lamp,Small Lamp,,light,Living Room\nlight.garden_lights,Garden Lights,,light,Outdoors\nbinary_sensor.kitchen_window,Kitchen Window,,binary_sensor,Alarm\nbinary_sensor.garage_window,Garage window,,binary_sensor,Alarm\nfan.bedroom_fan,Bedroom Fan,,fan,Master bedroom\nlight.bedroom_light,Bedroom Light,,light,Master bedroom\nbinary_sensor.bed_1_windows,Bed 1 Windows,,binary_sensor,Alarm\nbinary_sensor.bed_2_window,Bed 2 Window,,binary_sensor,Alarm\nbinary_sensor.bed_3_window,Bed 3 Window,,binary_sensor,Alarm\nbinary_sensor.study_window,Study Window,,binary_sensor,Alarm\nbinary_sensor.lounge_windows,Lounge Windows,,binary_sensor,Alarm\nbinary_sensor.ensuite_window,Ensuite Window,,binary_sensor,Alarm\nbinary_sensor.bathroom_window,Bathroom Window,,binary_sensor,Alarm\nbinary_sensor.front_door,Front Door,,binary_sensor,Alarm\nbinary_sensor.garage_rear_door,Garage Rear Door,,binary_sensor,Alarm\nbinary_sensor.laundry_door,Laundry Door,,binary_sensor,Alarm\nbinary_sensor.lounge_sliding,Lounge Sliding Door,,binary_sensor,Alarm\nbinary_sensor.dining_sliding,Dining Sliding Door,,binary_sensor,Alarm\nswitch.air_conditioner,Air Conditioner,,switch,Hallway\nsensor.hallway_temperature,Hallway Temperature,,sensor,Hallway\ncover.kitchen_blinds,Kitchen Blinds,,cover,Kitchen\nfan.study_fan,Study Fan,,fan,Study\nlight.study_light,Study Light,,light,Study\ncover.garage_door,Garage Door,,cover,Garage\nswitch.heater,Heater,,switch,Family Room\ntodo.chatgpt,Chatgpt,,todo,None\nlight.left_bed_lamp,Left Bed Lamp,,light,Master bedroom\nlight.right_bed_lamp,Right Bed Lamp,,light,Master bedroom\nlight.lounge_cabinet_downlights,Lounge Cabinet Downlights,,light,Living Room\nlight.kitchen_led_strip,Kitchen Led Strip,,light,Kitchen\nlight.family_room_led_strip,Family Room LED Strip,,light,Family Room\nlock.rear_entry_door,Rear entry door,Back door/Rear door,lock,Garage\nfan.air_purifier,Air Purifier,,fan,Master bedroom\ncamera.back_gate,Back Gate,,camera,Outdoors\ncamera.backyard,Backyard,,camera,Outdoors\ncamera.cat_room,Cat Room,,camera,Cat Room\ncamera.driveway,Driveway,,camera,Outdoors\ncamera.family,Family,,camera,Family Room\ncamera.front_door,Front Door,,camera,Outdoors\ncamera.front_gate,Front Gate,,camera,Outdoors\ncamera.front_of_house,Front Of House,,camera,Outdoors\ncamera.garage,Garage,,camera,Garage\ncamera.garage_door,Garage Door,,camera,Outdoors\ncamera.lounge,Lounge,,camera,Living Room\nswitch.tasmota2_3,Rear garden irrigation,,switch,Outdoors\nswitch.tasmota3_2,Rear grass irrigation,,switch,Outdoors\nswitch.tasmota4,Front garden irrigation,,switch,Outdoors\nlight.dining_room_light,Dining room Light,,light,Dining Room\nlight.family_room_lamp,Family room lamp,,light,Family Room\nlight.family_room_light,Family room light,,light,Family Room\nlight.front_hall_light,Front Hall light,,light,Front Hall\nlight.front_porch_light,Front porch light,,light,Front Hall\nlight.front_downlights,Front Downlights,,light,Front Hall\nlight.garage_light,Garage Light,,light,Garage\nlight.hollys_wall_switch,Holly's Light,,light,Holly's Bedroom\nlight.laundry_light,Laundry Light,,light,Laundry\nlight.backyard_lights,Backyard Lights,,light,Laundry\nlight.wardrobe_light,Wardrobe Light,,light,Master bedroom\nlight.tv_led_strip,TV Led Strip,,light,Living Room\nlight.kitchen_light,Kitchen light,,light,Kitchen\nlight.lounge_light,Lounge Light,,light,Living Room\nlight.name_light,Hollyâs name light,,light,Holly's Bedroom\nlight.bathroom_light,Bathroom light,,light,Bathroom\nlight.bedroom_3_light,Bedroom 3 light,,light,Cat Room\nsensor.indoor_temperature,Indoor Temperature,,sensor,Family Room\nsensor.indoor_carbon_dioxide,Indoor Carbon dioxide,,sensor,Family Room\nsensor.indoor_humidity,Indoor Humidity,,sensor,Family Room\nsensor.outdoor_station_temperature,Outside Temperature,,sensor,Outdoors\nsensor.outdoor_station_humidity,Outside Humidity,,sensor,Outdoors\nmedia_player.homepod_mini,Siri HomePod mini,,media_player,Family Room\nlight.display_cabinet_lights,Cabinet lights,Family room cabinet lights,light,Family Room\nbinary_sensor.chrisiphone13,Chrisiphone13,,binary_sensor,None\nlight.chrisiphone13_screen,Chrisiphone13 Screen,,light,None\nmedia_player.chrisiphone13,Chrisiphone13,,media_player,None\nmedia_player.living_room_receiver,Living room receiver,,media_player,None\nmedia_player.ceiling_speakers,Ceiling Speakers ,,media_player,None\n```"}, {'role': 'user', 'content': 'are there any lights on'}, {'role': 'function', 'name': 'get_state', 'content': '```csv entity,state light,None ```'}]
2024-07-01 08:21:56.381 INFO (MainThread) [custom_components.extended_openai_conversation] Response {'id': 'chatcmpl-9fxDgeXSnhQMnD4Pp2tpqfHstRGNf', 'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': 'There are no lights currently on.', 'role': 'assistant'}}], 'created': 1719786116, 'model': 'gpt-3.5-turbo-1106', 'object': 'chat.completion', 'system_fingerprint': 'fp_44132a4de3', 'usage': {'completion_tokens': 8, 'prompt_tokens': 1761, 'total_tokens': 1769}}
Try using the gpt-4o model instead to see if this makes a difference. Iâm just double checking here to be pedantic, but can you also confirm that youâve got âUse Toolsâ enabled in the settings dialogue window?
Use tools was turned off.
Just turned it on to try.
Im constantly getting this sort of error where it tried to use wildcards.
I did get it tell me the correct lights Once, after adding your prompts
GPT 4.0 works much better, but the cost is too high i think for my use case.
Do you have 4.0 enabled. What does yours do if you ask something simple like what lights are on.
My response looks quite similar. But you can customise this in your prompt directives. I run my system with gpt-4o; several commands on a daily basis and I spend about $5 per month depending on how much image processing Iâm doing on top of that. gpt-4o was designed to be better suited for assistant functions. Again, happy to pay at this stage, and it will invariably come down in cost as OpenAI releases more advanced models.
The cost is the primary reason why Iâve optimised the spec function calls. I do not need the assist pipeline to dump irrelevant information to it every time I make a request, but instead use a function call to poll the current state or attributes of an object (this allows more advanced features like setting light colour / temperature / brightness / light effect). The prompt only needs to provide enough information to let the AI know the entities exist.
I run an Ollama server locally as well that spins up a phi3-medium-128k instance to parse long text and articles (see https://www.reddit.com/r/homeassistant/comments/1dq89tj/extended_openai_summarise_articles/).
I have yet to find a open source LLM that I can run locally that gracefully handles function calls or that has a dedicated tools API. The Extended OpenAI plugin works well on local instances, its the LLM and recognising function calls that do not.
Thanks man. Appreciate the help.
ill send you a private message for more help and to chat. Sounds like you know your shit.
No worries ⌠I wouldnât say I know my shit; but certainly smart enough to be dangerous to myself.
So I was using Fallback conversation agent w/ HA as primary and openAi extended as secondary/fallback. But it appears the recent HA updates has broken this. Anyone in a similar situation?
1 Like
What broke? Is it this integration or the fallback agent? If without the fallbak agent the extended gpt integration works then this is the wrong thread. Few weeks back the fallback agent integration broke for me, but it got fixed. You should look into that integration probably.
Is there a way yet to continue conversation when using a wyoming satellite. For instance, i say the wakeword, the satelite doesnât hear properly and asks what device I want to control, id like it to then activate the assistant again and listen for a follow up response. Sometimes ill get a response that requires a follow up. Is this possible yet using the wyoming satelite devices specifically?
Follow up works well when using the chat interface in the home assistant apps.
@Chris_Hemmings Iâve been trying to figure this out for devices on StreamAssist as well. I would hope that whatever solution we find could be universally applicable.
After doing some testing I think the problem is that we donât seem to have a good way of knowing when TTS playback ends, and because of that we donât have an accurate state change to use to trigger something that listens for follow up.
I had the idea to set up a local media player on the server and simultaneously send TTS to that and the endpoint, assuming that a local media player would have no delay when going between idle and playing, but media player states seem to be universally inconsistent. Is there some polling rate that can be adjusted to make it more accurate?
Side note: I made a fork of Stream Assist that retains the conversation ID between interactions for a period of time, but itâs really in need of the ability to continue listening for follow up to get the most use out of it.
I tried this for the âconversation agentâ and was quite shocked at the cost. Asking âAssistâ to turn on a lamp was billed at 27ct. essentially ever query costs at least 27ct which would add up VERY quickly even if you only use it for basics like turning on and off some lamps and the radio.
The Google Gemini thing seems a lot cheaper, but also a lot more dumbâŚ
What model are you using? Iâm on GPT-4o with a pretty long system prompt and 125 entities exposed and my cost is about 1 cent for the initial query and far less for follow ups if continued conversation is used (like in the web or debug interface).
I must admit that I am not sure. I just set it up today and didnât really change any of the default settings