Is there anyway to use yarvis as a fallback to the default agent? I want sentences that are specific to homeassistant to be checked first, and then send all unmatched requests to OpenAI to get closer to a natural conversation
I.e. Hey Hal, turn on the lights > matches a assist sentence and turns on the lights
Hey Hal, Open the Pod bay doors, please > forwarded to GPT api > response: “I’m sorry, Dave, I’m afraid I can’t do that.”
No I don’t think that’s possible currently. There is no way to combine/fallback to multiple voice pipelines.
I agree 1000% that this needed!
Well actually there is, I’m doing just that, with the help of custom intent, and nodered.
Shortly described, I created a custom intent, with a wildcard, passing everything assist can’t do on to nodered. In nodered I’m then using my own OpenAI script, and sending the response back to HA.
I’m currently working on doing it with the extended OpenAI conversation, so it should be possible with only HA. I’m not just done yet.
The reason I’m going for assist at first, is faster response time, and to save money on the OpenAI api
That sounds great! Could you show us the custom intent w/wildcard? How do you know when there is no match? Is that in the esp32 firmware or in the yaml intent?
This will help with the custom intent w/wildcard, not with how to know if there is no match. I made one change, for the sentence it is “chat {question}” so it will only use openai when I start a sentence with “chat.”
Hello, I have seen this post and liked the idea, but like the most of you I’d like not to have a special trigger word for OpenAI. I liked the idea of having a fallback for the default agent.
So I gave it a try and implemented that as a custom component. I just finished this 10 minutes ago and didn’t have much time for testing, but i thought you should see this as soon as possible 
Just have a look at the code or try it out as you like. For me it seems to work quite good. Please read the README before setting it up, to understand how it works.
Actually it just allows to setup a list of agents which are called one after the other until an agent returns a successful result.
UPDATE: I have changed the repo-url and I’ve updated it in this post as well
This looks super cool! 
I’m really interested in your approach to creating a fallback mechanism for the default agent without the need for a special trigger word for OpenAI. I’ve had a look at your GitHub repo, and the concept of having a list of agents that are called sequentially until a successful result is obtained seems very practical.
I do have a question about how fast the fallback to OpenAI occurs. Does the system have to wait for a complete response from the initial assistant before it switches to OpenAI, and if so, does this mean there might be a significant delay before receiving an answer? I’m curious about how this impacts the overall responsiveness.
Great job on this, and thanks for sharing!
Hi, thanks for your feedback.
As I understand the API of the agent, I need to wait for the response of the agent to get its response object including the error property.
Fortunately the default agent seems to be quite fast when it comes to intent recognition - or at least to find that there is no match.
So I don’t feel any delay. I could add some debug logging to get an overview of the delays.
However, in my opinion, the delay of the default agent is negligible compared to the performance of ChatGPT itself.
——————
Update:
If the performance is that critical for you, one could call all agents in parallel and take the first (ordered) response. But I don’t find that practical and it could create strange side effects - and costs (OpenAI API)
Another issue I have seen is that the OpenAI integration caches its own conversation history by saving all input and output messages into a history object. With my component, the OpenAI agent doesn’t get all messages, but only those which are passed in as a fallback. So you could not ask OpenAI things related to the history („what was the last thing I turned off?“).
Great work! This should be a core feature.
This way you could add multilanguage too.
That’s a great idea, @DonNL !
I have improved the integration a bit and just released version 0.1.2.
Unfortunately there was a breaking change when I switched from “main” branch to GitHub Releases and additionally I renamed the Git-Repository (from hass to hacs)
So please make sure to remove the integration and delete the repo from HACS. Then add it again using the following repo. From now on, I’ll try to reduce breaking changes as much as possible 
.
For anyone else who wants to get more examples, here is my last Assist test conversation.
(I have told OpenAI to answer as a pirate )
You can find all the instructions to set this up in the README.md of the repo.
Funnily enough, I made the same thing a couple weeks ago!
I implemented it a little differently, mainly due to this being the first custom integration I’ve ever written.
Great minds think alike I guess!
One important feature for me that I included was debugging. I wanted the ability to see what conversation agent responded, and even what the failures were of each conversation agent, right in the final response, assuming debugging is enabled.
Thought you might find my implementation interesting as well.
Just wanted to chime in and thank you both, this was one of the last remaining blockers from my perspective in getting to a fully functional VA model for the house.
I ended up trying both repos linked here last night and did notice one behavioral difference I thought would be worth lifting up as I’m not sure if its WAD or not.
For @t0bst4r 's implementation, when I use a custom sentence I’ve implemented that utilizes a wildcard (e.g. “play classical radio on the living room speakers” where ‘classical radio’ is a wildcard slot) I found the chained agent skips Home Assistant and goes right to the backup agent (OpenAI in my case), failing to play the music as expected. All other custom sentences that do not have wildcard slots seem to work as expected.
When using @marisa 's implementation, the behavior is as expected: a request for “play classical music on the living room speakers” is recognized as an HA intent and the appropriate script is executed.
Not sure if anyone else is seeing this behavior or its WAD, but wanted to share. Again, thank you both for sharing your work here!
If I had to take a guess as to the reason, I assume it’s because @t0bst4r 's implementation doesn’t pass a language along, where as my implementation does. Intents are based on language, and custom intents probably just don’t trigger unless a language is provided.
Interesting! I’m fairly certain my other custom intents that did not have wildcard slots triggered fine, so I wonder if its tied to that wildcard? Either way, I appreciate the work and thought!
Hey there,
First i’d like to thank @kbromer and all the others for sharing your experience here, since i didn’t have that much time for testing and improving things in the last few weeks.
Next, i’ve had a look at @marisa 's repo and i love it
This was also my first integration and i just forked the OpenAI integration and changed the handler as needed. But i didn’t know about all the features already existing in the conversation package which make things a lot easier and more stable at the same time (e.g. Agent Manager).
I see no reason proceeding my integration from now on. Instead i’d offer my help to contribute in @marisa 's integration.
First i’ll mark my repo as deprecated / archived and link to marisas repo. Then i’d like to go through marisa’s code again and see where i could help or add things.
One thing I liked about yours was when you clicked the gear icon, you had another gear icon for the chatgtp settings. Its missing from the one @marisa did.
Thes are great. Will help to keep the chagtp bill down…
Yes, that’s quite easy to fix actually. Marisa is using a custom SelectSelector passing in the possible options by hand. I used a different Selector: the ConversationAgentSelector. It is the same selector component as in the assist pipeline setup screen.
That’s one thing i had in mind i’d like to improve when i get the time to do so.
@marisa i already archived my project in github and created a last release with a deprecation warning. I also switched to your integration in my HA installation
Yes, that’s quite easy to fix actually. Marisa is using a custom SelectSelector passing in the possible options by hand. I used a different Selector: the ConversationAgentSelector. It is the same selector component as in the assist pipeline setup screen.
Yeah, that is one change I’d like to make. I also like having the option of a collection of agents as opposed to just 2 options, and I was looking to see what the best way of implementing that was, or if I should just use your method. But alas, limited time this week.
Happy to work on this together.
To be honest, I cannot imagine one real use case where I would need three agents… ChatGPT will not return an error. Therefore there cannot be a fallback after that. And if you really need more than one fallback, you could create another fallback agent, which is using the first one as primary.
So I would not prioritize the multi-fallback feature.
PS: maybe I‘ll have some time tomorrow to implement and test the ConversationAgentSelector with your integration.
