Ask OpenAI questions from your default conversation agent!

t0bst4r · February 26, 2024, 8:46pm

Hey there,

First i’d like to thank @kbromer and all the others for sharing your experience here, since i didn’t have that much time for testing and improving things in the last few weeks.

Next, i’ve had a look at @marisa 's repo and i love it
This was also my first integration and i just forked the OpenAI integration and changed the handler as needed. But i didn’t know about all the features already existing in the conversation package which make things a lot easier and more stable at the same time (e.g. Agent Manager).

I see no reason proceeding my integration from now on. Instead i’d offer my help to contribute in @marisa 's integration.

First i’ll mark my repo as deprecated / archived and link to marisas repo. Then i’d like to go through marisa’s code again and see where i could help or add things.

Rich37804 · February 26, 2024, 9:12pm

One thing I liked about yours was when you clicked the gear icon, you had another gear icon for the chatgtp settings. Its missing from the one @marisa did.
Thes are great. Will help to keep the chagtp bill down…

t0bst4r · February 26, 2024, 9:28pm

Yes, that’s quite easy to fix actually. Marisa is using a custom SelectSelector passing in the possible options by hand. I used a different Selector: the ConversationAgentSelector. It is the same selector component as in the assist pipeline setup screen.

That’s one thing i had in mind i’d like to improve when i get the time to do so.

@marisa i already archived my project in github and created a last release with a deprecation warning. I also switched to your integration in my HA installation

marisa · February 27, 2024, 4:30pm

Yes, that’s quite easy to fix actually. Marisa is using a custom SelectSelector passing in the possible options by hand. I used a different Selector: the ConversationAgentSelector. It is the same selector component as in the assist pipeline setup screen.

Yeah, that is one change I’d like to make. I also like having the option of a collection of agents as opposed to just 2 options, and I was looking to see what the best way of implementing that was, or if I should just use your method. But alas, limited time this week.

Happy to work on this together.

t0bst4r · February 27, 2024, 10:04pm

To be honest, I cannot imagine one real use case where I would need three agents… ChatGPT will not return an error. Therefore there cannot be a fallback after that. And if you really need more than one fallback, you could create another fallback agent, which is using the first one as primary.
So I would not prioritize the multi-fallback feature.

PS: maybe I‘ll have some time tomorrow to implement and test the ConversationAgentSelector with your integration.

marisa · February 28, 2024, 12:32am

Yeah, that’s a fair point, and as of now, I haven’t seen any conversation agents that would provide functionality that you might want between HomeAssistant’s built in and an LLM solution.

Rich37804 · May 3, 2024, 5:14pm

Your integration has been broken for a few days now. Are you planning on fixing it?

marisa · May 16, 2024, 7:22am

Hey, sorry about that. I wasn’t aware of the breaking change, and was taking a mental health break this month from all things computer (due to burnout).

Thankfully, someone submitted a fix and it has no been merged and released.

I apologize for that.

paddy0174 · May 16, 2024, 11:35am

I’m sorry, but I feel the urgent need to comment on that.

You don’t have to apologize for anything! If you’re not well, or taking a pro-active break to care for your wellbeing, there is no need to apologize!

Hope you’re better now, otherwise extend the break! Nothing here that’s time critical!

Localuser5 · May 16, 2024, 6:19pm

My apologizes for asking for help but I’m a little stuck here… My OpenAI assistant works (using OpenAI Conversation) and my HA assistant works but when I set the Fallback Conversation Agent as the default Conversation agent I get the error below

Core - 2024.5.3
Supervisor - 2024.05.1
Operating System - 12.3
fallback_conversation - v1.0.2

Debug Log: 2024-05-16 13:44:17.158 ERROR (MainThread) [homeassistant.components.assist_pipe - Pastebin.com

Screenshot 2024-05-16 at 2.19.10 PM

Rich37804 · May 16, 2024, 11:23pm

Absolutely no reason to apologise for that! You need to unplug and decompress every now and then. Ive been there…

marisa · May 17, 2024, 8:05pm

You set your primary conversation agent to the Fallback conversation agent from the looks of it, which causes it to be called recursively. I think because you called it “Home Assistant”, you picked the wrong one.

Localuser5 · May 17, 2024, 8:41pm

I called it “Fallback Conversation Agent Test 1” but I assumed in the Fallback options you wouldn’t select that and you’d want “Home Assistant” setas the primary for HA stuff and “OpenAI Conversation” for all the other silly stuff.

I did I set it as “Fallback Conversation Agent Test 1” in the Voice Assistants menu under “Conversation agent”

Screenshot 2024-05-17 at 4.34.57 PM

DesNZ · May 18, 2024, 10:51pm

I’ve got the same issue:

Here’s the debug file from home assistant:

[
  {
    "init_options": {
      "start_stage": "intent",
      "end_stage": "intent",
      "input": {
        "text": "tallest nz building"
      },
      "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
      "conversation_id": null
    },
    "stage": "done",
    "run": {
      "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
      "language": "en",
      "runner_data": {
        "stt_binary_handler_id": null,
        "timeout": 300
      }
    },
    "events": [
      {
        "type": "run-start",
        "data": {
          "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
          "language": "en",
          "runner_data": {
            "stt_binary_handler_id": null,
            "timeout": 300
          }
        },
        "timestamp": "2024-05-18T22:46:30.719952+00:00"
      },
      {
        "type": "intent-start",
        "data": {
          "engine": "b683f3e4faf931d692d5251faa70a9c2",
          "language": "en",
          "intent_input": "tallest nz building",
          "conversation_id": null,
          "device_id": null
        },
        "timestamp": "2024-05-18T22:46:30.719977+00:00"
      },
      {
        "type": "error",
        "data": {
          "code": "intent-failed",
          "message": "Unexpected error during intent recognition"
        },
        "timestamp": "2024-05-18T22:46:30.776813+00:00"
      },
      {
        "type": "run-end",
        "data": null,
        "timestamp": "2024-05-18T22:46:30.776847+00:00"
      }
    ],
    "intent": {
      "engine": "b683f3e4faf931d692d5251faa70a9c2",
      "language": "en",
      "intent_input": "tallest nz building",
      "conversation_id": null,
      "device_id": null,
      "done": false
    },
    "error": {
      "code": "intent-failed",
      "message": "Unexpected error during intent recognition"
    }
  },
  {
    "init_options": {
      "start_stage": "intent",
      "end_stage": "intent",
      "input": {
        "text": "turn on bedroom light"
      },
      "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
      "conversation_id": null
    },
    "stage": "done",
    "run": {
      "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
      "language": "en",
      "runner_data": {
        "stt_binary_handler_id": null,
        "timeout": 300
      }
    },
    "events": [
      {
        "type": "run-start",
        "data": {
          "pipeline": "01hwp9m6dzewmfdvg5tkyefd51",
          "language": "en",
          "runner_data": {
            "stt_binary_handler_id": null,
            "timeout": 300
          }
        },
        "timestamp": "2024-05-18T22:46:27.841632+00:00"
      },
      {
        "type": "intent-start",
        "data": {
          "engine": "b683f3e4faf931d692d5251faa70a9c2",
          "language": "en",
          "intent_input": "turn on bedroom light",
          "conversation_id": null,
          "device_id": null
        },
        "timestamp": "2024-05-18T22:46:27.841648+00:00"
      },
      {
        "type": "intent-end",
        "data": {
          "intent_output": {
            "response": {
              "speech": {
                "plain": {
                  "speech": "Home Assistant responded with: Turned on the light",
                  "extra_data": null,
                  "original_speech": "Turned on the light",
                  "agent_name": "Home Assistant",
                  "agent_id": "conversation.home_assistant"
                }
              },
              "card": {},
              "language": "en",
              "response_type": "action_done",
              "data": {
                "targets": [],
                "success": [
                  {
                    "name": "Philips LCT010 Light",
                    "type": "entity",
                    "id": "light.philips_lct010_light"
                  }
                ],
                "failed": []
              }
            },
            "conversation_id": null
          }
        },
        "timestamp": "2024-05-18T22:46:27.888531+00:00"
      },
      {
        "type": "run-end",
        "data": null,
        "timestamp": "2024-05-18T22:46:27.888572+00:00"
      }
    ],
    "intent": {
      "engine": "b683f3e4faf931d692d5251faa70a9c2",
      "language": "en",
      "intent_input": "turn on bedroom light",
      "conversation_id": null,
      "device_id": null,
      "done": true,
      "intent_output": {
        "response": {
          "speech": {
            "plain": {
              "speech": "Home Assistant responded with: Turned on the light",
              "extra_data": null,
              "original_speech": "Turned on the light",
              "agent_name": "Home Assistant",
              "agent_id": "conversation.home_assistant"
            }
          },
          "card": {},
          "language": "en",
          "response_type": "action_done",
          "data": {
            "targets": [],
            "success": [
              {
                "name": "Philips LCT010 Light",
                "type": "entity",
                "id": "light.philips_lct010_light"
              }
            ],
            "failed": []
          }
        },
        "conversation_id": null
      }
    }
  }
]

As you can see calling local voice assist is fine but calling openai is failing.

My voice assist looks like this:

When calling openai directly it works:

[
  {
    "init_options": {
      "start_stage": "intent",
      "end_stage": "intent",
      "input": {
        "text": "tallest nz building"
      },
      "pipeline": "01hxns8pp87j6pw7c6fvs4tnjs",
      "conversation_id": null
    },
    "stage": "done",
    "run": {
      "pipeline": "01hxns8pp87j6pw7c6fvs4tnjs",
      "language": "en",
      "runner_data": {
        "stt_binary_handler_id": null,
        "timeout": 300
      }
    },
    "events": [
      {
        "type": "run-start",
        "data": {
          "pipeline": "01hxns8pp87j6pw7c6fvs4tnjs",
          "language": "en",
          "runner_data": {
            "stt_binary_handler_id": null,
            "timeout": 300
          }
        },
        "timestamp": "2024-05-18T22:55:31.022293+00:00"
      },
      {
        "type": "intent-start",
        "data": {
          "engine": "conversation.openai_conversation",
          "language": "*",
          "intent_input": "tallest nz building",
          "conversation_id": null,
          "device_id": null
        },
        "timestamp": "2024-05-18T22:55:31.022319+00:00"
      },
      {
        "type": "intent-end",
        "data": {
          "intent_output": {
            "response": {
              "speech": {
                "plain": {
                  "speech": "The tallest building in New Zealand is the Sky Tower in Auckland, standing at 328 meters (1,076 feet) tall.",
                  "extra_data": null
                }
              },
              "card": {},
              "language": "*",
              "response_type": "action_done",
              "data": {
                "targets": [],
                "success": [],
                "failed": []
              }
            },
            "conversation_id": "01HY6YNQPESTJSMXY87EST9N0B"
          }
        },
        "timestamp": "2024-05-18T22:55:32.176357+00:00"
      },
      {
        "type": "run-end",
        "data": null,
        "timestamp": "2024-05-18T22:55:32.176410+00:00"
      }
    ],
    "intent": {
      "engine": "conversation.openai_conversation",
      "language": "*",
      "intent_input": "tallest nz building",
      "conversation_id": null,
      "device_id": null,
      "done": true,
      "intent_output": {
        "response": {
          "speech": {
            "plain": {
              "speech": "The tallest building in New Zealand is the Sky Tower in Auckland, standing at 328 meters (1,076 feet) tall.",
              "extra_data": null
            }
          },
          "card": {},
          "language": "*",
          "response_type": "action_done",
          "data": {
            "targets": [],
            "success": [],
            "failed": []
          }
        },
        "conversation_id": "01HY6YNQPESTJSMXY87EST9N0B"
      }
    }
  }
]

@Localuser5 Have you found a fix yet?

marisa · May 19, 2024, 5:20pm

Oh, huh… I must have gotten confused I guess because the name of your pipeline was “Home Assistant”, sorry about that.

I suspect because of these lines:

2024-05-16 13:44:33.102 WARNING (MainThread) [homeassistant.components.conversation.agent_manager] Conversation agent <custom_components.fallback_conversation.FallbackConversationAgent object at 0x7fb76e13d0d0> is still loaded after config entry removal

There may be something misconfigured. Have you tried restarting Home Assistant?

But beyond that, I am not sure, it’s hard to tell.

marisa · May 19, 2024, 5:23pm

Do you have debug logging on, can you share system logs related to it as well?

Odd that it would behave this way.

Also, could you set debug level to Verbose and see what the responses are? This could potentially help. Also, what is your Fallback Agent’s settings looking like?

This is weird, I’ve not seen this behaviour before.

DesNZ · May 21, 2024, 10:24am

Setting my debug level to verbose doesn’t seem to show much:

Here are my logs with debug enabled on your integration and openai:

2024-05-21 22:19:12.386 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:19:12.386 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration nodered which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:19:12.387 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration fallback_conversation which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:19:12.387 WARNING (SyncWorker_0) [homeassistant.loader] We found a custom integration adaptive_lighting which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:19:19.636 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues
2024-05-21 22:19:26.365 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues
2024-05-21 22:22:09.339 DEBUG (MainThread) [custom_components.fallback_conversation] Processing in en using conversation.home_assistant with debug level verbose: turn on bedroom light
2024-05-21 22:22:15.102 DEBUG (MainThread) [custom_components.fallback_conversation] Processing in en using conversation.home_assistant with debug level verbose: nz tallest building
2024-05-21 22:22:15.185 ERROR (MainThread) [homeassistant.components.assist_pipeline.pipeline] Unexpected error during intent recognition
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/assist_pipeline/pipeline.py", line 994, in recognize_intent
    conversation_result = await conversation.async_converse(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/conversation/agent_manager.py", line 87, in async_converse
    return await method(
           ^^^^^^^^^^^^^
  File "/config/custom_components/fallback_conversation/__init__.py", line 90, in async_process
    agent_names[agent_id],
    ~~~~~~~~~~~^^^^^^^^^^
KeyError: 'conversation.openai_conversation'

Edit: Further logs after doing some tweaking:

2024-05-21 22:30:39.577 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration hacs which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:30:39.578 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration nodered which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:30:39.578 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration fallback_conversation which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:30:39.578 WARNING (SyncWorker_1) [homeassistant.loader] We found a custom integration adaptive_lighting which has not been tested by Home Assistant. This component might cause stability problems, be sure to disable it if you experience issues with Home Assistant
2024-05-21 22:30:41.925 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.components.frontend. This is deprecated and will stop working in Home Assistant 2024.9, it should be updated to import functions used from frontend directly at custom_components/hacs/frontend.py, line 68: hass.components.frontend.async_register_built_in_panel(, please create a bug report at https://github.com/hacs/integration/issues
2024-05-21 22:30:49.721 WARNING (MainThread) [homeassistant.helpers.frame] Detected that custom integration 'hacs' accesses hass.helpers.event. This is deprecated and will stop working in Home Assistant 2024.11, it should be updated to import functions used from event directly at custom_components/hacs/base.py, line 630: self.hass.helpers.event.async_track_time_interval(, please create a bug report at https://github.com/hacs/integration/issues
2024-05-21 22:31:21.970 ERROR (MainThread) [homeassistant.components.assist_pipeline.pipeline] Unexpected error during intent recognition
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/assist_pipeline/pipeline.py", line 994, in recognize_intent
    conversation_result = await conversation.async_converse(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/homeassistant/components/conversation/agent_manager.py", line 87, in async_converse
    return await method(
           ^^^^^^^^^^^^^
  File "/config/custom_components/fallback_conversation/__init__.py", line 90, in async_process
    agent_names[agent_id],
    ~~~~~~~~~~~^^^^^^^^^^
KeyError: 'conversation.openai_conversation'

Edit 2:

In an attempt to fix this error I did some research and faulty yaml files regarding intents often showed this error. Clearing and reconfiguring the core.config_entries did nothing. I did see that yarvis left some junk data there after removing that integration. Possibly related.

Rich37804 · May 21, 2024, 10:42am

Are you using Open AI conversation, or Extended Open AI conversation? You want to use Extended Open AI conversation.

DesNZ · May 21, 2024, 10:55am

AFAIK this integration doesn’t require extended openai because calls to ha services are handled by my local assistant and only passed to openai as fallback when ha doesn’t have a response.

In saying that extended openai worked, however differently to how I see the integration work as above.

Edit:

Changing the prompt on extended openai gives me the desired output.

Is there any reason this doesn’t work with the built in integration?

Thanks for your help!

t0bst4r · May 21, 2024, 12:38pm

Just one idea, without checking the code:
Did they change anything in the default Home Assistant agent how unrecognized intents are handled?

The fallback integration assumes that the primary agents returns a specific error object in case the intent was not recognized. Then the fallback mechanism kicks in.
For me the screenshots look like that the HA agent prints the red error message and then responds with a valid (but wrong) answer „turned off the light“. And for the fallback agent it looks like that the primary HA agent handled the intent successfully.

Or am I wrong?