Multi Agents in Home Assistant using Extended OpenAI Conversation

Share your Projects!
21

I still use an Extended OpenAI Agent to act as a bridge between HA and my own Python agents.

Prompt Template:

I want you to act as Smart AI manager. You do not respond to queries directly. Instead, you pass them to a relevant AI Agent to respond to.

You have access to the following AI Agents:

meteorologist_agent - Can provide weather forecasts
executive_assistant_agent - Can interact with the user's calendar and todo lists
smart_home_agent - Can interact with the user's smart home and smart home devices

Functions:

- spec:
    name: call_agent_by_id
    description: Pass user query to relevant AI Agent
    parameters:
      type: object
      properties:
        query:
          type: string
          description: The users query
        agent_id:
          type: string
          description: ID of the AI Agent
          enum:
          - meteorologist_agent
          - executive_assistant_agent
          - smart_home_agent
      required:
      - query
      - agent_id
  function:
    type: composite
    sequence:
    - type: script
      sequence:
      - service: rest_command.call_agent
        data:
          query: "{{ query }}"
          agent: "{{ agent_id }}"
        response_variable: _function_result
      response_variable: res
    - type: template
      value_template: >-
        {% set res = res.content %}
        {{ {'agent_response': res} }}
1 Like
22

Just one follow up question:

How/where do you write the custom code for additional tools like web scraping, api calls, etc ?

I am kinda lost how to approach this custom side of home assistant

23

I started tackling a similar problem last night. What I ended up doing was installing the N8N Add-On via HACS. I created a custom conversation agent to connect to the chat webhook endpoint. This allows me to create complex workflows which use different LLM models and tools to successfully respond to a given request. N8N turned out to be surprisingly robust and performant. It is no/low-code so it abstracts me from having to deal with the complexity of using LangChain and it can run locally.

If folks are interested I will HACS-enable the conversation agent once I’ve tested and debugged it and post the repo here for you to try out.

Now, if only there was a way to pass metadata from the speech-to-text agent to the conversation agent so I could enable my workflow agent to have an idea who is speaking, I’d really be cooking with fire. As far as I can tell this isn’t possible with the current voice pipeline, but if anyone knows a reliable way to do this within the context of the pipeline please let me know.

7 Likes
24

I tried to use this mechanism but failed.

I want to make use to the Home Assistant ChatGPT/OpenAI integration’s internet search capabilities and call that as a voice assistant agent from within the extended OpenAI Conversation HACS component. But keep all other functionality inside this extended OpenAI Conversation Voice Assistant.

I added the call_agent_by_id function and extended my existing full extended OpenAI conversation prompt following text: When you can’t answer the user’s request as a smart home manager pass the query to the Internet search AI agent. Use following agent ID ‘’ of the Internet search AI agent when using the call_agent_by_id tool for this:

But it didn’t work. I get “Unexpected error during intent recognition”

25

That means it failed when it ran the intent. there should be a log line telling you what failed.

26

This is what I could find, but it’s Chinese to me.

Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/assist_pipeline/pipeline.py", line 1189, in recognize_intent
    conversation_result = await conversation.async_converse(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
    )
    ^
  File "/usr/src/homeassistant/homeassistant/components/conversation/agent_manager.py", line 117, in async_converse
    result = await method(conversation_input)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 275, in async_process
    return await self.async_handle_message(user_input, chat_log)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 197, in async_handle_message
    query_response = await self.query(user_input, chat_log.content, exposed_entities, 0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 379, in query
    return await self.execute_function_call(user_input, messages, message, exposed_entities, n_requests + 1)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 402, in execute_function_call
    return await self.execute_function(user_input, messages, message, exposed_entities, n_requests, function)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/extended_openai_conversation/__init__.py", line 421, in execute_function
    result = await function_executor.execute(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        self.hass, function["function"], arguments, user_input, exposed_entities
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/config/custom_components/extended_openai_conversation/helpers.py", line 652, in execute
    result = await function_executor.execute(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        hass, executor_config, arguments, user_input, exposed_entities
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/config/custom_components/extended_openai_conversation/helpers.py", line 457, in execute
    result = await script.async_run(
             ^^^^^^^^^^^^^^^^^^^^^^^
        run_variables=arguments, context=user_input.context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1827, in async_run
    return await asyncio.shield(create_eager_task(run.async_run()))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 460, in async_run
    await self._async_step(log_exceptions=False)
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 526, in _async_step
    self._handle_exception(
    ~~~~~~~~~~~~~~~~~~~~~~^
        ex, continue_on_error, self._log_exceptions or log_exceptions
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 556, in _handle_exception
    raise exception
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 524, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 1008, in _async_step_call_service
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<9 lines>...
    )
    ^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 624, in _async_run_long_action
    return await long_task
           ^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2759, in async_call
    processed_data: dict[str, Any] = handler.schema(service_data)
                                     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/voluptuous/schema_builder.py", line 205, in __call__
    return self._compiled([], data)
           ~~~~~~~~~~~~~~^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/voluptuous/schema_builder.py", line 549, in validate_dict
    return base_validate(path, data.items(), out)
  File "/usr/local/lib/python3.13/site-packages/voluptuous/schema_builder.py", line 382, in validate_mapping
    raise er.MultipleInvalid(errors)
voluptuous.error.MultipleInvalid: invalid agent ID for dictionary value @ data['agent_id']
1 Like
27

Read the very last line - invalid agent_id

You’re not passing something correctly. (which agent to invoke).

28

OK, thanks. I assumed this is the agent ID, am I mistaken?

image
image811×152 7.3 KB