LLM Assistant web search tool/intent

rjmarques · January 5, 2025, 2:15pm

Hey everyone,

I’m trying to get the LLM voices assistants to use a web search intent I’ve been developing, in other to find accurate information on the web when trying to respond to the user’s prompt.

I’ve created a new Intent that integrates with the Tavily web search API. The OpenAI conversation integration is able to use it and it seems to work fine.

However, one limitation I’m facing is that I must give the LLM control over home assistant for the tools (intents) to be loaded. In other words, if this config is not available, the system tools (intents) are not used:

github.com

home-assistant/core/blob/32b7b5aa66faa087c5273402de591f6175a99754/homeassistant/components/openai_conversation/conversation.py#L140


      
          user_name: str | None = None
          llm_context = llm.LLMContext(
              platform=DOMAIN,
              context=user_input.context,
              user_prompt=user_input.text,
              language=user_input.language,
              assistant=conversation.DOMAIN,
              device_id=user_input.device_id,
          )
          
          if options.get(CONF_LLM_HASS_API):
              try:
                  llm_api = await llm.async_get_api(
                      self.hass,
                      options[CONF_LLM_HASS_API],
                      llm_context,
                  )
              except HomeAssistantError as err:
                  LOGGER.error("Error getting LLM API: %s", err)
                  intent_response.async_set_error(
                      intent.IntentResponseErrorCode.UNKNOWN,

I undertstand the logic, but it would be cool to have intents that can be made available to all conversation entities, even if the latter can’t control other entities within home assistant.

Or maybe I’m missing something, and there’s currently a way to define an intent that always gets made available to conversation entities?

Thanks in advance!

Cheers,
Ricardo