Assist ignore Automations with conversation.process

SchmidSP · June 26, 2025, 1:46pm

I use a automation to ask Assist (linked to Azure OpenAI ChatGPT Conversation) what it can see on a Camera.

The call of the automation works great via Home Assistant App - Assist or via Assist Chat from the PC.

When I use conversation.process (because I also use GitHub - fabianosan/HomeAssistantAssist: Alexa Skill that integrates Home Assistant Assist or your preferred Generative AI via the conversation API and also allows you to open your favorite dashboard on Echo Show.), it ignores the automation.

So how can I use the trigger:

triggers:
  - trigger: conversation
    command:
      - Was siehst Du im Hof
      - Wer steht vor der Haustür
      - Was ist vor der Haustür
      - Wer ist im Hof
      - Siehst Du was im Hof
      - Wer steht im Hof

with action:

action: conversation.process
data:
  text: Wer ist im Hof
  language: de
  agent_id: conversation.azure_openai_conversation

Response over Chat (App, Browser,etc.) → working

init_options:
  start_stage: intent
  end_stage: intent
  input:
    text: Wer steht im Hof
  pipeline: ***
  conversation_id: null
stage: done
run:
  pipeline: ***
  language: de
  conversation_id: ***
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: ***
      language: de
      conversation_id: ***
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2025-06-26T13:42:50.467398+00:00"
  - type: intent-start
    data:
      engine: conversation.azure_openai_conversation
      language: de-DE
      intent_input: Wer steht im Hof
      conversation_id: ***
      device_id: null
      prefer_local_intents: true
    timestamp: "2025-06-26T13:42:50.467427+00:00"
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: Ich kann keine Person oder ein Tier in diesem Bild erkennen.
              extra_data: null
          card: {}
          language: "*"
          response_type: action_done
          data:
            targets: []
            success: []
            failed: []
        conversation_id: ***
        continue_conversation: false
    timestamp: "2025-06-26T13:42:52.500402+00:00"
  - type: run-end
    data: null
    timestamp: "2025-06-26T13:42:52.500469+00:00"
intent:
  engine: conversation.azure_openai_conversation
  language: de-DE
  intent_input: Wer steht im Hof
  conversation_id: ***
  device_id: null
  prefer_local_intents: true
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: Ich kann keine Person oder ein Tier in diesem Bild erkennen.
          extra_data: null
      card: {}
      language: "*"
      response_type: action_done
      data:
        targets: []
        success: []
        failed: []
    conversation_id: ***
    continue_conversation: false

Response over conversation.process (Developer - Action or HomeAssistantAssit Alexa) → not working

response:
  speech:
    plain:
      speech: >-
        Im Hof gibt es aktuell keine Hinweise auf Bewegung oder Anwesenheit. Die
        Lichter im Hof sind aus. Es sind keine Sensoren aktiv, die eine Person
        im Hof erkennen würden.
      extra_data: null
  card: {}
  language: de
  response_type: action_done
  data:
    targets: []
    success: []
    failed: []
conversation_id: ***
continue_conversation: false

Anybody a idea why chat (text / voice) and conversation.process different?

Thanks!

mchk · June 26, 2025, 6:21pm

Assist works correctly with conversation.process and any other text data.

Show us the full code for your automation.

SchmidSP · June 27, 2025, 6:41am

No Problem.

Automation:

alias: "LLM: Was siehst Du im Hof"
description: ""
triggers:
  - trigger: conversation
    command:
      - Was siehst Du im Hof
      - Wer steht vor der Haustür
      - Was ist vor der Haustür
      - Wer ist im Hof
      - Siehst Du was im Hof
      - Wer steht im Hof
conditions: []
actions:
  - action: llmvision.image_analyzer
    *** Shortened because irrelevant ***
    response_variable: response
  - set_conversation_response: "{{response.response_text}}"
mode: single

Call via Assist Web:

stage: done
run:
  pipeline: ***
  language: de
  conversation_id: ***
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: ***
      language: de
      conversation_id: ***
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2025-06-27T06:30:14.715723+00:00"
  - type: intent-start
    data:
      engine: conversation.azure_openai_conversation
      language: de-DE
      intent_input: Wer ist im Hof
      conversation_id: ***
      device_id: null
      prefer_local_intents: true
    timestamp: "2025-06-27T06:30:14.715758+00:00"
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: Ich kann keine Person oder ein Tier in dem Bild erkennen.
              extra_data: null
          card: {}
          language: "*"
          response_type: action_done
          data:
            targets: []
            success: []
            failed: []
        conversation_id: ***
        continue_conversation: false
    timestamp: "2025-06-27T06:30:17.457137+00:00"
  - type: run-end
    data: null
    timestamp: "2025-06-27T06:30:17.457198+00:00"
intent:
  engine: conversation.azure_openai_conversation
  language: de-DE
  intent_input: Wer ist im Hof
  conversation_id: ***
  device_id: null
  prefer_local_intents: true
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: Ich kann keine Person oder ein Tier in dem Bild erkennen.
          extra_data: null
      card: {}
      language: "*"
      response_type: action_done
      data:
        targets: []
        success: []
        failed: []
    conversation_id: ***
    continue_conversation: false

= Working

Call via Action (Dev Console or via HomeAssistantAssist-AlexaSkill)

action: conversation.process
data:
  text: Wer ist im Hof
  language: de
  agent_id: conversation.azure_openai_conversation

Response:

response:
  speech:
    plain:
      speech: >-
        Es gibt keine Sensoren oder Hinweise darauf, dass aktuell jemand im Hof
        ist. Bewegungsmelder im Hofbereich melden nichts.
      extra_data: null
  card: {}
  language: de
  response_type: action_done
  data:
    targets: []
    success: []
    failed: []
conversation_id: ***
continue_conversation: false

= Wrong/differnet answer.

It doesn’t use the automation it use tool assist access.

So again:

Why if I call the sentence in Web or ios app it calls the automation and didn’t call the automation if I use conversation.process action?

Thanks again1

mchk · June 27, 2025, 6:52am

Consider the conversation.process action as a single request to the LLM. It does not interact with other objects unless you explicitly pass information through a variable or attach files. And does not run intents/automations

SchmidSP · June 27, 2025, 6:55am

Sorry again.

But I don’t understand why it works when I use the chat/voice assistant directly, but not via conversation which should be the same for my logic.

If I talk to the LLM via Webchat direct, it works. This is also a single request to LLM.

Why is this treated differently? Does anyone know?

Idea of a work around?

mchk · June 27, 2025, 6:59am

SchmidSP:

  - trigger: conversation
    command:
      - Was siehst Du im Hof
      - Wer steht vor der Haustür
      - Was ist vor der Haustür
      - Wer ist im Hof
      - Siehst Du was im Hof
      - Wer steht im Hof

You may be confused by the naming, but these are completely different things. You specified phrases for the trigger, and when you say them, the automation is triggered.
The conversation.process action has nothing to do with this.

SchmidSP · June 27, 2025, 7:07am

The same question has already been asked here:

github.com/home-assistant/core

Error when trying to use assist or conversation process to trigger automation.

opened 09:21PM - 24 Feb 25 UTC

closed 08:08PM - 18 Jun 25 UTC

limitless-code

stale integration: assist_satellite

### The problem Error in logs when I try to use assist to trigger an automation… or use "conversation process" to trigger automation with sentence as a trigger. Used to work but no longer working. The same automation works when using voice PE assistant device, when using key words announce, tell or forecast. ### What version of Home Assistant Core has the issue? core-2025.2.5 ### What was the last working version of Home Assistant Core? core-2025.1.0 ### What type of installation are you running? Home Assistant OS ### Integration causing the issue Conversation ### Link to integration documentation on our website https://www.home-assistant.io/integrations/conversation/ ### Diagnostics information _No response_ ### Example YAML snippet ```yaml Snippet from automation 1 action: conversation.process metadata: {} data: agent_id: conversation.home_assistant text: Announce, " {{ (var_message) }} " Automation 2 alias: "Announce, Tell or Forecast " description: "" triggers: - trigger: conversation command: - Announce {sentence} - Notify {sentence} - Tell {sentence} id: Announce - trigger: conversation command: Forecast {sentence} id: Forecast conditions: [] actions: - variables: current_volume_level_1: " {{ state_attr('media_player.home_assistant_voice_090503_dining_room_assistant', 'volume_level') }}" current_volume_level_2: " {{ state_attr('media_player.home_assistant_voice_0974ab_small_bedroom_assistant', 'volume_level') }}" enabled: true - wait_template: " {{ is_state('binary_sensor.piper_running', 'on') }} " continue_on_timeout: true timeout: "00:05:00" - action: media_player.volume_set metadata: {} data: volume_level: 0.6 target: entity_id: - media_player.home_assistant_voice_0974ab_small_bedroom_assistant - media_player.home_assistant_voice_090503_dining_room_assistant - action: assist_satellite.announce metadata: {} data: message: " {{ (announce) }} " target: label_id: home - action: media_player.volume_set metadata: {} data: volume_level: " {{ (current_volume_level_1) }} " target: entity_id: media_player.home_assistant_voice_090503_dining_room_assistant - if: - condition: not conditions: - condition: state entity_id: media_player.home_assistant_voice_0974ab state: unavailable then: - action: media_player.volume_set metadata: {} data: volume_level: " {{ (current_volume_level_2) }} " target: entity_id: media_player.home_assistant_voice_0974ab_small_bedroom_assistant - if: - condition: trigger id: - Announce then: - action: notify.mobile_app_poco_f2_pro data: message: " {{ (announce) }} " - action: notify.living_room_tv data: message: " {{ (announce) }} " - action: notify.html5 data: message: " {{ (announce) }} " variables: media_player_id: " {{ trigger.device_id }} " media_player: " {{ device_entities(media_player_id) |select('search','^media_player.')|first }} " friendly_name: " {{ states[media_player]['attributes']['friendly_name'] }} " prepend: media_player. voice_assistant_device: " {{ media_player }} " announce_sentence: " {{ trigger.sentence }} " announce: " {{ (announce_sentence) | regex_replace(find='(^Announce|Notify|Tell|Forecast),?', replace='', ignorecase=True) }} " mode: single ``` ### Anything in the logs that might be useful for us? ```txt Error rendering variables: TemplateError: Invalid domain name '' ``` ### Additional information _No response_

unfortunately without an answer.

If it doesn’t work, it would be great if someone had an idea of how I could still do the automation via:

Home Assistant Voice Assit - works well every call
Home Assistant Voice Assit Text Chat - works well every call
and
Home Assistant Assit (Alexa skill that accesses conversation.process) - it doesn’t work now

maybe someone an idea?

mchk · June 27, 2025, 7:18am

Could you provide a link to the instructions for this method that you used to configure it?

SchmidSP · June 27, 2025, 7:23am

Sure:

Everything (lights, shades, asking for room temperature,…) is
working with this skill over conversation.process in the same way if I use
for example the Text Chat in the Home Assistant App (Assist).

But automations don’t work with this (but with for example the Text Chat in the Home Assistant App Assist).

So I search for a tip to trigger the same automation with the skill as with the Assist.

The skill call is here
HomeAssistantAssist/lambda/lambda_function.py at main · fabianosan/HomeAssistantAssist · GitHub

ha_api_url = "{}/api/conversation/process".format(home_assistant_url)

mchk · June 27, 2025, 7:38am

It is best to check with the authors of these integrations for information about custom integrations.

SchmidSP · June 27, 2025, 8:08am

Please excuse me, but it is not a problem of the author of the skill,
but a friendly question to the forum, why an automation can be started via LLM Voice or Text Chat in HA, but can’t be start via conversation.process.

Maybe someone else in the forum has an idea how this can be solved to make an api call similar to conversation.process to start the automation with a sentence.

Since the problem does not work via HA → Dev → Actions → conversation.process , it is not a problem of the Alexa Skill or the HomaAssistantAssit author.

See manual:
Conversation - Home Assistant
Quote:
" The Conversation integration allows you to converse with Home Assistant. You can either converse by pressing the microphone in the frontend (supported browsers only (no iOS)) or by calling the conversation/process action with the transcribed text."

The instructions state that there is NO difference between frontend and action (as you say).
Again:
However, automations are addressed correct via frontend or microphone, but not via the action conversation.process.

Thank you.

mchk · June 27, 2025, 8:39am

The instructions state that there is NO difference between frontend and action

You are dealing with open source The information is not always up to date.

However, undocumented methods of interacting with the pipeline are best discussed with the author of the integration.

SchmidSP · June 27, 2025, 8:53am

Just forget about the custom skill.

Just try it out:

Pass a sentence via Developer Tools → Actions:

action: conversation.process
data:
  text: Execute my automation
  language: de
  agent_id: conversation.azure_openai_conversation

=
It will not execute automation.

Make the same with

image372×471 8.44 KB

It works.

I can’t see in your issue listing why it shouldn’t work or that the Home Assistant instructions wouldn’t be up to date.

mchk · June 27, 2025, 9:10am

This is not a list of issues, but a log of updates to the documentation page. It can be said that it has not been updated since 2023.
You do not want to accept the information at all; I have already explained everything to you in my previous answers.

Assist ignore Automations with conversation.process

Make the same with image372×471 8.44 KB

Make the same with

image372×471 8.44 KB