Issue with parsing specific response by Conversation Agent

[croogie (Michał Świtoń)]

Hello HA Community! It’s my first post. I tried to put everything I see relevant to understand my problem but maybe something is missing - so tell me and I will fill the gap.

My agent call have issues with properly process the response from the model.
I believe that I see the issue but have no clue how to fix that.

Let me show two calls to the same model. I’m using Extended OpenAI Conversation.

First message: proper one

So the first one is just calling agent (built using Docker Agent). The thing is answer does not require usage of any tools.

“who are you?”

System prompt address this so the answer is immediate: “Jestem Chef Zostałem stworzony przez Michała Świtonia” (it’s in Polish but it doesn’t matter)

Directly calling API returns:

{
	"id": "chatcmpl-9daaab51-ec45-4736-90ec-dcbae627c29a",
	"object": "chat.completion",
	"created": 1789121717,
	"model": "root",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Jestem Chef Zostałem stworzony przez Michała Świtonia"
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 2008,
		"completion_tokens": 21,
		"total_tokens": 2029
	}
}

and here is the result of Debug for Conversation Agent:

stage: done
run:
  pipeline: 01m26sbzekmd14rxpxrsfsw3mm
  language: pl
  conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
  satellite_id: assist_satellite.home_assistant_voice_09a897_urzadzenie_asystenta_glosowego
  tts_output:
    token: G8f1FB-9LOYBDwwaG61jjg.flac
    url: /api/tts_proxy/G8f1FB-9LOYBDwwaG61jjg.flac
    mime_type: audio/flac
    stream_response: true
events:
  - type: run-start
    data:
      pipeline: 01m26sbzekmd14rxpxrsfsw3mm
      language: pl
      conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
      satellite_id: >-
        assist_satellite.home_assistant_voice_09a897_urzadzenie_asystenta_glosowego
      tts_output:
        token: G8f1FB-9LOYBDwwaG61jjg.flac
        url: /api/tts_proxy/G8f1FB-9LOYBDwwaG61jjg.flac
        mime_type: audio/flac
        stream_response: true
    timestamp: '2026-09-11T09:44:23.090698+00:00'
  - type: stt-start
    data:
      engine: stt.home_assistant_cloud
      metadata:
        language: pl-PL
        format: wav
        codec: pcm
        bit_rate: 16
        sample_rate: 16000
        channel: 1
      audio_processing:
        requires_external_vad: true
        prefers_auto_gain_enabled: true
        prefers_noise_reduction_enabled: true
    timestamp: '2026-09-11T09:44:23.091788+00:00'
  - type: stt-vad-start
    data:
      timestamp: 1200
    timestamp: '2026-09-11T09:44:24.256647+00:00'
  - type: stt-vad-end
    data:
      timestamp: 2670
    timestamp: '2026-09-11T09:44:25.722079+00:00'
  - type: stt-end
    data:
      stt_output:
        text: Jak masz na imię?
    timestamp: '2026-09-11T09:44:25.778141+00:00'
  - type: intent-start
    data:
      engine: conversation.extended_openai_conversation
      language: pl-PL
      intent_input: Jak masz na imię?
      conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
      device_id: 5cdab4b7306e8adaf75e46ab646a7999
      satellite_id: >-
        assist_satellite.home_assistant_voice_09a897_urzadzenie_asystenta_glosowego
      prefer_local_intents: true
    timestamp: '2026-09-11T09:44:25.779276+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        role: assistant
    timestamp: '2026-09-11T09:44:25.833025+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: J
    timestamp: '2026-09-11T09:44:27.584072+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: est
    timestamp: '2026-09-11T09:44:27.585450+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: em
    timestamp: '2026-09-11T09:44:27.586548+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Chef'
    timestamp: '2026-09-11T09:44:27.590763+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: .
    timestamp: '2026-09-11T09:44:27.592259+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Z'
    timestamp: '2026-09-11T09:44:27.703823+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: osta
    timestamp: '2026-09-11T09:44:27.709939+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: łem
    timestamp: '2026-09-11T09:44:27.712076+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' st'
    timestamp: '2026-09-11T09:44:27.714191+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: w
    timestamp: '2026-09-11T09:44:27.717744+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: orz
    timestamp: '2026-09-11T09:44:27.724476+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ony
    timestamp: '2026-09-11T09:44:27.727291+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' przez'
    timestamp: '2026-09-11T09:44:27.731339+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Micha'
    timestamp: '2026-09-11T09:44:27.732578+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ła
    timestamp: '2026-09-11T09:44:27.861375+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Ś'
    timestamp: '2026-09-11T09:44:27.866378+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: wit
    timestamp: '2026-09-11T09:44:27.868025+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: onia
    timestamp: '2026-09-11T09:44:27.870468+00:00'
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: Jestem Chef. Zostałem stworzony przez Michała Świtonia
              extra_data: null
          card: {}
          language: pl-PL
          response_type: action_done
          data:
            success: []
            failed: []
        conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
        continue_conversation: false
    timestamp: '2026-09-11T09:44:28.164762+00:00'
  - type: tts-start
    data:
      engine: tts.home_assistant_cloud
      language: pl-PL
      voice: MarekNeural
      tts_input: Jestem Chef. Zostałem stworzony przez Michała Świtonia
      acknowledge_override: false
    timestamp: '2026-09-11T09:44:28.169479+00:00'
  - type: tts-end
    data:
      tts_output:
        media_id: media-source://tts/-stream-/G8f1FB-9LOYBDwwaG61jjg.flac
        token: G8f1FB-9LOYBDwwaG61jjg.flac
        url: /api/tts_proxy/G8f1FB-9LOYBDwwaG61jjg.flac
        mime_type: audio/flac
    timestamp: '2026-09-11T09:44:28.174865+00:00'
  - type: run-end
    data: null
    timestamp: '2026-09-11T09:44:28.175693+00:00'
started: 2026-09-11T09:44:23.090Z
stt:
  engine: stt.home_assistant_cloud
  metadata:
    language: pl-PL
    format: wav
    codec: pcm
    bit_rate: 16
    sample_rate: 16000
    channel: 1
  audio_processing:
    requires_external_vad: true
    prefers_auto_gain_enabled: true
    prefers_noise_reduction_enabled: true
  done: true
  stt_output:
    text: Jak masz na imię?
intent:
  engine: conversation.extended_openai_conversation
  language: pl-PL
  intent_input: Jak masz na imię?
  conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
  device_id: 5cdab4b7306e8adaf75e46ab646a7999
  satellite_id: assist_satellite.home_assistant_voice_09a897_urzadzenie_asystenta_glosowego
  prefer_local_intents: true
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: Jestem Chef. Zostałem stworzony przez Michała Świtonia
          extra_data: null
      card: {}
      language: pl-PL
      response_type: action_done
      data:
        success: []
        failed: []
    conversation_id: 01M27XBS3FXZ0PE5YAMQPHJHFM
    continue_conversation: false
tts:
  engine: tts.home_assistant_cloud
  language: pl-PL
  voice: MarekNeural
  tts_input: Jestem Chef. Zostałem stworzony przez Michała Świtonia
  acknowledge_override: false
  done: true
  tts_output:
    media_id: media-source://tts/-stream-/G8f1FB-9LOYBDwwaG61jjg.flac
    token: G8f1FB-9LOYBDwwaG61jjg.flac
    url: /api/tts_proxy/G8f1FB-9LOYBDwwaG61jjg.flac
    mime_type: audio/flac
finished: 2026-09-11T09:44:28.175Z

and in the end a message is properly processed and returned to me (written or said on my HA Voice PE)

Second message: failed one

Tell me what shopping lists do you have available

This this processing requires tool calls.

Raw answer from model look like this:

{
	"id": "chatcmpl-202b8c60-c210-461c-9d39-11df9def386a",
	"object": "chat.completion",
	"created": 1789122135,
	"model": "root",
	"choices": [
		{
			"index": 0,
			"message": {
				"role": "assistant",
				"content": "Dostępne listy to Spożywcze Pielęgnacja i zdrowie Dom i wyposażenie oraz Dzieciaki",
				"tool_calls": [
					{
						"id": "call_jtwHxas14ImHZLhzMySJ7qom",
						"type": "function",
						"function": {
							"name": "read_skill",
							"arguments": "{\"name\":\"bring\"}"
						}
					},
					{
						"index": 1,
						"id": "call_XA2T71h8bkOFGUQQ0ZLqlsZx",
						"type": "function",
						"function": {
							"name": "shell",
							"arguments": "{\"cmd\":\"bring lists --format json\",\"cwd\":null,\"timeout\":30}"
						}
					}
				]
			},
			"finish_reason": "stop"
		}
	],
	"usage": {
		"prompt_tokens": 3286,
		"completion_tokens": 32,
		"total_tokens": 3318
	}
}

I see that response contains escaped arguments which in this case causes issues.

Here is Output presented by Debug Conversation screen:

init_options:
  start_stage: intent
  end_stage: intent
  input:
    text: Tell me what shopping lists do you have available
  pipeline: 01m26sbzekmd14rxpxrsfsw3mm
  conversation_id: null
stage: done
run:
  pipeline: 01m26sbzekmd14rxpxrsfsw3mm
  language: pl
  conversation_id: 01M27ZZ6VJNSP3MVR0GEAVR1P1
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01m26sbzekmd14rxpxrsfsw3mm
      language: pl
      conversation_id: 01M27ZZ6VJNSP3MVR0GEAVR1P1
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: '2026-09-11T10:24:48.501499+00:00'
  - type: intent-start
    data:
      engine: conversation.extended_openai_conversation
      language: pl-PL
      intent_input: Tell me what shopping lists do you have available
      conversation_id: 01M27ZZ6VJNSP3MVR0GEAVR1P1
      device_id: null
      satellite_id: null
      prefer_local_intents: false
    timestamp: '2026-09-11T10:24:48.501763+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        role: assistant
    timestamp: '2026-09-11T10:24:48.539972+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: Mas
    timestamp: '2026-09-11T10:24:56.027529+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: z
    timestamp: '2026-09-11T10:24:56.033706+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' list'
    timestamp: '2026-09-11T10:24:56.035689+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: 'y'
    timestamp: '2026-09-11T10:24:56.037624+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Spo'
    timestamp: '2026-09-11T10:24:56.039851+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ży
    timestamp: '2026-09-11T10:24:56.207092+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: wc
    timestamp: '2026-09-11T10:24:56.213066+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ze
    timestamp: '2026-09-11T10:24:56.215242+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' P'
    timestamp: '2026-09-11T10:24:56.216623+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: iel
    timestamp: '2026-09-11T10:24:56.218432+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ę
    timestamp: '2026-09-11T10:24:56.372562+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: gn
    timestamp: '2026-09-11T10:24:57.438848+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: acja
    timestamp: '2026-09-11T10:24:57.677539+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' i'
    timestamp: '2026-09-11T10:24:57.680209+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' zdrow'
    timestamp: '2026-09-11T10:24:57.921537+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ie
    timestamp: '2026-09-11T10:24:58.197704+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' Dom'
    timestamp: '2026-09-11T10:24:58.205556+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' i'
    timestamp: '2026-09-11T10:24:58.207095+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' wyposaż'
    timestamp: '2026-09-11T10:24:58.208540+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: enie
    timestamp: '2026-09-11T10:24:58.209914+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' oraz'
    timestamp: '2026-09-11T10:24:58.212577+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ' D'
    timestamp: '2026-09-11T10:24:58.214247+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: zie
    timestamp: '2026-09-11T10:24:58.215403+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: ci
    timestamp: '2026-09-11T10:24:58.216887+00:00'
  - type: intent-progress
    data:
      chat_log_delta:
        content: aki
    timestamp: '2026-09-11T10:24:58.218513+00:00'
  - type: error
    data:
      code: intent-failed
      message: Unexpected error during intent recognition
    timestamp: '2026-09-11T10:24:58.232678+00:00'
  - type: run-end
    data: null
    timestamp: '2026-09-11T10:24:58.232791+00:00'
started: 2026-09-11T10:24:48.501Z
intent:
  engine: conversation.extended_openai_conversation
  language: pl-PL
  intent_input: Tell me what shopping lists do you have available
  conversation_id: 01M27ZZ6VJNSP3MVR0GEAVR1P1
  device_id: null
  satellite_id: null
  prefer_local_intents: false
  done: false
finished: 2026-09-11T10:24:58.232Z
error:
  code: intent-failed
  message: Unexpected error during intent recognition

and now we see: Unexpected error during intent recognition

Can anyone help me with that. I don’t think I have possibility to impact how tool calls look like in response from Docker Agent.

Docs for Docker Agent does not mention anything regarding that.

I don’t assume it’s an issue of Extended OpenAI Conversation addon.
I’m on latest HA version:

[NathanCu (Nathan Curtis)]

@croogie

Unexpected error during intent recognition

Short version.. ^^^ that is the generic error that gets thrown when your tool fails and throws an error and the conversation agent can’t deal with whatever the tool did.

Meaning in almost all cases when you get this error… you MUST fix the tool. (and thank goodness because my Polish sucks and I really didn’t want to have to read it… )

If you did not write that tool you need to go to the person who did and get it corrected. If you did write it then correct the error

Other things that produce this result include timeouts never returning the result to the agent - at about 90 seconds you get this…