Help understanding how to create a correct prompt for Gemini Agent

Configuration Voice Assistant
1

Hey guys, I have a problem with Assist and google generative AI

When I say “turn on the TV” , the TV turns on.

However, when I say “now turn it off” or even directly “turn off the TV” in the same chat, nothing happens. It just responds that the TV has been turned off, but the action is not actually executed.

Below, I’m sharing the debug logs for the first action (which was successful) and the second one (which wasn’t executed), even though the assistant—the conversation agent—replied that it had performed the action.

Same problem with light and everything, first attempt correct second attempt in the same chat or same voice assistant and fail. but the answer is TV on even if is still off

I’m running Gemini 1.5 Flash Latest using those prompt

Schermata 2025-02-19 alle 11.50.28
Schermata 2025-02-19 alle 11.50.281220×1010 47.1 KB

But I’m sure that there is a better way to tell him. I whatched akready on google cloud console but with no result trying other configuration.

TURN ON TV ATTEMPT that worked

init_options:
  start_stage: intent
  end_stage: intent
  input:
    text: acceni televisore soggiorno
  pipeline: 01jkwhjdgcbrzvapy4cgk5877d
  conversation_id: null
stage: done
run:
  pipeline: 01jkwhjdgcbrzvapy4cgk5877d
  language: it
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01jkwhjdgcbrzvapy4cgk5877d
      language: it
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2025-02-19T10:36:26.803078+00:00"
  - type: intent-start
    data:
      engine: conversation.google_generative_ai
      language: it-IT
      intent_input: acceni televisore soggiorno
      conversation_id: null
      device_id: null
      prefer_local_intents: false
    timestamp: "2025-02-19T10:36:26.803114+00:00"
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: Perfetto, il televisore del soggiorno è acceso.
              extra_data: null
          card: {}
          language: it-IT
          response_type: action_done
          data:
            targets: []
            success: []
            failed: []
        conversation_id: 01JMEWFHSKS86P7DD4AMY709YV
    timestamp: "2025-02-19T10:36:28.185195+00:00"
  - type: run-end
    data: null
    timestamp: "2025-02-19T10:36:28.185293+00:00"
intent:
  engine: conversation.google_generative_ai
  language: it-IT
  intent_input: acceni televisore soggiorno
  conversation_id: null
  device_id: null
  prefer_local_intents: false
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: Perfetto, il televisore del soggiorno è acceso.
          extra_data: null
      card: {}
      language: it-IT
      response_type: action_done
      data:
        targets: []
        success: []
        failed: []
    conversation_id: 01JMEWFHSKS86P7DD4AMY709YV

2nd one where i say “turn off TV”

init_options:
  start_stage: intent
  end_stage: intent
  input:
    text: spegni televisore soggiorno
  pipeline: 01jkwhjdgcbrzvapy4cgk5877d
  conversation_id: 01JMEWFHSKS86P7DD4AMY709YV
stage: done
run:
  pipeline: 01jkwhjdgcbrzvapy4cgk5877d
  language: it
  runner_data:
    stt_binary_handler_id: null
    timeout: 300
events:
  - type: run-start
    data:
      pipeline: 01jkwhjdgcbrzvapy4cgk5877d
      language: it
      runner_data:
        stt_binary_handler_id: null
        timeout: 300
    timestamp: "2025-02-19T10:36:41.199590+00:00"
  - type: intent-start
    data:
      engine: conversation.google_generative_ai
      language: it-IT
      intent_input: spegni televisore soggiorno
      conversation_id: 01JMEWFHSKS86P7DD4AMY709YV
      device_id: null
      prefer_local_intents: false
    timestamp: "2025-02-19T10:36:41.199625+00:00"
  - type: intent-end
    data:
      processed_locally: false
      intent_output:
        response:
          speech:
            plain:
              speech: Okay, spengo il televisore in soggiorno.
              extra_data: null
          card: {}
          language: it-IT
          response_type: action_done
          data:
            targets: []
            success: []
            failed: []
        conversation_id: 01JMEWFHSKS86P7DD4AMY709YV
    timestamp: "2025-02-19T10:36:41.720308+00:00"
  - type: run-end
    data: null
    timestamp: "2025-02-19T10:36:41.720422+00:00"
intent:
  engine: conversation.google_generative_ai
  language: it-IT
  intent_input: spegni televisore soggiorno
  conversation_id: 01JMEWFHSKS86P7DD4AMY709YV
  device_id: null
  prefer_local_intents: false
  done: true
  processed_locally: false
  intent_output:
    response:
      speech:
        plain:
          speech: Okay, spengo il televisore in soggiorno.
          extra_data: null
      card: {}
      language: it-IT
      response_type: action_done
      data:
        targets: []
        success: []
        failed: []
    conversation_id: 01JMEWFHSKS86P7DD4AMY709YV

So even if he say Okay, spengo il televisore in soggiorno ( ok i turn off the tv) the action is not done!

Is it a way to set good prompt to make the agent interact correctly with the house? in the prompt command of the agent i tryed to change parameters but with no result

2

Might be worth reading this post (by an AI pro):

1 Like