Probelm with AI TASK

Configuration Voice Assistant
,
1

Every now and then I have a problem with “AI TASK Gemini Generate data”…

I have an automation that when the washing machine finishes sends messages to the phones and a message is generated that is played on a speaker.

it generates a message that is read by TTS.

Except that the text read doesn’t just contain the message it’s trained to read.
It also reads a series of messages like: “service underscore, underscore, underscore, device hash, notify with your mobile device… underscore etc etc” then reads the correct message and ends with “mode single”.

It almost seems like it’s reading parts of the automation.

But this doesn’t always happen… rarely, but it does happen.

any advice?

2

If it’s just text, use conversation.process instead of ai_task.

3

This sounds like an issue with your config. Are you using YAML or the UI? I think you should post your config.

4

I am using UI:

action: ai_task.generate_data
metadata: {}
data:
  instructions: >-
Let me know the cleaner has finished washing and the laundry can be hung out to dry.
Reply in a cynical and humorous way and don't use markdown (**).
Avoid asterisks and emoticons. Use a maximum of two lines of text.
  task_name: Lavatrice ha finito
  entity_id: ai_task.google_ai_task
response_variable: response

Then the answer is sent to a speaker…

action: chime_tts.say
metadata: {}
data:
  message: "\"{{ response.data }}\""
  chime_path: bells
  volume_level: 0.6
target:
  entity_id: media_player.homepod

The automation includes, in addition to the generated message, the sending of notifications to smartphones.

As mentioned, the automation works flawlessly most of the time, but every now and then… it also reads something else. Specifically this is what he reads:

“service notify dot mobile underscore underscore underscore device hash replace with the notification service for your mobile device or other. Data title: “(title of the data)” Message: (Here you can read the title of the message) Mode: Single.”

This is the complete automation:

alias: Lavatrice Finito Lavaggio IA
description: ""
triggers:
  - entity_id:
      - sensor.shellypm3_lavatrice_w
    for:
      hours: 0
      minutes: 2
      seconds: 0
    below: 5
    trigger: numeric_state
    id: Inferiore 5 per 2 minuti
conditions:
  - condition: state
    entity_id: input_boolean.lavatrice_in_funzione
    state: "on"
  - condition: time
    after: "07:00:00"
    before: "22:00:00"
actions:
  - action: input_boolean.turn_off
    data: {}
    target:
      entity_id: input_boolean.lavatrice_in_funzione
  - action: ai_task.generate_data
    metadata: {}
    data:
      instructions: >-
        Avvisami che la lavratice ha finito il lavaggio e i panni possono essere
        stesi. Rispondi in maniera cinica e divertente e non mettere markdown
        (**), evita asterischi ed emoticon. Usa massimo due linee di testo.
      task_name: Lavatrice ha finito
      entity_id: ai_task.google_ai_task
    response_variable: response
  - action: chime_tts.say
    metadata: {}
    data:
      message: "\"{{ response.data }}\""
      chime_path: bells
      volume_level: 0.6
    target:
      entity_id: media_player.homepod
  - alias: Scegli Azione (Notifica Valerio)
    choose:
      - conditions:
          - condition: or
            conditions:
              - condition: zone
                entity_id: person.valerio
                zone: zone.home
              - condition: zone
                entity_id: person.valerio
                zone: zone.in_citta
        sequence:
          - action: notify.mobile_app_iskyphone_16p
            metadata: {}
            data:
              message: La Lavatrice ha finito il lavaggio
              title: Lavatrice
  - alias: Scegli Azione (Notifica Emy)
    choose:
      - conditions:
          - condition: or
            conditions:
              - condition: zone
                entity_id: person.emiliana
                zone: zone.home
              - condition: zone
                entity_id: person.emiliana
                zone: zone.in_citta
        sequence:
          - action: notify.mobile_app_iphone_di_emiliana
            metadata: {}
            data:
              title: Lavatrice
              message: La Lavatrice ha finito il lavaggio
  - alias: Annuncio Bagno
    choose:
      - conditions:
          - type: is_occupied
            condition: device
            device_id: 908d5af428ca203145e253fc70757d3c
            entity_id: 818f0f80d670d6c8dad2419968c4fa45
            domain: binary_sensor
        sequence:
          - action: assist_satellite.announce
            metadata: {}
            data:
              message: "\"{{ response.data }}\""
              preannounce: true
              preannounce_media_id:
                media_content_id: media-source://media_source/local/sounds/bells.mp3
                media_content_type: audio/mpeg
                metadata:
                  title: bells.mp3
                  thumbnail: null
                  media_class: music
                  children_media_class: null
                  navigateIds:
                    - {}
                    - media_content_type: app
                      media_content_id: media-source://media_source
                    - media_content_type: ""
                      media_content_id: media-source://media_source/local/sounds
            target:
              entity_id: assist_satellite.assistant_bagno_09f998_assist_satellite
    enabled: true
  - alias: Seleziona un'opzione (Letto)
    choose:
      - conditions:
          - type: is_occupied
            condition: device
            device_id: cd5f0675a5f043fb274deda216c970d8
            entity_id: 8766a705421c9b435a3a584814fdd5ba
            domain: binary_sensor
        sequence:
          - action: media_player.volume_set
            metadata: {}
            data:
              volume_level: 0.06
            target:
              entity_id: media_player.respeaker
            alias: Volume 0,06
          - action: assist_satellite.announce
            metadata: {}
            data:
              message: "\"{{ response.data }}\""
              preannounce: true
              preannounce_media_id:
                media_content_id: media-source://media_source/local/sounds/bells.mp3
                media_content_type: audio/mpeg
                metadata:
                  title: bells.mp3
                  thumbnail: null
                  media_class: music
                  children_media_class: null
                  navigateIds:
                    - {}
                    - media_content_type: app
                      media_content_id: media-source://media_source
                    - media_content_type: ""
                      media_content_id: media-source://media_source/local/sounds
            target:
              entity_id: assist_satellite.respeaker_assist_satellite
          - alias: Volume 0,6
            action: media_player.volume_set
            metadata: {}
            data:
              volume_level: 0.6
            target:
              entity_id: media_player.respeaker
  - alias: Annuncio Balcone
    choose:
      - conditions:
          - condition: state
            entity_id: input_boolean.movimento_balcone
            state: "on"
        sequence:
          - action: assist_satellite.announce
            metadata: {}
            data:
              message: "\"{{ response.data }}\""
              preannounce: true
              preannounce_media_id:
                media_content_id: media-source://media_source/local/sounds/bells.mp3
                media_content_type: audio/mpeg
                metadata:
                  title: bells.mp3
                  thumbnail: null
                  media_class: music
                  children_media_class: null
                  navigateIds:
                    - {}
                    - media_content_type: app
                      media_content_id: media-source://media_source
                    - media_content_type: ""
                      media_content_id: media-source://media_source/local/sounds
            target:
              entity_id:
                - assist_satellite.m5stack_atom_echo_fdaf90_assist_satellite
mode: restart

I saw the action: “conversation.process”

But I don’t see an “instructions” section where Google AI can generate the response based on my instructions. There’s only a “text” section.

5

Can you look at the automation trace and show what this response variable looked like when it went wrong?

The escaping of message doesn’t seem right. I don’t think it’s your issue, but it should be:

data:
  message: "{{ response.data }}"

Otherwise, there will be literal quotes included in the message.

6

thanks… I hadn’t noticed this, it had escaped me…

It’s very difficult to run tests and analyze data because this rarely happens… (and as often happens, I’m not at home) but from what I think I remember, it only happens with GEMINI; not with OpenAI. (Or maybe I don’t remember trying to change).

7

I was facing a similar problem. The solution was to specify a response variable and then use [variable name].data as the text to be sent to tts. For the instructions, I added the sentence “The response should include only the text of the message and nothing else.”

This is an example of what worked:

action: ai_task.generate_data
metadata: {}
data:
task_name: say fact
instructions: >-
Generate the text for a message that wishes Allen a good morning in a
unique and clever way. The response should include only the text of the
message and nothing else.
entity_id: ai_task.claude_ai_task
response_variable: ai_text

Then for the tts action:

action: tts.cloud_say
metadata: {}
data:
cache: false
message: "{{ ai_text.data }}"
entity_id: media_player.first_player

8

I'm starting to get a little bored with Cloud AI.

Every time there's a problem: tokens run out (even though they're paid for), misunderstood actions, actions not completed,the system is too busy, actions or messages that don't execute etc.

But the main problem is that sometimes they work perfectly, then they stop working, then magically they start working again.

Then you have to modify the output because they've complicated things, etc. etc.

I think HOMEASSISTANT should be something that simplifies.