AI is not triggering any light on, and instead it's hallucinating

1

I am testing the new feature of starting conversations from automations.
This is a very simple one where I want the HA voice preview to ask me if I want to turn on the lights when the necessary condition matches. However when I answer “yes I want you to turn on the lights” it just says “enjoy the new mood of your room” or something like that but nothing happens. I was honestly expecting to get some kind of boolean or answer from the llm instead of letting it do whatever it wants.
How is people doing this automations?

Here is the YAML, sorry for the Spanish on it:

alias: AI pregunta si encender la luz del salón
description: ""
triggers:
  - trigger: state
    entity_id:
      - group.restroom_lights
    for:
      hours: 1
      minutes: 0
      seconds: 0
    to: "off"
  - trigger: state
    entity_id:
      - binary_sensor.motion_7_occupancy
    to: "on"
conditions:
  - condition: and
    conditions:
      - condition: time
        after: "18:00:00"
        before: "22:00:00"
      - condition: numeric_state
        entity_id: sensor.motion_7_illuminance_lux
        below: 40
      - condition: state
        entity_id: group.restroom_lights
        for:
          hours: 1
          minutes: 0
          seconds: 0
        state: "off"
actions:
  - action: assist_satellite.start_conversation
    metadata: {}
    data:
      preannounce: true
      start_message: Parece que está oscuro, ¿Quieres que encienda la luz?
      extra_system_prompt: >-
        El usuario deberá decidir si quiere o no encender la luz del salón.
        Intenta encender la escena llamada "cena", si no, la luz más tenue que
        se pueda, probablemente "redonda"
    target:
      area_id: affdb988d1c04054962b41c896ac5311
mode: single
2

This is usually a context or context confusion issue.

That means I need to read what is in that extra prompt. I kind of read Spanish so I’ll give it a shot but it’d be better if you could translate?

In Ai prompt words matter. And I’d like the translation from your perspective not Google.

3

Ok this is what I see…

Add explicit instructions to the extra_system_prompt. Something like:

“Use the scene.turn_on service for ‘cena’, or light.turn_on with brightness 10 for ‘light.redonda’.”

More below.

Double-check that:

The “cena” scene exists and is spelled exactly that way

“redonda” is a valid light entity in the target area_id

You need to setup the guardrails (you are. Listening for X and make a decision to Y, W, or Z) and make those choices lead to instructions that are EXPLICIT, leave nothing to chance.

Use intent x with setting to…

Now if this is happening a LOT and not just here then you may be blowing out context window…

4

Thanks, those are good points, I will try them.
I use AI APIs and they usually offer a way to tell the agent to “use tools”, I was expecting HA agent to be more inclined to use tools that to provide random text responses. This , more than ever, feels like “spells” more than programming :rofl:

5

Read my Friday’s Party post. In detail.

This will only be as gothe context you are giving it. Assume at this point it has whatever prompt you already use for the running agent and this is slapped at the bottom so you don’t have to redefine tools but you ABSOLUTELY MUST describe to the ai what the current condition is and what you expect it to do and be AS EXPLICIT about it as you can.

I do something like this with my garage door and the extra prompt is something like

Friday. The garage door has triggered an alert because it has been open for 15 minutes (this changes if its dark the timer goes down to 5)

You are being cast to the nearest voice assistant to the family… Please tell the fam that the garage has been open and that if you do not hear anything saying that it should remain open, close it by issuing (exact intent to close the garage - I think it’s hass_turn_off the garage doors group entity…)

She wakes announces the garage is open and probably should be closed do you want her to do it. If you don’t do anything. The garage closes.

6

Your prompt is the issue here do you have your scene expose?
i just made a test with this

alias: AI pregunta si encender la luz del comedor
description: ""
triggers:
  - trigger: numeric_state
    entity_id:
      - sensor.asistente_oficina_lux
    below: 50
conditions:
  - condition: and
    conditions:
      - condition: state
        entity_id: binary_sensor.192_168_1_95
        state: "on"
actions:
  - action: assist_satellite.start_conversation
    metadata: {}
    data:
      preannounce: true
      start_message: Parece que está oscuro, ¿Quieres que encienda la luz?
      extra_system_prompt: si el usuario dice si prende la luz comedor
    target:
      entity_id: assist_satellite.asistente_oficina_assist_satellite
mode: single

it runs without issues

tu prompt es el problema tiennes la scena expuesta?
hize una prueba con este codigo y funciona sin problema

1 Like
7

Yes, the scene is available.
And just yesterday I discovered that it was indeed called, but the only light it turns on was not working because the latest local-tuya update.

That said, it only called it once out of the 3 times I tried it.
Will try to follow @NathanCu suggestions, but thank you very much for your answer too

1 Like